The Effect of Sensor Type and Location on Reported Air Quality Index Reliability in New York

Abstract

In recent years, air pollution has dramatically increased, creating a potentially dangerous environment as high levels of exposure can lead to life-long cardiopulmonary diseases and even cancer. To make matters worse, many of the substances that contribute to air pollution are invisible to the naked eye, making it nearly impossible for the average person to determine how polluted their air is. Thus, people must rely on professional sensors to accurately determine air quality. This study aimed to measure the accuracy of low-cost air quality detection sensors. This study utilized a 3 x 10 design, comparing the air quality index outputs of three different sensor companies (government, PurpleAir, and Airbeam 3) and ten different ZIP codes across New York, United States of America (five of which belonged to highly polluted areas and five of which belonged to lowly polluted areas) on the similarity of their respective outputs. A t-test revealed that the air quality index (AQI) reported by Airbeam 3 was not significantly different from those reported by the government (p = .296); however, those reported by PurpleAir were (p = .005). Furthermore, a two-way ANOVA revealed no interactions between ZIP code and sensor type; however, differing ZIP codes had a significant effect on reported AQI values (p = .006). Currently, most corporations, schools, and other organizations utilize PurpleAir to detect air pollution levels; however, the results of this study provide strong evidence that Airbeam 3 might be the better option.

Keywords: AQI, air pollution detection, low-cost sensors, PurpleAir, Airbeam 3

Introduction

Background

Despite recent attempts to curb the amount of pollutants released into the air, air quality remains a significant concern worldwide. In 2019, air pollution destruction and costs reached a staggering $8.1 trillion and are only increasing¹. Aside from the economic aspect, each year, air pollution contributes to roughly 6.5 million deaths worldwide². Furthermore, exposure to air pollution, short or long, can increase the likelihood of respiratory infections, heart disease, and lung cancer³.

Given the fast-paced nature of technological developments, air quality monitoring is an ever-changing science. Recently, within the United States, guidelines initiated by the Environmental Protection Agency (EPA), mainly focusing on the restriction of carbon emissions from automobiles and commercial facilities, have managed to reduce certain pollutants like carbon dioxide and lead levels⁴. Despite these guidelines, other pollutants, such as particulate matter less than 10 µm in diameter, or PM10, have increased⁵. In July of 2023, air pollution levels in New York reached an all-time high. The daily average, particulate matter less than 2.5µm in diameter, or PM2.5, has reached a staggering 117 μg m-3, significantly greater than the World Health Organization (WHO) guideline of 15 μg m-3.Using air pollution detection devices is vital to avoid highly polluted environments and find ways to correct these situations. However, not all air pollutants are visible, making it difficult for people to rely on naturalistic observations to determine air quality. Many of the most dangerous gasses, such as ozone, sulfur, and carbon dioxide, are invisible to the naked eye⁶.

Thus, air quality monitors must be used to detect air pollution.

Existing air pollution monitors used by large businesses and organizations are costly, making it harder for individual consumers to obtain monitors for their own homes⁷.

Furthermore, many companies make it difficult for individual consumers to purchase their monitors in the first place because they are geared toward large companies and institutions. However, great strides have recently been made in creating cheaper and more affordable monitors by reducing their technical components. Low-cost sensors typically range in price between $100 to $2,500, a fraction of what high-cost sensors cost⁷.

The problem with this, however, is that reducing the amount of technology within them also, in turn, limits their capabilities. This makes newer, low-cost air quality monitors less reliable and less capable than higher cost and air quality monitors⁸. Previous research has shown that certain low-cost air quality monitors are more accurate than others⁹,¹⁰,¹¹  . This performance difference is attributed to certain low-cost air quality monitors working differently than others. PurpleAir is one of the most commonly used air quality monitors and it works by using a fan to create a vacuum and draw in air. Afterward, a laser creates reflections within the air to detect particles between .3 and 10 micrometers in diameter.¹² On the other hand, sensors like Temptop M2000 2 also use a laser, however, it uses a joint algorithm to detect the presence of air pollutants¹³. 

Low-cost sensors could be improved, as their lacking technology is noticeable compared to high-cost sensors’ performance. For instance, one prior study placed low, medium, and high-cost sensors in chambers with simulated aerosol/microparticle contents. Ultimately, the researchers discovered that low-cost sensors had difficulty detecting microparticles on the smaller side8. Another study randomly selected real locations and simulated the conditions from that area within a lab. Low-cost sensors were placed in each of these simulated conditions, and it was determined that the low-cost sensors generally under classified air pollution levels¹¹.  The poor performance of these low-cost sensors at detecting smaller particles is likely due to their less advanced technology when compared to that of medium high-cost sensors. Since different areas contain different levels of pollution, this raises a concern that perhaps low-cost sensors cannot be used and trusted everywhere, especially in regions with lower pollution levels.

Currently, most consumers use low-cost air quality monitors to determine their homes’ particulate and pollution levels. In addition to their geographical location, the performance of low-cost air quality monitors can also be impacted by how they are used. For instance, studies have found that low-cost air quality monitors are generally more effective at detecting indoor air and particulate levels than outdoors. This is because a closed environment allows these monitors to more accurately measure air pollution and particulate levels¹⁴. 

While not being the most accurate, evidence does suggest that low-cost sensors can be a viable, long-term solution to monitoring outdoor pollution levels¹⁵. For instance, one study created an air quality monitoring system that incorporated a Plantower PM2.5 sensor and an Alphasense carbon monoxide (CO) sensor and placed it in a series of outdoor locations. Ultimately, these monitors could generally pick up everything regarding air pollution and particulate levels.

Across the United States, PurpleAir tends to be the most popular option for a low-cost air quality monitor; however, other companies, such as Airbeam, have recently made great strides in perfecting and coming out with new monitors. This study aims to determine the best and most effective low-cost air quality monitor. 

Research Question

This study aimed to determine the effectiveness of low-cost air quality monitors,

PurpleAir and Airbeam 3, in detecting air pollution levels in New York. 

Hypotheses

Based on the literature reviewed, we hypothesize the following: 

1. Compared to Airbeam 3, PurpleAir will more accurately report nationally accepted air quality index values.
   a. Airbeam 3 and PurpleAir are more likely to under-report PM2.5 concentrations than over-report.

• Compared to areas with low PM2.5 concentrations, PurpleAir and Airbeam 3’s concentration readings will need to be more accurate in areas with high PM2.5 concentrations.

Materials and Methods

The study utilized a 3 x 10 design, comparing the effect of preexisting pollution levels and different types of sensors on sensor accuracy. Over a two-week testing period, the air quality index for PM2.5 concentrations from each sensor in various ZIP codes was collected daily via various open-source websites from 6:00 pm (EST) to 7:00 pm (EST). Afterward, these data points were compiled into Google Sheets. Since PM2.5 concentrations fluctuate throughout the day, it was crucial to maintain a consistent testing schedule to ensure accurate comparisons.

This study used data from PurpleAir, Airbeam 3, and the EPA’s AirNow program.

PurpleAir and Airbeam 3 are low-cost air quality monitors and act as experimental conditions. The government-reported averages from the EPA acted as the population and a basis for comparison against the performance of the low-cost sensors¹⁶. PurpleAir and Airbeam 3 were chosen because they are two of the most popular sensors. PurpleAir dominates the low-cost sensor market and is used by various universities. As mentioned earlier, these sensors use a fan-laser system to take a sample of the surrounding air and scan it for microparticles and pollutants¹².. 

To determine the effect of preexisting pollution levels on detection capabilities, we intentionally chose five of the most polluted ZIP codes in New York and five of the cleanest. The five reported dirtiest ZIP codes in New York included the following: 11040, 10475, 10010, 11001, and 10044¹⁷. The 5-reported cleanest ZIP codes in New York included the following: 10309, 10306, 10308, 10474, and 10312. The dirtiest ZIP codes represent the regions with the highest preexisting pollution levels, whereas the cleanest ZIP codes represent the regions with the lowest levels. Typically, low-cost sensors have more difficulty detecting smaller particles¹⁸. Thus, it was hypothesized that they would perform the worst in areas with high PM2.5 concentrations, as more pollution particles of any size exist within these regions.

A series of open-source websites were used to collect the real-time air quality index of PM2.5 concentrations among PurpleAir, Airbeam 3, and nationally reported averages to determine sensor accuracy. PurpleAir’s official website has a map that displays PM2.5 concentrations (in terms of the air quality index) worldwide¹⁹ (Figure 1). A website entitled

Habitat Map was used to collect data on the PM2.5 concentrations of Airbeam 3²⁰ (Figure 2). The problem with Habitat Map is that the data it reports is in terms of PM2.5 concentrations, not the air quality index. Thus, a separate website, provided by The World Air Quality Index Project, was needed to convert the concentration values to air quality index values²¹. For reference, this website utilized the same scale used by the EPA to convert raw PM2.5 concentrations to AQI—meaning the AQI values outputted by all sensors were comparable and utilized the same scale. As stated before, the EPA’s AirNow website was utilized to collect data on the nationally reported averages¹⁶. The ZIP code was entered into their respective websites in each condition and the regional average was noted.

At the end of the testing period, data from Google Sheets was compiled into SPSS. Afterward, a series of ANOVAs and independent sample t-tests were run to analyze the data.

Results and Discussion

Results

A three-way ANOVA did not reveal a significant interaction between ZIP codes, preexisting categorization level, and sensor type. However, it did reveal a significant interaction between ZIP code and sensor type (p = .021). No other significant interactions were found. Below is a summary statistical output from SPSS as a result of this ANOVA showcasing the mean AQI and standard deviation values within each sensor type and ZIP code (Table 1).

Descriptive Statistic			Mean AQI	Standard Deviation of AQI
Sensor Type		EPA AirNow	31.13	13.15
		PurpleAir	23.67	16.49
		Airbeam 3	36.28	14.50
ZIP Codes	High Pollution Categories	10312	28.78	16.47
		10309	33.81	14.73
		10306	32.92	14.83
		10308	31.81	14.50
		10474	38.81	17.72
	Low Pollution Categories	10010	24.00	16.65
		10044	29.61	14.06
		10010	30.92	11.60
		10475	30.56	13.52
		11040	21.42	16.65

To compare the accuracy of AQI reporting in PurpleAir and Airbeam 3 compared to nationally reported values, two separate t-tests were run. The first t-test revealed that the AQI reported by PurpleAir devices significantly differed from those provided by the government (p = .005). Furthermore, PurpleAir generally reported, on average, lower AQI values than EPA AirNow. The second t-test revealed that the AQI reported by Airbeam 3 was not significantly different from those provided by the government (p = .296). The AQI values reported by Airbeam 3 were generally higher than those reported by the EPA AirNow. Below is a summary statistical output from SPSS showcasing the t-test results described previously and a chart graphing the mean AQI values reported by each sensor across the experimental period (Table 2; Figure 3).

	Sensor Type	N	t	Significance
AQI	Purple Air	120	3.874	0.005
	Airbeam 3	120	-2.878	0.296

To further analyze the practical significance of the performance of PurpleAir and Airbeam 3 when compared to AirNow, Cohen’s d was manually calculated for each t-test by utilizing the following formula:

   

For reference, n1 and n2 refer to the sample sizes used to perform the t-test and t refers to the test-statistic of each test. Utilizing this information and the above formula, we determined the Cohen’s d values of Purple Air and Airbeam 3’s t-tests to be approximately 0.707 and -0.525, respectively. Both values indicate a medium level of effect size, building onto the case that the results presented in this study have practical implications and relevance.

To evaluate the impact of location on sensor reliability, we specifically looked at the effect of different ZIP codes and their respective pollution level categorization (high or low). An independent t-test revealed a significant difference between the reported AQIs in ZIP codes in low and high-pollution areas (p = .000). Furthermore, sensors in ZIP codes that have had historically high pollution levels, on average, reported higher AQI values than those in historically lower pollution levels. Despite this, upon further inspection, a one-tailed t-test revealed that the average AQI reported in high-pollution level regions was not significantly higher than those reported in low-pollution level regions (p = .160). A simple main effects analysis under the previously run ANOVA revealed that ZIP code significantly affected the reported AQI values from each sensor (p = .006).

Discussion  

While it was hypothesized that PurpleAir would report more accurate AQI values than the nationally reported ones compared to Airbeam 3, t-tests revealed the opposite. The AQI values reported from PurpleAir were significantly different from those reported by the government, while those reported from Airbeam 3 were not. This provides evidence that

PurpleAir is less reliable than Airbeam 3, making Airbeam 3 the better low-cost sensor.

We hypothesized that PurpleAir and Airbeam 3 would be more likely to report lower AQI values than the government. The truth, however, was that PurpleAir was more likely to lowerAQI values, while Airbeam 3 was more likely to report higher AQI values (Figure 3). This finding contradicts our original hypothesis but is understandable. Airbeam 3 was released much more recently than PurpleAir, and fewer testing has been done on its capabilities, meaning we had less resources to create a hypothesis to estimate its performance compared to PurpleAir. Overall, these results support the idea that Airbeam 3 is the better low-cost sensor, as generally, it is more dangerous for a sensor to report lower AQI values than higher values.

Lastly, it was hypothesized that low-cost sensors would have a harder time producing accurate readings in regions with high pollution than in regions with low pollution levels. While the AQI reported values in each region differed significantly, one was not significantly higher.  The results of this study provide strong evidence that both sensor type and location heavily impact sensor accuracy in reporting AQI values. Furthermore, while both location and sensor type significantly affect reported AQI values individually, there is also a significant interaction between the two variables.  Contrary to the findings of this study, in previous literature, researchers have found that usually low-cost sensors have a harder time detecting particulate matter in lower pollution areas⁹,²³ . This difference is likely due to the fact that these studies utilized simulated environments, which raises the potential for various inaccuracies and measurement biases. Since this study utilized a ‘real-world’ sample, the results presented in this study are more accurate to the types of environments sensors will be located and exposed to.

Limitations

Despite the experiment’s outcome, various factors may have impacted the results. Firstly, while using the government website, AirNow automatically gave the AQI reading of an entire ZIP code; we had to manually calculate it when collecting data for PurpleAir and Airbeam 3. This opens up the potential for measurement bias when manually calculating the data. Secondly, while PurpleAir and AirNow reported PM2.5 concentrations using the AQI system, Airbeam 3 reported these values in their raw form. Thus, we used a conversion website to convert these values into AQI values. This opens the possibility of misconverting these values, as the conversion was based on a scale provided by the EPA. That being said, since AirNow and Purple also used this scale (albeit automatically), this likely minimizes the likelihood of measurement biases. Lastly, while we did know the zip codes that each sensor was located in, we had little to no information on their exact location. Sensors closer to a busy street or intersection would have higher AQI readings when compared to sensors placed in residential areas.   

Conclusion

PurpleAir is the most commonly used low-cost sensor across the United States. However, the results of this study suggest that Airbeam 3 may be the better option because compared to PurpleAir, this device reported more accurate AQI values and tended to report higher values as opposed to lower values. From a safety standpoint, it is better for a device to report higher values because this will only lead to an increase in caution and preparedness, as people attempt to protect themselves from high pollution exposure. On the contrary, if they believe the pollution levels to be lower than they are, this can lead to them being underprepared and ill-equipped to deal with the pollution and open their risk to the potential of pollution-related illnesses.  

Acknowledgments 

I would like to thank Jayson Toweh for guiding me in the research and paper-writing process. Additionally, I would also like to thank Lumiere for providing me the opportunity to work with Jayson Toweh.

References

1. World Bank. (2023). World Bank supports air quality improvement in the Kyrgyz Republic. https://www.worldbank.org/en/news/press-release/2023/11/29/world-banksupports-air-quality-improvement-in-the-kyrgyzrepublic#:~:text=Air%20pollution%20is%20the%20leading,6.1%20percent%20of%20global%20GDP. [↩]
2.    Hyslop, N. (2009). Impaired visibility: The air pollution people see. Atmospheric Environment, 43(1), 182-195. https://doi.org/10.1016/j.atmosenv.2008.09.067 [↩]
3. World Health Organization. (2019). Health consequences of air pollution on populations. https://www.who.int/news/item/15-11-2019-what-are-health-consequencesof-air-pollution-onpopulations#:~:text=Exposure%20to%20high%20levels%20of,people%20who%20are%20already%20ill. [↩]
4. National Institute of Environmental Health Sciences. (n.d.). Air pollution and your health. https://www.niehs.nih.gov/health/topics/agents/air-pollution [↩]
5. United States Environmental Protection Agency. (n.d.). Air quality- National challenges. https://www.epa.gov/air-trends/air-quality-national-summary [↩]
6. Hyslop, N. (2009). Impaired visibility: The air pollution people see. Atmospheric Environment, 43(1), 182-195. https://doi.org/10.1016/j.atmosenv.2008.09.067 [↩]
7. Puget Sound Clean Air Agency. (n.d.). Air Quality Sensors. https://www.pscleanair.gov/539/Air-Quality-Sensors#:~:text=These%20sensors%20are%20available%20to,nitrogen%20oxide%2C%20and%20many%20others. [↩] [↩]
8. Considine, E. (2020). Calibrating low-cost air pollution sensors using machine learning. Earth Lab. https://earthlab.colorado.edu/blog/calibrating-low-cost-air-pollutionsensors-using-machinelearning#:~:text=One%20of%20the%20challenges%20with,accurate%20by%20using%2 0statistical%20methods [↩]
9. Sousan, S., Koehler, K., Thomas, G., Park, J., Hillman, M., Halterman, A., & Peters, T. (2015). Inter-comparison of low-cost sensors for measuring the mass concentration of occupation aerosols. Aerosol Science and Technology, 50(5), 462-473. https://doi.org/10.1080/02786826.2016.1162901 [↩] [↩]
10. Liu, X.,  Jayarante, R., Thai, P., Kuhn, T., Zing, I., Christensen, B., Lamont, R., 

    Dunbabin, M., Zhu, S., Gao, J., Wainwright, D., Neale, D., Kan, R., Kirkwood, J., & Morawska, L. (2020). Low-cost sensors as an alternative for long-term air quality monitoring. Environmental Research, 185, 1-11. https://doi.org/10.1016/j.envres.2020.109438 [↩]

11. Liu, X.,  Jayarante, R., Thai, P., Kuhn, T., Zing, I., Christensen, B., Lamont, R., Dunbabin, M., Zhu, S., Gao, J., Wainwright, D., Neale, D., Kan, R., Kirkwood, J., & Morawska, L. (2020). Low-cost sensors as an alternative for long-term air quality monitoring. Environmental Research, 185, 1-11. https://doi.org/10.1016/j.envres.2020.109438 [↩] [↩]
12. Mojave Desert Air Quality Management District. (n.d.). PurpleAir- community sensor map. https://www.mdaqmd.ca.gov/air-quality/purple-air-community-monitormap#:~:text=PurpleAir%20sensors%20use%20a%20fan,from%20particles%20in%20the %20air [↩] [↩]
13. South Coast Air Quality Management District. (n.d.). Sensor Detail. https://www.mdaqmd.ca.gov/air-quality/purple-air-community-monitormap#:~:text=PurpleAir%20sensors%20use%20a%20fan,from%20particles%20in%20the %20air [↩]
14. Connolly, R., Yu, Q., Wang, Z., Chen, Y., Liu, J., Collier-Oxandale, A., Papapostolou, V., Polidori, A., & Zhu, Y. (2022). Long-term evaluation of a low-cost air sensor network for monitoring indoor and outdoor air quality at the community scale. Science of the Total Environment, 807, 1-11. https://doi.org/10.1016/j.scitotenv.2021.150797 [↩]
15. Liu, X.,  Jayarante, R., Thai, P., Kuhn, T., Zing, I., Christensen, B., Lamont, R.,  Dunbabin, M., Zhu, S., Gao, J., Wainwright, D., Neale, D., Kan, R., Kirkwood, J., & Morawska, L. (2020). Low-cost sensors as an alternative for long-term air quality monitoring. Environmental Research, 185, 1-11. https://doi.org/10.1016/j.envres.2020.109438 [↩]
16. Environmental Protection Agency. (n.d.). AirNow. https://www.airnow.gov/ [↩] [↩]
17. Shea, T. (2023). There are the dirtiest and cleanest IP ZIP codes in NYC: Study. NBC. https://www.nbcnewyork.com/news/local/these-are-the-dirtiest-and-cleanest-zipcodes-in-nyc-study/4604885/ [↩]
18. Sousan, S., Koehler, K., Thomas, G., Park, J., Hillman, M., Halterman, A., & Peters, T. (2015). Inter-comparison of low-cost sensors for measuring the mass concentration of occupation aerosols. Aerosol Science and Technology, 50(5), 462-473. https://doi.org/10.1080/02786826.2016.1162901 [↩]
19. PurpleAir. (n.d.). Real-time Air Quality Map. https://map.purpleair.com/1/a/mAQI/a0/p86400/cC0#13.1/40.57312/-74.12499 [↩] [↩]
20. 16. Habitat Map. (n.d.). Air Casting. http://aircasting.habitatmap.org/mobile_map#?map=%7B%22zoom%22:14,%22lat%22:40.59320371094067,%22lng%22:-74.09148166966598,%22mapType%22:%22roadmap%22,%22hasChangedProgrammati cally%22:false%7D&data=%7B%22sensorId%22:%22Particulate%20Matter-airbeampm2.5%20(%C2%B5g%2Fm%C2%B3)%22,%22location%22:%22%22,%22tags%22:%22%22,%22usernames%22:%22%22,%22timeFrom%22:%221680048000%22,%22timeTo%22:%221711756799%22,%22heat%22:%7B%22lowest%22:0,%22low%22:12,%22mid%22:35,%22high%22:55,%22highest%22:150%7D,%22gridResolution%22:31,%22c rowdMap%22:false%7D [↩]
21. The World Air Quality Index Project. (n.d.). AQI calculator- US EPA scale convertor. https://aqicn.org/calculator/ [↩]
22. Habitat Map. (n.d.). Air Casting. http://aircasting.habitatmap.org/mobile_map#?map=%7B%22zoom%22:14,%22lat%22:40.59320371094067,%22lng%22:-74.09148166966598,%22mapType%22:%22roadmap%22,%22hasChangedProgrammati cally%22:false%7D&data=%7B%22sensorId%22:%22Particulate%20Matter-airbeampm2.5%20(%C2%B5g%2Fm%C2%B3)%22,%22location%22:%22%22,%22tags%22:%22%22,%22usernames%22:%22%22,%22timeFrom%22:%221680048000%22,%22timeTo%22:%221711756799%22,%22heat%22:%7B%22lowest%22:0,%22low%22:12,%22mid%22:35,%22high%22:55,%22highest%22:150%7D,%22gridResolution%22:31,%22c rowdMap%22:false%7D [↩]
23. Liu, X.,  Jayarante, R., Thai, P., Kuhn, T., Zing, I., Christensen, B., Lamont, R., 

    Dunbabin, M., Zhu, S., Gao, J., Wainwright, D., Neale, D., Kan, R., Kirkwood, J., & Morawska, L. (2020). Low-cost sensors as an alternative for long-term air quality monitoring. Environmental Research, 185, 1-11. https://doi.org/10.1016/j.envres.2020.109438 [↩]