wfgarnett3 Posted December 30, 2019 Report Share Posted December 30, 2019 To summarize the below inquiry: On July 11, 2019 the South Jersey Regional airport inaccurate TMIN of 50 degrees Fahrenheit was 16 and 18 degrees lower than the values reported from nearby Neshaminy Falls and Trenton Mercer airport and appropriately failed the spatial consistency QA test. Yet on October 2, 2019 the Neshaminy Falls inaccurate TMIN of 52 degrees Fahrenheit was 15 and 18 degrees lower than the values reported from nearby Trenton Mercer and South Jersey Regional airport yet did not fail the spatial consistency QA test. Was a spatial consistency test really done by NOAA on the October 2 TMIN value for Neshaminy Falls? If not, why not? If so, why did it pass where the similar example from July 11 for South Jersey failed the test? Does it have to do with the way the math behind the spatial regression and spatial corroboration tests are structured? (I e-mailed NOAA also and will post an update when I hear from them) -------------------------------------------------- For September 17, 2019 a minimum temperature value of 49 degrees Fahrenheit was reported for the Neshaminy Falls (GHCND ID: USC00366194) COOP weather station located in Pennsylvania just outside Philadelphia. On September 18, 2019 I contacted The National Weather Service (NWS) / NOAA and I informed them this value was impossible. NWS / NOAA agreed and changed the TMIN value for September 17 for Neshaminy Falls from 49 degrees to 55 degrees Fahrenheit which was the At Observation temperature (TOBS) reported just before end of day midnight. On October 2, 2019 a TMIN value of 52 degrees Fahrenheit was reported for Neshaminy Falls. I contacted NOAA once again and I informed them this value was impossible (I did not hear back from them). October 2 was the day the TMAX temperature was over 90 degrees Fahrenheit from the Southeastern United States all the way up to the New York City area. No weather station in the Philadelphia area, not even Mount Pocono near 2000 feet elevation and much further north got anywhere near a TMIN of 52 degrees Fahrenheit that day. In addition for the ASOS stations in the entire area the hourly reported dewpoint values for the whole day were over 60 degrees Fahrenheit thefore since actual temperature can never go below dewpoint temperature it was scientifically impossible to have a TMIN of 52 degrees at Neshaminy Falls. At the link below it states archive-ready data sources are available 45 to 60 days after the close of a data month, and for COOP weather stations the sflag will flip from "H" to "7". Today is the 60th day from the close of October and the flag has not flipped - I called NCEI and they stated there is a delay for the reprocessing but that the QA would have already been done. Thus for Neshaminy Falls the TMIN of 52 degrees Fahrenheit (11.1 degrees Celsius) on October 2 would be official. https://www.ncdc.noaa.gov/ghcn-daily-description We can compare the Neshaminy Falls (40 feet elevation) station's October 2 TMIN value of 52 degrees to ASOS Trenton Mercer Airport, New Jersey weather station (USW00014792 190 feet elevation), which is located 11.46 miles away, and ASOS South Jersey Regional Airport, NJ (USW00093780 49 feet elevation), which is located 15.53 miles away. All 3 weather stations report nearly on a daily midnight to midnight schedule for TMAX and TMIN so are appropriate for comparisons. ASOS Northeast Philadelphia Airport and ASOS Philadelphia International Airport sometimes have an urban heat island effect -- they reported TMIN values of 72 and 70 Fahrenheit respectively that day. Trenton Mercer Airport for October 2 reported a TMIN of 67 degrees Fahrenheit while South Jersey Regional Aiport reported a TMIN of 70 degrees Fahrenheit. Thus Neshaminy Falls inaccurate TMIN value of 52 degrees on October 2 is 15 and 18 degrees lower than these 2 other nearby New Jersey weather stations. Over at the link below at climate.gov NOAA states: "Observers submit their raw data to NOAA’s National Center for Environmental Information (NCEI) on a daily or monthly schedule for further quality checks." "Weather data are also checked for consistency across a region. Scientists observe data sets from comparable stations to see if the data makes sense for the region and time of year." "Once the COOP data has passed quality control, it becomes part of the larger data record known as the Global Historical Climate Network-Daily (GHCN-D) database. The data can then be processed to generate climate products such as maps and graphs." https://www.climate.gov/maps-data/primer/processing-climate-data Below are the links detailing the QA process for GHCN-Daily, including the Durre et al. paper "Comprehensive Automated Quality Assurance of Daily Surface Observations" which is easy to read and understand: https://www.ncdc.noaa.gov/ghcn-daily-methods https://journals.ametsoc.org/doi/full/10.1175/2010JAMC2375.1 I understand as stated that the test thresholds are chosen to yield the highest error detection rate without exceeding 20% false-positive rate. Also obviously, for instance, flagging temperature values that might be only 1 degree off is imprudent. Section 6 and Table 4 detail the "spatial consistency" tests (regression and corroboration). I read the explanations and I get the "idea" of the math behind the tests but note the Neshaminy Falls TMIN on October 2 exceeds a 8 degree Celsius (14.4 degree Fahrenheit) difference from the reported TMIN for BOTH Trenton Mercer aiport and South Jersey Regional airport that day and also has exactly a 10 degree Celsius (18 degree Fahrenheit) difference from South Jersey Regional airport. Also note in the GHCN-Daily on July 11, 2019 for South Jersey Regional airport the TMIN of 50 degrees Fahrenheit (10 degrees Celsius) has a qflag of "S" and thus failed the spatial consistency test. On that same July 11 day Neshaminy Falls reported a TMIN of 66 degrees Fahrenheit while Trenton Mercer Airport reported 68 degrees Fahrenheit (20 degrees Celsius). So we have the case where on July 11 South Jersey Regional airport inaccurate TMIN was 16 and 18 degrees Fahrenheit lower than the values reported from Neshaminy Falls and Trenton Mercer airport and appropriately failed the spatial consistency test, yet on October 2 Neshaminy Falls inaccurate TMIN was 15 and 18 degrees Fahrenheit lower than the values reported from Trenton Mercer airport and South Jersey Regional airport yet did not fail the spatial consistency test. Also note the last time Neshaminy Falls failed a spatial consistency test was the TMIN value of 24 degrees Fahrenheit on October 23, 2016 (that Trenton Mercer airport and South Jersey Regional airport reported respectively 43 degrees and 46 degrees Fahrenheit). Focusing on the October 2, 2019 TMIN value of 52 degrees Fahrenheit for Neshaminy Falls, the next logical questions I have are: 1. Did NOAA really run a spatial consistency test on the TMIN value on October 2 for Neshaminy Falls? a) If the answer is no, why not? 2. If a spatial consistency test was done and it passed QA does this mean regardless of the fact the value had a difference from nearby weather stations that exceeded 8 degrees Celsius is this due to the way the math behind the spatial consistency tests (both regression and corroboration) are structured? a) In light of the examples above would it make sense to modify the math behind the spatial consistency tests? 3. Why did the July 11 example from South Jersey Regional airport fail the spatial consistency test yet the similar October 2 example from Neshaminy Falls did not? 4. Do you think since the eastern United States had hot 90 degree plus Fahrenheit afternoon temperatures on October 2 that somehow that TMIN of 52 by Neshaminy Falls should have been caught by some test since it so "obviously" wrong? 5. Furthermore, not counting the unarchived November 1 TMAX of 73 and the December 11 TMIN of 15 for Neshaminy Falls that are obviously wrong I have 2 other recent archived examples from Neshaminy Falls. September 15 TMIN of 54 Fahrenheit (not sure about this one since it did rain). October 1 TMIN of 53 degrees Fahrenheit (that is obviously wrong for the same reasons stated above for October 2). For September 15 and October 1 for Neshaminy Falls the difference from the 2 nearby Trenton Mercer airport and South Jersey Regional airport weather stations was only 7-9 degrees Fahrenheit. Would it make "prudent" sense to make the test threshold even tighter or will this lead to more false positives/more workload/not feasible with budget constraints/etc. ? 6. Also in regards to (4) note as stated there were two days in a row (October 1 and October 2) that the TMIN values were obviously wrong for Neshaminy Falls yet that was not enough to cause either the spatial regression or the spatial corroboration check tests to fail for at least one of those days? 7. Finally note for South Jersey Regional airport the inaccurate TMIN of 28 degrees Fahrenheit on September 27 appropriately failed the spatial consistency test. (Neshaminy Falls and Trenton Mercer airport were 47 degrees and 50 degrees Fahrenheit respectively). However the South Jersey Regional inaccurate TMAX of 94 degrees Fahrenheit on July 23, 2019 DID NOT fail the spatial consistency test. It is an inaccurate value -- Neshaminy Falls And Trenton Mercer airport reported respectively 76 and 75 degrees Fahrenheit on that day (and thus the difference is at least 10 degrees Celsius for both cases). Notice too no hourly temperature values were reported for that entire day for this ASOS station so there must have been some issue going on: https://www7.ncdc.noaa.gov/CDO/cdoselect.cmd Regardless, since a TMAX of 94 was submitted why did it not fail the spatial consistency test? Quote Link to comment Share on other sites More sharing options...
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.