General Weather Data Acquisition

This page will detail the global data acquisition steps.

Purpose:

The purpose of the global data acquisition is to provide regional context to the localized measurements. To acomplish this possible large perspective various available data souces will be used. A key objective is to obtain a minimized lag between remotely managed sites and locally managed sites. The further push to real time unified observations will increase the value of the resulting products for individuals on the ground.

Data Sources, Providers and Products

  • Meteorological Assimilation Data Ingest System (MADIS)
    The MADIS system is the NOAA portal for acquiring weather data products. Products that are available are listed on this list of feed details.
    Typical delay is 5 minutes to 1 hour for stations to report in mainly due to 5 minute batch compression. Additionally it is noted the most complete data set is typically 2 hours delay. 

    • The Unidata Local Data Manager (LDM)
      Initially streaming weather occured through satellite via a expensive downlink from the weather services. With the high speeds available on the internet a expensive satelite solution was no longer required. The available datafeeds were routed into a relay network of servers that work on a simple upstream-downstream data flow relationship. In order to join the network a initial primary feed provider is needed and a idea of what products are of interest.
  • Unidata – University Corporation for Atmospheric Research
  • Unidata is a NSF funded LDM data source. Unidata provides a non-compressed stream of the metar stations. The update frequency is faster than MADIS but the number of stations reported is considerably less.

    Example of the Global LDM Network
  • APRSWX – APRS2.NET
    Automatic Packet Reporting System is a long running decentralized amateur radio data flow system. Communication is point to multi-point at speeds up to 1200 baud on 144.390 Mhz. Due to the multi-point nature various civil operators provide receive and decoding services and publish the results on the internet for collaboration. Coverage in urban areas is considerably good, while rural areas tend to be pockets of coverage around towns. Additionaly civilian operators construct weather stations that feed data into the APRS. Citizen Weather Observation Program provides a framework and suggested methods of constructing a weather station.
  • Local Weather Observation Tower
    This resource will be able to provide sensor measurements on a 5 minute update rate.Example Data Flow Model 

    Example Flow of Incoming Measurement Data and Sources

Results

– Acquire the available data
– MADIS Compressed Surface Observations Data sets (Done)
– Unidata Surface Metar Stream (done)
– APRS (Done)
– Store the available data (PostgresSQL is setup and allows GIS queries for later filtering)
– Observe data lag and try to quantify data quality. The next figure shows the age of the data by the time each observation is inserted into the local database.

Example of LDM Server Startup

Data Volume and Lag for weather reports in northern Illinois

Station Map For 0830-0930 7/3/13

Plotting and validation:

Initial results presented large QC issues with inflow from the mesonet and CWOP. Many resources exist for input on QC values. I have taken the approach of not accepting any questionable data with a failed QC test. The mesonet data provides a QCD value for a QC summary. If the value is greater than 0 then the test value failed at least one QC test. http://www.unidata.ucar.edu/software/netcdf/examples/madis-mesonet.cdl has all of the binary breakdowns for each failure case.

In our area of observation I have found approximately 5 stations reporting unlikely measurements. These stations have been filtered at the injest/monitoring stage prior to the database insertion.

This previous plot presents the streamed data verses the public news data. The general shape and areas of high and low temperatures are consistent, this positive moving forward. The public image is updated every 15 minutes.