Sensor Data Stream Aggregation System

Date: Monday, August 12th
Time: 2:00-3:00 pm
The current increase in monitoring systems, supported by robust and
high-throughput communication links, has led to the creation of massive
point based high temporal-fidelity datasets. However, in many cases
the sensor observations are streamed through multiple fractured data
platforms. The individual platforms are aimed at various end-users
and are not uniformly compatible with various scientific applications,
posing formidable technical and logistical challenges. For example,
one of the largest publicly available monitoring systems is the expanding
set of surface weather observations. The weather data from the Greater
Chicago area is available in several forms including a standard enterprise
level upstream provider network (LDM), a series of XML HTTP request,
and standard ASCII streams. These data forms, their disseminating platforms
and communication mechanisms pose challenges due to the wide heterogeneity.
We have realized a framework for aggregating weather observations
from the Chicago metropolitan area. The framework aggregates weather
data from multiple sources and makes the aggregated data available
in NetCDF format through an OpenDAP portal. The current system outputs
one weather data point every 10 seconds for every available station.
The format of the NetCDF file was selected to follow accepted conventions
and allow additional processing using standard CF-1.X compliant analysis
software. The complete platform has been designed around open standards
and libraries.

In this talk I will describe the architecture of this platform and the
decision making process behind the major technical components. I will
briefly discuss the sources of data acquisition, the quality control
process and the OpenDAP portal that provides a unified assess method
to all of the resulting data products. A couple of example weather
products generated using the platform will be presented. The talk will end
with a brief discussion on the limitations of the platform and proposed
methods of improvement.