IN53A-1179
Web Service Access and Display of USGS Oceanographic Time-Series Data Using the NOAA Earth Research Division's Data Access Program (ERDDAP)
The sediment transport group of the U.S. Geologic Survey Coastal Marine Geology Program (USGS CMGP)
maintains an archive of more than 4400 NetCDF files collected over the last 30 years (Montgomery et al,
2007). The conventions used in these NetCDF files were determined long before the emerging standard
Climate and Forecast (CF) conventions for NetCDF, and web access has been traditionally been limited to
simple downloading of the NetCDF files.
To take advantage of a growing suite of software that works with CF-compliant data, A combination of NcML
and the THREDDS Data Server were used to allow web services access of CF compliant data via the OGC
WCS service and OPeNDAP.
The primary users of these coastal oceanographic measurements are modelers who are facile with netCDF
files and URL references. Other users, however, may prefer to obtain the data in another format or perhaps
just plot a variable. To assist both groups of users, we have evaluated NOAA's Earth Research Division's
Data Access Program (ERDDAP) as a potential method of providing a more flexible and powerful interface to
the data. This versatile program is able to access data from a variety of web services, including OPeNDAP,
and then deliver the data using web services in a very wide variety of formats, from common image formats
such as PNG and JPG (pictures of plots), to NetCDF, Matlab, text and spreadsheet formats.
Installation and configuration of ERDDAP was straightforward. The software written in Java, and delivered as
a War file that runs on a standard Tomcat server. Configuration of the user interface and the dataset list is
controlled by XML files. The documentation is well written and much of the XML generation is handled by the
supplied autogen function that reads a netCDF file and generates XML based on the file attributes. We are
working on a Matlab program that will completely automate the process by interrogating our data holdings
and producing the completely formed XML.
Our initial assessment is that the ERDDAP server does indeed enable a more versatile way for users to
interact with our data. Allowing users to select specific variables, time ranges and plot or data formats is a
huge improvement.
References
Montgomery, E., Martini, M., Lightsom, F.L., and Butman, B., 2008, Documentation of the U.S. Geological
Survey Oceanographic Time Series Measurement Database: U.S. Geological Survey Open-File Report 2007-
1194 (http://woodshole.er.usgs.gov/pubs/of2007-1194/)
http://coast-
enviro.er.usgs.gov/erddap
IN53A-1180
Data Interoperability and Standards Within A Large International Remote Sensing Project
The Group for High Resolution Sea Surface Temperature (GHRSST) project is an international collaboration
which, in 2002, initiated a pilot project under the auspices of GODAE (Global Ocean Data Assimilation
Experiment) to address an emerging need for accurate high resolution satellite-based sea surface
temperature (SST) products for ocean modeling. The GHRSST project brings together international space
agencies, research institutes, universities, and government agencies to collectively address the scientific,
logistical and managerial challenges posed by creating the SST data products and services.
Currently, the project produces over 30 unique SST products from over 11 different satellite sensors at
varying spatial scales and processing levels on a daily basis for a variety of applications including ocean
modeling, weather forecasting, climate research and fisheries management. Commensurate with the large
data volumes and diversity of satellite, in situ and ancillary oceanographic data for the GHRSST, a significant
investment was made in its data management infrastructure. This has included task sharing between NASA
and NOAA on distribution and archiving, adherence to community standards with regard to data and
metadata protocols and interoperability, and use of contemporary distribution and data discovery
mechanisms. We will describe some of these components in detail, review some of the lessons learned and
give an overview of some of emerging protocols under consideration including the ISO 19115-2 metadata
format and the netCDF version 4 file format.
http://ghrsst.jpl.nasa.gov
IN53A-1181
Integrating Data Distribution and Data Assimilation Between the OOI CI and the NOAA DIF
The Ocean Observatories Initiative (OOI) is an NSF funded program to establish the ocean observing infrastructure of the 21st century benefiting research and education. It is currently approaching final design and promises to deliver cyber and physical observatory infrastructure components as well as substantial core instrumentation to study environmental processes of the ocean at various scales, from coastal shelf-slope exchange processes to the deep ocean. The OOI's data distribution network lies at the heart of its cyber- infrastructure, which enables a multitude of science and education applications, ranging from data analysis, to processing, visualization and ontology supported query and mediation. In addition, it fundamentally supports a class of applications exploiting the knowledge gained from analyzing observational data for objective-driven ocean observing applications, such as automatically triggered response to episodic environmental events and interactive instrument tasking and control. The U.S. Department of Commerce through NOAA operates the Integrated Ocean Observing System (IOOS) providing continuous data in various formats, rates and scales on open oceans and coastal waters to scientists, managers, businesses, governments, and the public to support research and inform decision-making. The NOAA IOOS program initiated development of the Data Integration Framework (DIF) to improve management and delivery of an initial subset of ocean observations with the expectation of achieving improvements in a select set of NOAA's decision-support tools. Both OOI and NOAA through DIF collaborate on an effort to integrate the data distribution, access and analysis needs of both programs. We present details and early findings from this collaboration; one part of it is the development of a demonstrator combining web-based user access to oceanographic data through ERDDAP, efficient science data distribution, and scalable, self-healing deployment in a cloud computing environment. ERDDAP is a web-based front-end application integrating oceanographic data sources of various formats, for instance CDF data files as aggregated through NcML or presented using a THREDDS server. The OOI-designed data distribution network provides global traffic management and computational load balancing for observatory resources; it makes use of the OpenDAP Data Access Protocol (DAP) for efficient canonical science data distribution over the network. A cloud computing strategy is the basis for scalable, self-healing organization of an observatory's computing and storage resources, independent of the physical location and technical implementation of these resources.
IN53A-1182
OOI CyberInfrastructure - Next Generation Oceanographic Research
Software has become a key enabling technology for scientific discovery, observation, modeling, and exploitation of natural phenomena. New value emerges from the integration of individual subsystems into networked federations of capabilities exposed to the scientific community. Such data-intensive interoperability networks are crucial for future scientific collaborative research, as they open up new ways of fusing data from different sources and across various domains, and analysis on wide geographic areas. The recently established NSF OOI program, through its CyberInfrastructure component addresses this challenge by providing broad access from sensor networks for data acquisition up to computational grids for massive computations and binding infrastructure facilitating policy management and governance of the emerging system-of-scientific-systems. We provide insight into the integration core of this effort, namely, a hierarchic service-oriented architecture for a robust, performant, and maintainable implementation. We first discuss the relationship between data management and CI crosscutting concerns such as identity management, policy and governance, which define the organizational contexts for data access and usage. Next, we detail critical services including data ingestion, transformation, preservation, inventory, and presentation. To address interoperability issues between data represented in various formats we employ a semantic framework derived from the Earth System Grid technology, a canonical representation for scientific data based on DAP/OPeNDAP, and related data publishers such as ERDDAP. Finally, we briefly present the underlying transport based on a messaging infrastructure over the AMQP protocol, and the preservation based on a distributed file system through SDSC iRODS.
IN53A-1183
Interoperable Data Access Services for NOAA IOOS
The Integrated Ocean Observing System (IOOS) is intended to enhance our ability to collect, deliver, and
use ocean information. The goal is to support research and decision-making by providing data on our open
oceans, coastal waters, and Great Lakes in the formats, rates, and scales required by scientists, managers,
businesses, governments, and the public. The US National Oceanic and Atmospheric Administration (NOAA)
is the lead agency for IOOS. NOAA's IOOS office supports the development of regional coastal observing
capability and promotes data management efforts to increase data accessibility. Geospatial web services
have been established at NOAA data providers including the National Data Buoy Center (NDBC), the Center
for Operational Oceanographic Products and Services (CO-OPS), and CoastWatch, and at regional data
provider sites. Services established include Open-source Project for a Network Data Access Protocol
(OpenDAP), Open Geospatial Consortium (OGC) Sensor Observation Service (SOS), and OGC Web
Coverage Service (WCS). These services provide integrated access to data holdings that have been
aggregated at each center from multiple sources. We wish to collaborate with other groups to improve our
service offerings to maximize interoperability and enhance cross-provider data integration, and to share
common service components such as registries, catalogs, data conversion, and gateways. This paper will
discuss the current status of NOAA's IOOS efforts and possible next steps.
http://ioos.noaa.gov/
IN53A-1184
From Many to Many More: Instant Interoperability Through the Integrated Ocean Observing System Data Assembly Center
As the Integrated Ocean Observing System (IOOS) Data Assembly Center (DAC), NOAA's National Data
Buoy Center (NDBC) collects data from many ocean observing systems, quality controls the data, and
distributes them nationally and internationally. The DAC capabilities provide instant interoperability of any
ocean observatory with the national and international agencies responsible for critical forecasts and warnings
and with the national media. This interoperability is an important milestone in an observing system's
designation as an operational system.
Data collection begins with NDBC's own observing systems – Meteorological and Oceanographic Buoys and
Coastal Stations, the Tropical Atmosphere Ocean Array, and the NOAA tsunameter network. Leveraging the
data management functions that support NDBC systems, the DAC can support data partners including ocean
observations from IOOS Regional Observing Systems, the meteorological observations from the National
Water Level Observing Network, meteorological and oceanographic observations from the National
Estuarine Research Reserve System, Integrated Coral Observing Network, merchant ship observations from
the Voluntary Observing Ship program, and ocean current measurements from oil and gas platforms in the
Gulf of Mexico and from Coastal HF Radars.
The DAC monitors and quality controls IOOS Partner data alerting the data provider to outages and quality
discrepancies. After performing automated and manual quality control procedures, the DAC prepares the
observations for distribution. The primary means of data distribution is in standard World Meteorological
Organization alphanumeric coded messages distributed via the Global Telecommunications System,
NOAAPort, and Family of Services. Observing systems provide their data via ftp to an NDBC server using a
simple XML.
The DAC also posts data in real-time to the NDBC webpages in columnar text format and data plots that
maritime interests (e.g., surfing, fishing, boating) widely use. The webpage text feeds the Dial-A-Buoy
capability that reads the latest data from webpages and the latest NWS forecast for the station to a user via
telephone. The DAC also operates a DODS/OPenDAP server to provide data in netCDF. Recently the DAC
implemented the NOAA IOOS Data Integration Framework, which facilitates the exchange of data between
IOOS Regional Observing Systems by standardizing data exchange formats and incorporating needed
metadata for the correct application of the data. The DAC has become an OceanSITES Global Data
Assembly Center - part of the Initial Global Observing System for Climate.
Supported by the NOAA IOOS Program, the DAC provides round-the-clock monitoring, quality control, and
data distribution to ensure that its IOOS Partners can conduct operations that meet the NOAA definition of:
Sustained, systematic, reliable, and robust mission activities with an institutional commitment to deliver
appropriate, cost-effective products and services.
http://www.ndbc.noaa.gov/ioos.shtml
IN53A-1185
Semantic Sensor Observation Networks in a Billion-Sensor World
In 2010, there will be 10,000 telemetric devices for every human in the planet (prediction by Ernest and
Young). Some of these devices will be collecting data from coastal phenomena. Some will be connected to
adaptive sampling systems, which allow observing a phenomenon, forecasting its advance, and triggering of
other numerical models, new missions or changes to the sampling frequency of other sensors. These highly
sophisticated autonomous and adaptive sensors will help improve the understating of coastal phenomena;
however, collaborative arrangements among communities need to happen to be able to interoperate in a
world of billions of sensors. Arrangements will allow discovery and sharing of sensor descriptions and
understanding and usage of observed data.
OOSTethys is an open source collaborative project that helps implement ocean observing system
components. Some of these components include sensor interfaces, catalogs of services, and semantic
mediators. The OOSTethys team seeks to speed up collaborative arrangements by studying the best
standards available, creating easy-to-adopt toolkits, and publishing guides that facilitate the implementation
of these components. The interaction of some observing system components, and lessons learned about
developing Semantic Sensor Networks using OGC Sensor Observation Services and ontologies, will be
discussed.
http://www.oostethys.org/
IN53A-1186
Semantic Integration for Marine Science Interoperability Using Web Technologies
The Marine Metadata Interoperability Project, MMI (http://marinemetadata.org) promotes the exchange, integration, and use of marine data through enhanced data publishing, discovery, documentation, and accessibility. A key effort is the definition of an Architectural Framework and Operational Concept for Semantic Interoperability (http://marinemetadata.org/sfc), which is complemented with the development of tools that realize critical use cases in semantic interoperability. In this presentation, we describe a set of such Semantic Web tools that allow performing important interoperability tasks, ranging from the creation of controlled vocabularies and the mapping of terms across multiple ontologies, to the online registration, storage, and search services needed to work with the ontologies (http://mmisw.org). This set of services uses Web standards and technologies, including Resource Description Framework (RDF), Web Ontology language (OWL), Web services, and toolkits for Rich Internet Application development. We will describe the following components: MMI Ontology Registry: The MMI Ontology Registry and Repository provides registry and storage services for ontologies. Entries in the registry are associated with projects defined by the registered users. Also, sophisticated search functions, for example according to metadata items and vocabulary terms, are provided. Client applications can submit search requests using the WC3 SPARQL Query Language for RDF. Voc2RDF: This component converts an ASCII comma-delimited set of terms and definitions into an RDF file. Voc2RDF facilitates the creation of controlled vocabularies by using a simple form-based user interface. Created vocabularies and their descriptive metadata can be submitted to the MMI Ontology Registry for versioning and community access. VINE: The Vocabulary Integration Environment component allows the user to map vocabulary terms across multiple ontologies. Various relationships can be established, for example exactMatch, narrowerThan, and subClassOf. VINE can compute inferred mappings based on the given associations. Attributes about each mapping, like comments and a confidence level, can also be included. VINE also supports registering and storing resulting mapping files in the Ontology Registry. The presentation will describe the application of semantic technologies in general, and our planned applications in particular, to solve data management problems in the marine and environmental sciences.
IN53A-1187
A Tale of Two Observing Systems: Interoperability in the World of Microsoft Windows
Louisiana Universities Marine Consortium's (LUMCON) and Dauphin Island Sea Lab's (DISL) Environmental Monitoring System provide a unified coastal ocean observing system. These two systems are mirrored to maintain autonomy while offering an integrated data sharing environment. Both systems collect data via Campbell Scientific Data loggers, store the data in Microsoft SQL servers, and disseminate the data in real- time on the World Wide Web via Microsoft Internet Information Servers and Active Server Pages (ASP). The utilization of Microsoft Windows technologies presented many challenges to these observing systems as open source tools for interoperability grow. The current open source tools often require the installation of additional software. In order to make data available through common standards formats, "home grown" software has been developed. One example of this is the development of software to generate xml files for transmission to the National Data Buoy Center (NDBC). OOSTethys partners develop, test and implement easy-to-use, open-source, OGC-compliant software., and have created a working prototype of networked, semantically interoperable, real-time data systems. Partnering with OOSTethys, we are developing a cookbook to implement OGC web services. The implementation will be written in ASP, will run in a Microsoft operating system environment, and will serve data via Sensor Observation Services (SOS). This cookbook will give observing systems running Microsoft Windows the tools to easily participate in the Open Geospatial Consortium (OGC) Oceans Interoperability Experiment (OCEANS IE).
IN53A-1188
Building an Interoperable Relational Database for the National Deep Submergence Facility (NDSF)
The National Deep Submergence Facility (NDSF) operates the Human Occupied Vehicle (HOV) Alvin, the Remote Operated Vehicle (ROV) Jason 2, and the Autonomous Underwater Vehicle (AUV) Sentry. Data acquired with these platforms is provided both to the science party on each expedition, and to the Woods Hole Oceanographic Institution (WHOI) Data Library. Although several data sets are inventoried online, and some data are accessible, there has not yet been a coordinated effort to construct an interoperable NDSF database that can serve data to other data systems. We present on progress made with the creation of an NDSF relational database maintained at WHOI and developed through collaborative efforts with the Marine Geoscience Data System (MGDS). While our initial efforts focus on standardizing vehicle metadata and developing techniques for digitally acquiring metadata at sea, the creation of a relational database backend provides opportunities for improved data access and interoperability. Constructing the database using PostgresSQL, coupled with spatial database extensions (PostGIS), and an online GIS backend (MapServer), enables a searchable database and Open Geospatial Consortium (OGC) compliant map based web services (Web Map Service and Web Feature Service) that can be used to provide data access pathways through several programmatic interfaces.
IN53A-1189
Making Data Available via the Biological and Chemical Oceanography Data Management Office – Implementation Details
The Biological and Chemical Oceanography Data Management Office (BCO-DMO) was created from the U.S.
Joint Global Ocean Flux Study (U.S. JGOFS) and the U.S. GLOBal ocean ECosystems dynamics (U.S.
GLOBEC) Data Management Offices. The BCO-DMO is a NSF funded project that provides support for
scientists funded by either the NSF's Biological or Chemical Oceanography Program Office to facilitate
making their projects' data publically accessible. To extend the domains of the U.S. JGOFS and U.S.
GLOBEC programs and to enable new capabilities, the BCO-DMO formalized our metadata collection efforts
and designed and created the BCO-DMO metadata database. This database, together with our new website
content (http://www.bco-dmo.org) and a geospatial interface based on the University of Minnesota's
MapServer software, currently provide access to information and data from nine science programs and their
associated 27 projects. This poster highlights some of the details of our system's design decisions that
support the data discoverability, access, display, download and interoperability features, and capabilities of
the BCO-DMO data interface.
Initial efforts to use existing metadata schemas were unsuccessful as they did not address our specific needs
or were overly generalized and therefore more complicated than necessary. The database design has
evolved over time as we have learned more about what information needs to be preserved to support
multiple interfaces and to enable machine-to-machine interoperability. Our latest enhancements include
database tables to store additional information about the field or variable names that further describe the
experimental, at sea, and historical data in order to support our new geospatial interface. Other features will
facilitate data interoperability, provide flexibility in supporting different input data formats, capture data
provenance information and allow creation of metadata records that are in compliance with community
adopted standards.
http://www.bco-dmo.org
IN53A-1190
Collating SOLAS (Surface-Ocean Lower-Atmosphere Study) data: beyond stewardship
SOLAS is an international programme with the specific aim of trying to understand the dynamics of climate-
relevant gases and particles both near and across the air-sea interface. To fully assess the significance of
such processes at a global level, it is necessary to evaluate and interpret all existing data as a whole and
from this generate flux datasets and climatologies.
This project (SOLAS Project Integration) has sought to work with the relevant communities to realise this aim.
Over the past 2 years, the SOLAS Project Integrator (Dr Tom Bell) has interacted with data managers, the
measurement community and the potential users of any end-product – the modelling community.
Although currently only part-way through the project, we will discuss the strategies and funding sources that
have been utilised so far in order to realise its goals. A specific challenge of this data type has been the
requirement for relatively novel approaches. In particular, we will focus on the creation of a SOLAS Meta-
database and Portal (using NASA's Global Change Master Directory) and the need for targeted products with
specific scientific value.
http://www.bodc.ac.uk/solas_integration/
IN53A-1191
Surface Ocean Carbon dioxide Atlas
For the past decades underway carbon dioxide data have been reported heterogeneously. Differences in derivations and missing essential parameters made a direct comparison of data difficult and sometimes impossible. Underway data from approximately 1800 cruises were compiled to a common data set while missing data was extracted from standard data sets (World Ocean Atlas 2005, NCEP/NCAR). The fugacity for carbon dioxide for air in equilibrium with seawater at sea surface temperature was calculated. The data set now contains more than 6 million quality controlled measurements all in the same format from 1968 - 2007 including detailed metadata. Besides the calculated fCO2 it contains the reported carbon dioxide parameters, salinity, sea surface temperature and other important reported or added parameters. About 6 million measurements are public and the rest of the data will go public as soon as being published. All scripts for converting data to a common format and calculations are publicly available. The transparency of our work is essential for assuring the best quality data product in the future since everybody can directly see the changes we made to the data. Regional groups are in charge for the secondary quality control. All data is available to those regional working groups via LAS and will be after made available. Due to the amount of a data was an interactive webpage developed that allows the working groups to visualize all data and data on a cruise level and edit accordingly. This also allows a detailed documentation of all changes made to the data which will be published along with the final data set.
IN53A-1192
NODC-i - National Infrastructure for Access to Oceanographic and Marine Data and Information in The Netherlands
This poster will present the Dutch NODC-i project, the national infrastructure for access to oceanographic and marine data and information in The Netherlands. The NODC-i project is run by the National Oceanographic Data Committee (NL-NODC) of The Netherlands. The NL-NODC is the national representative in the EU-SeaDataNet project. The NODC-i project is a technical project which will result in the Dutch node in the SeaDataNet infrastructure. The goals of the NODC-i project are therefore very similar to the goals of the EU-SeaDataNet project, albeit aimed at a national level interconnecting the data centres of the following Dutch institutes: RWS, KNMI, NIOZ, NIOO-CEME, TNO B&O.