NCCOOS Trac Projects: Top | Web | Platforms | Processing | Viz | Sprints | Sandbox | (Wind)

Process Level 0 data to Level 1 (raw2proc)

Process Level 0 (raw form) to Level 1 (monthly netCDF), which generally involves parsing ascii or binary data, combining measured parameters to get specific derived parameters of interest, and perform some basic data quality checks. While standing up the framework to add new platforms and packages to the processing stream has been accomplished to arrive at Version 0.1 (raw2proc-v0.1), much remains to be done and holes to fill.

Pyhton Module Dependencies

  • numpy
  • pycdf
  • dateutil

Python Modules

  • raw2proc -- based on configuration files for platform and packages knows how and what to process all this fun data
  • ncutil -- create, update to netCDF files, and get data from them
  • procutil -- lots of time handling functions

Processes raw ascii- or binary-data from different NCCOOS sensors (ctd, adcp, waves-adcp, met) based on manual or automated operation. raw2proc manages what configuration files to use for the specified platform and month to process. Then based on specified package, it determines (via rigid directory structure) what raw files to look for to process. Finally, raw2proc loads parsing and output methods pre-determined in config files for specific package-type to handle processing for specific data formats. For example, adcp data has multiple data formats that we might want to process (log data, spec data, and binary data for both currents and waves). Each format has different information that dictates what can be output. raw2proc also organizes the data into monthly netCDF files in the Level1 dataset.

Rules on Platform Configuration Files

  • build config_tester() for config-file compliance to run under raw2proc
  • but until then, follow these rules: *

STILL TO DO

  • organize/brainstorm where to store configuration files (for now they are in the trunk)
  • ditto for proc-filetype modules

raw2proc

  • add command-line interface thru getopt
  • in find_raw() override looking in "yyyy_mm" rawdata subdirectories for multi-month deployments for adcp (e.g. _RDI_000.000, _RDI_001.000, _RDI_003.000 in one subdirectory for one deployment.)

ncutil

  • add get_last_sample_time() -- get last time-stamp in netCDF

procutil

  • use dateutil timezone functions to handle setting timezone in time field of data

test

  • Sara needs to learn more about test code and knows she should have been doing it all along
  • lots of testing needed