__color__,ticket,summary,component,milestone,type,owner,__style__,priority,created,_changetime,_description,_reporter 2,30,error parseing dates,raw2proc,raw2proc-1.0,defect,haines *,None,critical,1291930181,1291931620,We started having a problem with filt_datetime() and the regexp that YYYYMMDD or YYMMDD for dates of Nov and Dec 2010. The date string '2010-12' was parsed as (2020_ 10_ 12) not (2010_ 12_ 01). We should try using the parser provided by the python module dateutil-1.2 Try {{{>>from dateutil.parser import parse}}} {{{>>parse('2010_12'_ fuzzy=True)}}} {{{datetime.datetime(2010_ 12_ 1_ 0_ 0) }}},haines 3,24,verify salinity values from AVPs YSI 6600 V2,raw2proc,,defect,haines *,None,major,1222287439,1222289987,Data from the YSI 6600 V2 unit on the automated vertical profilers (AVPs) report lower salinity values than calculated salinity from cond_ depth and temp using UNESCO seawater function. We need to verify which is correct value. Profile Time: 23:30:00 Profile Date: 09/22/2008 Profile Depth: 362.0 cm Profile Location: Morgan Bay Serial No: 00016B55_ ID: Ford_SERDP 09/22/08 23:30:07 22.81 33.57 21.06 0.155 7.91 1.9 10.5 7.84 month_ day_ year_ hour_ min_ sec_ temp (deg. C)_ conductivity (mS/cm)_ salinity (ppt)_ depth (meters)_ pH_ turbidity (NTU)_ chlorophyll (micrograms per liter)_ DO (micrograms per liter) mS/cm == mmho/cm matlab salin= sw_salt(cond/sw_c3515_ wtemp_ press) >> salin= sw_salt(33.57/sw_c3515_ 22.81_0.155) salin = 22.0735 Compared to 21.06 reported from ysi. See [/dataproc/wiki/raw2proc/docs raw2proc page] for info on this component.,haines 3,26,transform height above bottom to depth from surface for ADCPs,raw2proc,,defect,haines,color: #777; background: #ddd; border-color: #ccc;,major,1222288519,1224272311,Currently both bogue and jpier are bottom mounted. z dimension in processed data is based on height above the bottom and should be transformed to depth from surface. ADCP bin height values are determined from the transducer height_ blanking above the transducer_ bin size and sin of the distance to a bin (since beam is at an 20 deg angle). This is necessary for correct surface masking (sharp reflection of ping at the surface). Requires nominal depth or better yet_ average depth (from deployment-long (or longer if relocated to same exact location) record measurement of depth by the pressure. This will go into the config-file under platform information. This will then be reflected in proc2latest output and formed correctly for ingestion into SECOORA commons. See [/dataproc/wiki/raw2proc/docs raw2proc page] for info on this component.,haines 3,27,Apply beam angle correction to height above transducer for all ADCPs,raw2proc,,defect,haines,color: #777; background: #ddd; border-color: #ccc;,major,1224075938,1224272500,The height above the transducers is computed along beam. The angle to nadir for true height above transducer face of the ADCP can be corrected by the cosine of theta where theta is the angle between nadir and beam direction (usually fixed at 20 deg). All proc_*_adcp.py modules to computer where surface is located above ADCP. Directly needed for computing height down from surface. Related Ticket ([ticket:26 Ticket 26]) * proc_rdi_logdata_adcp * proc_nortek_wps_adcp See [/dataproc/wiki/raw2proc/docs raw2proc page] for info on this component.,haines 3,28,Apply additional algorithm to help with ADCP surface masking,raw2proc,,defect,haines *,None,major,1224272751,1224272829,Using either error or echo determine best algorithm using other resources from the ADCP to insert NAN values in z-locations that contain or are above the surface. This is a follow-on to other tickets ([ticket:26 Ticket 26] and [ticket:27 Ticket 27]).,haines 3,23,Implement udunits conversion in procutil,raw2proc,,task,haines,color: #777; background: #ddd; border-color: #ccc;,major,1221768796,1272391639,Implement udunits conversion in procutil. This will allow us to not have a gazillion conversion functions like meters2feet(meters) and millibar2inches_Hg(millibar). See [/dataproc/wiki/raw2proc/docs raw2proc page] for info on this component and module. ,haines 3,25,apply magnetic variation to wind data from AVPs,raw2proc,,task,haines *,None,major,1222287830,1223485579,Verify that u_v component and wind direction are NOT adjusted for magnetic variation at the AVP sites (Hampton_ Morgan_ and Stones). Then apply the correction from mvar value in proc_avp_ysi_6600_v2.py provided in each config-file for the platform location. This info is from Tony Whipple. Wind: Date_ time_ speed (m/s)_ direction (magnetic)_ compass direction (magnetic)_ North (m/s)_ East (m/s)_ n-samples See [/dataproc/wiki/raw2proc/docs raw2proc page] for info on this component.,haines 3,29,clean up procutil module,raw2proc,,task,haines *,None,major,1291755512,1291755545,There is no need for functions like meters2feet()_ etc now that udunits is implemented. Clean-up code and make appropriate changes to calls to old conversions throughout plotting and processing.,haines 3,31,change dim names to match var name in netCDF (ntime to time),raw2proc,raw2proc-1.0,defect,haines *,None,major,1340724268,1340724582,In each proc_*.py creator module_ change dimension names to match corresponding variable name. It might not make reading the file by humans easy_ but it will allow pydap to work correctly to grab data dependency. For example_ pydap cannot load d2.wtemp[:] because ntime dimension does not match time var. pydap is looking for the dependency based on dimension values. {{{ >>> d2= pydap.open_url('http://whewell.marine.unc.edu/dods/nccoos/level1/b2/ctd1/b2_ctd1_2011_11.nc') >>> d2.wtemp {'wtemp': _ 'ntime': } }}} {{{ >>> d2.wtemp[:] Traceback (most recent call last): File ""_ line 1_ in File "/opt/env/haines/dataproc/lib/python2.6/site-packages/pydap/model.py"_ line 662_ in __getitem__ var.data = var.data[slice_] File "/opt/env/haines/dataproc/lib/python2.6/site-packages/pydap/proxy.py"_ line 115_ in __getitem__ resp_ data = request(url) File "/opt/env/haines/dataproc/lib/python2.6/site-packages/pydap/util/http.py"_ line 50_ in request raise ServerError(msg) pydap.exceptions.ServerError: 'Server error 42: "\'ntime\'"' }}} However_ we can still read the data via pydap_ but have to know to get time separately_ e.g. {{{ >>> d2.wtemp.array[:] array([ 23.10750008_ 23.14500046_ 23.38870049_ ..._ 22.0807991 _ 22.06220055_ 22.07489967]_ dtype=float32) >>> d2.time.array[:] array([1320121566_ 1320132726_ 1320137406_ ..._ 1322696548_ 1322696908_ 1322697268]) }}} So change the following in creator(): {{{ # dimension names use tuple so order of initialization is maintained dim_inits = ( ('ntime'_ NC.UNLIMITED)_ ('nlat'_ 1)_ ('nlon'_ 1)_ ('nz'_ 1)_ ) }}} to {{{ # dimension names use tuple so order of initialization is maintained dim_inits = ( ('time'_ NC.UNLIMITED)_ ('lat'_ 1)_ ('lon'_ 1)_ ('z'_ 1)_ ) }}} and any ntime to time (or nlat to lat or nlon to lon or nz to z_ etc) {{{ # (varName_ varType_ (dimName1_ [dimName2]_ var_inits = ( # coordinate variables ('time'_ NC.INT_ ('time'_))_ ('lat'_ NC.FLOAT_ ('lat'_))_ ('lon'_ NC.FLOAT_ ('lon'_))_ ('z'_ NC.FLOAT_ ('z'_))_ # data variables ('wtemp'_ NC.FLOAT_ ('time'_))_ ('cond'_ NC.FLOAT_ ('time'_))_ ('press'_ NC.FLOAT_ ('time'_))_ # derived variables ('depth'_ NC.FLOAT_ ('time'_))_ ('salin'_ NC.FLOAT_ ('time'_))_ ('density'_ NC.FLOAT_ ('time'_))_ ) }}} ,haines