__color__ ticket summary component milestone type owner __style__ priority created _changetime _description _reporter 2 30 error parseing dates raw2proc raw2proc-1.0 defect haines * None critical 1291930181 1291931620 We started having a problem with filt_datetime() and the regexp that YYYYMMDD or YYMMDD for dates of Nov and Dec 2010. The date string '2010-12' was parsed as (2020, 10, 12) not (2010, 12, 01). We should try using the parser provided by the python module dateutil-1.2 Try {{{>>from dateutil.parser import parse}}} {{{>>parse('2010_12', fuzzy=True)}}} {{{datetime.datetime(2010, 12, 1, 0, 0) }}} haines 3 24 verify salinity values from AVPs YSI 6600 V2 raw2proc defect haines * None major 1222287439 1222289987 Data from the YSI 6600 V2 unit on the automated vertical profilers (AVPs) report lower salinity values than calculated salinity from cond, depth and temp using UNESCO seawater function. We need to verify which is correct value. Profile Time: 23:30:00 Profile Date: 09/22/2008 Profile Depth: 362.0 cm Profile Location: Morgan Bay Serial No: 00016B55, ID: Ford_SERDP 09/22/08 23:30:07 22.81 33.57 21.06 0.155 7.91 1.9 10.5 7.84 month, day, year, hour, min, sec, temp (deg. C), conductivity (mS/cm), salinity (ppt), depth (meters), pH, turbidity (NTU), chlorophyll (micrograms per liter), DO (micrograms per liter) mS/cm == mmho/cm matlab salin= sw_salt(cond/sw_c3515, wtemp, press) >> salin= sw_salt(33.57/sw_c3515, 22.81,0.155) salin = 22.0735 Compared to 21.06 reported from ysi. See [/dataproc/wiki/raw2proc/docs raw2proc page] for info on this component. haines 3 26 transform height above bottom to depth from surface for ADCPs raw2proc defect haines color: #777; background: #ddd; border-color: #ccc; major 1222288519 1224272311 Currently both bogue and jpier are bottom mounted. z dimension in processed data is based on height above the bottom and should be transformed to depth from surface. ADCP bin height values are determined from the transducer height, blanking above the transducer, bin size and sin of the distance to a bin (since beam is at an 20 deg angle). This is necessary for correct surface masking (sharp reflection of ping at the surface). Requires nominal depth or better yet, average depth (from deployment-long (or longer if relocated to same exact location) record measurement of depth by the pressure. This will go into the config-file under platform information. This will then be reflected in proc2latest output and formed correctly for ingestion into SECOORA commons. See [/dataproc/wiki/raw2proc/docs raw2proc page] for info on this component. haines 3 27 Apply beam angle correction to height above transducer for all ADCPs raw2proc defect haines color: #777; background: #ddd; border-color: #ccc; major 1224075938 1224272500 The height above the transducers is computed along beam. The angle to nadir for true height above transducer face of the ADCP can be corrected by the cosine of theta where theta is the angle between nadir and beam direction (usually fixed at 20 deg). All proc_*_adcp.py modules to computer where surface is located above ADCP. Directly needed for computing height down from surface. Related Ticket ([ticket:26 Ticket 26]) * proc_rdi_logdata_adcp * proc_nortek_wps_adcp See [/dataproc/wiki/raw2proc/docs raw2proc page] for info on this component. haines 3 28 Apply additional algorithm to help with ADCP surface masking raw2proc defect haines * None major 1224272751 1224272829 Using either error or echo determine best algorithm using other resources from the ADCP to insert NAN values in z-locations that contain or are above the surface. This is a follow-on to other tickets ([ticket:26 Ticket 26] and [ticket:27 Ticket 27]). haines 3 23 Implement udunits conversion in procutil raw2proc task haines color: #777; background: #ddd; border-color: #ccc; major 1221768796 1272391639 Implement udunits conversion in procutil. This will allow us to not have a gazillion conversion functions like meters2feet(meters) and millibar2inches_Hg(millibar). See [/dataproc/wiki/raw2proc/docs raw2proc page] for info on this component and module. haines 3 25 apply magnetic variation to wind data from AVPs raw2proc task haines * None major 1222287830 1223485579 Verify that u,v component and wind direction are NOT adjusted for magnetic variation at the AVP sites (Hampton, Morgan, and Stones). Then apply the correction from mvar value in proc_avp_ysi_6600_v2.py provided in each config-file for the platform location. This info is from Tony Whipple. Wind: Date, time, speed (m/s), direction (magnetic), compass direction (magnetic), North (m/s), East (m/s), n-samples See [/dataproc/wiki/raw2proc/docs raw2proc page] for info on this component. haines 3 29 clean up procutil module raw2proc task haines * None major 1291755512 1291755545 There is no need for functions like meters2feet(), etc now that udunits is implemented. Clean-up code and make appropriate changes to calls to old conversions throughout plotting and processing. haines 3 31 change dim names to match var name in netCDF (ntime to time) raw2proc raw2proc-1.0 defect haines * None major 1340724268 1340724582 In each proc_*.py creator module, change dimension names to match corresponding variable name. It might not make reading the file by humans easy, but it will allow pydap to work correctly to grab data dependency. For example, pydap cannot load d2.wtemp[:] because ntime dimension does not match time var. pydap is looking for the dependency based on dimension values. {{{ >>> d2= pydap.open_url('http://whewell.marine.unc.edu/dods/nccoos/level1/b2/ctd1/b2_ctd1_2011_11.nc') >>> d2.wtemp {'wtemp': , 'ntime': } }}} {{{ >>> d2.wtemp[:] Traceback (most recent call last): File "", line 1, in File "/opt/env/haines/dataproc/lib/python2.6/site-packages/pydap/model.py", line 662, in __getitem__ var.data = var.data[slice_] File "/opt/env/haines/dataproc/lib/python2.6/site-packages/pydap/proxy.py", line 115, in __getitem__ resp, data = request(url) File "/opt/env/haines/dataproc/lib/python2.6/site-packages/pydap/util/http.py", line 50, in request raise ServerError(msg) pydap.exceptions.ServerError: 'Server error 42: "\'ntime\'"' }}} However, we can still read the data via pydap, but have to know to get time separately, e.g. {{{ >>> d2.wtemp.array[:] array([ 23.10750008, 23.14500046, 23.38870049, ..., 22.0807991 , 22.06220055, 22.07489967], dtype=float32) >>> d2.time.array[:] array([1320121566, 1320132726, 1320137406, ..., 1322696548, 1322696908, 1322697268]) }}} So change the following in creator(): {{{ # dimension names use tuple so order of initialization is maintained dim_inits = ( ('ntime', NC.UNLIMITED), ('nlat', 1), ('nlon', 1), ('nz', 1), ) }}} to {{{ # dimension names use tuple so order of initialization is maintained dim_inits = ( ('time', NC.UNLIMITED), ('lat', 1), ('lon', 1), ('z', 1), ) }}} and any ntime to time (or nlat to lat or nlon to lon or nz to z, etc) {{{ # (varName, varType, (dimName1, [dimName2], var_inits = ( # coordinate variables ('time', NC.INT, ('time',)), ('lat', NC.FLOAT, ('lat',)), ('lon', NC.FLOAT, ('lon',)), ('z', NC.FLOAT, ('z',)), # data variables ('wtemp', NC.FLOAT, ('time',)), ('cond', NC.FLOAT, ('time',)), ('press', NC.FLOAT, ('time',)), # derived variables ('depth', NC.FLOAT, ('time',)), ('salin', NC.FLOAT, ('time',)), ('density', NC.FLOAT, ('time',)), ) }}} haines