Difference between revisions of "DataPro and related utilities"
(Created page with " ===Background=== This program and related utilities started out in 2006 or 2007 as a way to leverage the gp.py program for web page creation. Basically, as a way to allow we...") |
imported>Bob |
||
(3 intermediate revisions by one other user not shown) | |||
Line 1: | Line 1: | ||
===Background=== | ===Background=== | ||
− | This program and related utilities started out in 2006 or 2007 as a way to leverage the gp.py program for web page creation. Basically, as a way to allow web pages generated by that tool to display data coming from Array styled outputting data loggers (Campbell Scientific cr10x & cr23x etc). Along the way it turned into a more full featured data processing system written in python with a host of support utilities. Bob Busey wrote the original and then student (now staff member) Ross Spicer did a nice rewrite and organized it like a proper computer scientist might.<br | + | This program and related utilities started out in 2006 or 2007 as a way to leverage the gp.py program for web page creation. Basically, as a way to allow web pages generated by that tool to display data coming from Array styled outputting data loggers (Campbell Scientific cr10x & cr23x etc). Along the way it turned into a more full featured data processing system written in python with a host of support utilities. Bob Busey wrote the original and then student (now staff member) Ross Spicer did a nice rewrite and organized it like a proper computer scientist might.<br> |
===Where to find it=== | ===Where to find it=== | ||
The current release can be found on Ross' github page:<br> | The current release can be found on Ross' github page:<br> | ||
https://github.com/rwspicer/csv_utilities | https://github.com/rwspicer/csv_utilities | ||
− | (example starter files here:)< | + | (example starter files here:)<br> |
https://github.com/frankohanlon/DataPro_Sample_Files | https://github.com/frankohanlon/DataPro_Sample_Files | ||
+ | Finally, see a semi-real workflow for it here:<br> | ||
+ | '''[[DataPro setup and workflow]]''' | ||
+ | <br> | ||
There are several ways to get it. Easy way is just to download the package as a zip file. Proper way is to use open source tool Git (from command line or graphical interface):<br> | There are several ways to get it. Easy way is just to download the package as a zip file. Proper way is to use open source tool Git (from command line or graphical interface):<br> | ||
<pre>git clone https://github.com/frankohanlon/DataPro_Sample_Files</pre> | <pre>git clone https://github.com/frankohanlon/DataPro_Sample_Files</pre> | ||
Line 18: | Line 21: | ||
datapro.py -- Datapro 3.0 the data processing utility | datapro.py -- Datapro 3.0 the data processing utility | ||
step_function_utility.py -- applies a correction to data values over given | step_function_utility.py -- applies a correction to data values over given | ||
− | time periods | + | time periods. Helps with applying temperature offsets to soil temperature data. |
rh_calculator.py -- given data on precipitation and dew point caclualtes the | rh_calculator.py -- given data on precipitation and dew point caclualtes the | ||
relative humidity | relative humidity | ||
− | precip_utility.py -- | + | precip_utility.py -- an additional level of qa that gets applied to the summer precipitation data |
− | plotter.py -- | + | plotter.py -- a plotting utility for making png charts |
PNGen.sh -- generates the plots for each csv file in a directory | PNGen.sh -- generates the plots for each csv file in a directory | ||
get_ip -- a utility to get the ip adderess of a machine | get_ip -- a utility to get the ip adderess of a machine | ||
rad_decoder.py -- takes a cdf File and converts to csv file | rad_decoder.py -- takes a cdf File and converts to csv file | ||
+ | qc.py -- this one is handy. So, consider the situation where you find a bunch of bad data. You can use this utility to make your corrections in excel and then qc.py will copy those corrections back into the original output from datapro... and make note of what has changed in the qa log. | ||
tz_shift.py -- for shifting between utc - 0 and akst timezones | tz_shift.py -- for shifting between utc - 0 and akst timezones | ||
wunder_formatter.py -- takes csv file data and converts it to a url that can | wunder_formatter.py -- takes csv file data and converts it to a url that can | ||
Line 31: | Line 35: | ||
LoggerLink -- a porgram for uploading/ downloading data and programs from | LoggerLink -- a porgram for uploading/ downloading data and programs from | ||
cr1000 data loffers | cr1000 data loffers | ||
− | + | tutorial.py -- should show how to use all of these things. | |
− | barrow_monthly.py -- | + | Probably less interesting things: |
− | + | barrow_monthly.py -- something he wrote to grab Barrow CRN data | |
− | + | noaa_data.py -- another something he wrote to grab Barrow CRN data | |
+ | LoggerLink.py -- another way to send programs to pakbus data loggers (like CR1000 or CR800 etc etc) | ||
+ | rad_decoder.py -- take ARM netCDF files and extract the data into csv files. | ||
+ | |||
+ | </pre> | ||
+ | ===Known Bugs=== | ||
+ | * Until they are quashed, here are some known irritants: | ||
+ | ** if you perform a direct download from a site and run that data through datapro, you may see this mysterious error: | ||
+ | <pre> | ||
+ | set([]) | ||
+ | Array ID 228 has no records in data file. No data to process. | ||
</pre> | </pre> | ||
+ | This actually means the first line of the file may start with a comma, as is the case when downloading allll of the data from a completely full logger. If you look at the data file it will start something like: | ||
+ | <pre>,1800,14.15,62.44,156.5,164.7,0,0,0,23.29,.5,-.708,13.7,14.38</pre> | ||
+ | * think that will be fixed shortly. In the meantime, just delete the first line from the data file. |
Latest revision as of 09:49, 1 June 2016
Background
This program and related utilities started out in 2006 or 2007 as a way to leverage the gp.py program for web page creation. Basically, as a way to allow web pages generated by that tool to display data coming from Array styled outputting data loggers (Campbell Scientific cr10x & cr23x etc). Along the way it turned into a more full featured data processing system written in python with a host of support utilities. Bob Busey wrote the original and then student (now staff member) Ross Spicer did a nice rewrite and organized it like a proper computer scientist might.
Where to find it
The current release can be found on Ross' github page:
https://github.com/rwspicer/csv_utilities
(example starter files here:)
https://github.com/frankohanlon/DataPro_Sample_Files
Finally, see a semi-real workflow for it here:
DataPro setup and workflow
There are several ways to get it. Easy way is just to download the package as a zip file. Proper way is to use open source tool Git (from command line or graphical interface):
git clone https://github.com/frankohanlon/DataPro_Sample_Files
Also useful if you decide to use it is this is a windows / linux graphical utility which helps generate the configuration files:
https://github.com/rwspicer/sc_apps
What is included
datapro.py -- Datapro 3.0 the data processing utility step_function_utility.py -- applies a correction to data values over given time periods. Helps with applying temperature offsets to soil temperature data. rh_calculator.py -- given data on precipitation and dew point caclualtes the relative humidity precip_utility.py -- an additional level of qa that gets applied to the summer precipitation data plotter.py -- a plotting utility for making png charts PNGen.sh -- generates the plots for each csv file in a directory get_ip -- a utility to get the ip adderess of a machine rad_decoder.py -- takes a cdf File and converts to csv file qc.py -- this one is handy. So, consider the situation where you find a bunch of bad data. You can use this utility to make your corrections in excel and then qc.py will copy those corrections back into the original output from datapro... and make note of what has changed in the qa log. tz_shift.py -- for shifting between utc - 0 and akst timezones wunder_formatter.py -- takes csv file data and converts it to a url that can be sent to weather under ground LoggerLink -- a porgram for uploading/ downloading data and programs from cr1000 data loffers tutorial.py -- should show how to use all of these things. Probably less interesting things: barrow_monthly.py -- something he wrote to grab Barrow CRN data noaa_data.py -- another something he wrote to grab Barrow CRN data LoggerLink.py -- another way to send programs to pakbus data loggers (like CR1000 or CR800 etc etc) rad_decoder.py -- take ARM netCDF files and extract the data into csv files.
Known Bugs
- Until they are quashed, here are some known irritants:
- if you perform a direct download from a site and run that data through datapro, you may see this mysterious error:
set([]) Array ID 228 has no records in data file. No data to process.
This actually means the first line of the file may start with a comma, as is the case when downloading allll of the data from a completely full logger. If you look at the data file it will start something like:
,1800,14.15,62.44,156.5,164.7,0,0,0,23.29,.5,-.708,13.7,14.38
- think that will be fixed shortly. In the meantime, just delete the first line from the data file.