DataPro setup and workflow

Initial setup

It's good to settle on a standard directory structure if you can. For linux, most things go in a bin directory. You can do the same with windows. Recommended though to not put any spaces characters ' ' in folder names etc as I don't know if there are ill consequences. Be conservative, descriptive names, well organized, use underscore character in place of spaces. Logical places you might put them: /var/data/bin/ /home/site/bin/ c:\data\bin/

Then, suggested directory structure for the site's themselves: /var/data/$site_name - root directory of the site /var/data/$research_area/$site_name - another logical location for the root directory of the site. /var/data/$site_name/config/ - location of the configuration files used by datapro and related utilities for processing the data /var/data/$site_name/raw/ - location for the raw data (I personally think a good practice is to have the original data from LoggerNet / loggerdata etc /var/data/$site_name/outputs/ - location for the automatically processed data: raw data that has been analyzed by datapro /var/data/$site_name/web/ - location for a shorter time series, like the last 3 weeks suitable for a web page. /var/data/$site_name/qc/ - location for the automated qa/qc log /var/data/$site_name/error/ - location for datapro error logs

To run: python /var/data/bin/csv_utilities/datapro.py --key_file=/var/data/$site_name/config/keyfile.txt

Then, watch for error messages. Depending on your system there may be some python libraries that need to be installed. If you use windows, enthought python distribution is a nice all-encompassing choice or cygwin for making sure you have all of the right things on the computer.

If you have any questions just ask Bob Busey.