DataPro setup and workflow - Revision history

24.237.87.170: /* Version 2 of this workflow description */

2018-10-19T21:20:48Z

Version 2 of this workflow description

24.237.87.170: /* Initial setup */

2018-10-19T20:55:28Z

Initial setup

137.229.92.213: /* Initial setup */

2016-03-28T18:51:46Z

Initial setup

imported>Bob at 22:10, 13 November 2015

2015-11-13T22:10:01Z

Show changes

137.229.29.173 at 03:31, 4 November 2015

2015-11-04T03:31:43Z

137.229.29.173: Created page with "==Initial setup== It's good to settle on a standard directory structure if you can. For linux, most things go in a bin directory. You can do the same with windows. Recommen..."

2015-11-04T03:24:41Z

Created page with "==Initial setup== It's good to settle on a standard directory structure if you can. For linux, most things go in a bin directory. You can do the same with windows. Recommen..."

New page

==Initial setup==
It's good to settle on a standard directory structure if you can. For linux, most things go in a bin directory. You can do the same with windows. Recommended though to not put any spaces characters ' ' in folder names etc as I don't know if there are ill consequences. Be conservative, descriptive names, well organized, use underscore character in place of spaces. Logical places you might put them:
/var/data/bin/
/home/site/bin/
c:\data\bin/

Then, suggested directory structure for the site's themselves:
/var/data/$site_name - root directory of the site
/var/data/$research_area/$site_name - another logical location for the root directory of the site.
/var/data/$site_name/config/ - location of the configuration files used by datapro and related utilities for processing the data
/var/data/$site_name/raw/ - location for the raw data (I personally think a good practice is to have the original data from LoggerNet / loggerdata etc
/var/data/$site_name/outputs/ - location for the automatically processed data: raw data that has been analyzed by datapro
/var/data/$site_name/web/ - location for a shorter time series, like the last 3 weeks suitable for a web page.
/var/data/$site_name/qc/ - location for the automated qa/qc log
/var/data/$site_name/error/ - location for datapro error logs

To run:
python /var/data/bin/csv_utilities/datapro.py --key_file=/var/data/$site_name/config/keyfile.txt

Then, watch for error messages. Depending on your system there may be some python libraries that need to be installed. If you use windows, enthought python distribution is a nice all-encompassing choice or cygwin for making sure you have all of the right things on the computer.

If you have any questions just ask Bob Busey.

@@ Line 33: / Line 33: @@
 # once it's running I do a couple more things.  ln -s is used to create symlinks for a few other items.  I maintain an automatically generated diagnostics web page at http://ngeedata.iarc.uaf.edu/data/data.html.  It expects battery voltage to be placed in 'battery.csv' and Internal Panel Temperature to be in 'paneltemp.csv' if you don't use those names you can use ln -s to create a symlink. I then will also make sym links so that the processed data is available over the internet without making a second full copy of the data.
-==Version 2 of this workflow description ==
+==Version 2 of this workflow for fully automated sites ==
-Here is the code for DataPro plus some other information... I should figure out a good way to transfer the rest of this to the wiki at some point, too:<br>
+Here is the code for DataPro plus some other information...:<br>
 [[DataPro_and_related_utilities]]
 All of this definitely works in linux / mac osx.  I think datapro runs in Windows but a lot of the helper utilities and stuff are more linux / osx specific just due to that being where all of this stuff is running. Probably wouldn't be too hard to get it running in windows if you wanted to.  I have some other utilities, too.  Check out these in here:<br>
@@ Line 45: / Line 50: @@
 That loads it up.  If you look at main_cron (which is a text file) you'll see line 16 is the schedule for running bash script 'cron_04min.sh'.
-<pre>6      *      *    *     * (/home/bbusey/working_files/bin/cron_04min.sh) > /home/bbusey/working_files/bin/logs/ND_4min_log.txt</pre>
+<pre>6      *      *    *     * (/var/site/bin/cron_04min.sh) > /var/site/logs/ND_4min_log.txt</pre>
 If you open 'cron_04min.sh' you'll see a few things.  I use the information on lines 2 & 3 to track how long it takes for this script to run:

← Older revision		Revision as of 03:31, 4 November 2015
Line 22:		Line 22:

	If you have any questions just ask Bob Busey.		If you have any questions just ask Bob Busey.
		+
		+	===Initial site set up workflow===
		+
		+	# create the initial directory structure listed above.
		+	# I usually don't use the site creator though I certainly could. More often I start with an existing site's configuration files... copy them to a new config and edit to reflect the new site.
		+	# the file that controls qa/qc parameters, filenames, any functions to be applied etc is the parameter csv file. It's often easiest to just edit this on a computer with a graphical spreadsheet program like Excel or Libre Office to get things proper. The header information can often be copied & pasted straight from the raw .dat file
		+	# next, edit the .txt key file to reflect where file and directory locations are all placed.
		+	# run datapro as shown above.
		+	# if it works, awesome. if it doesn't, make note of the errors and try to address them.
		+	# once it's running I do a couple more things. ln -s is used to create symlinks for a few other items. I maintain an automatically generated diagnostics web page at http://ngeedata.iarc.uaf.edu/data/data.html. It expects battery voltage to be placed in 'battery.csv' and Internal Panel Temperature to be in 'paneltemp.csv' if you don't use those names you can use ln -s to create a symlink. I then will also make sym links so that the processed data is available over the internet without making a second full copy of the data.

@@ Line 1: / Line 1: @@
 ==Initial setup==
 It's good to settle on a standard directory structure if you can.  For linux, most things go in a bin directory.  You can do the same with windows.  Recommended though to not put any spaces characters ' ' in folder names etc as I don't know if there are ill consequences.  Be conservative, descriptive names, well organized, use underscore character in place of spaces.  Logical places you might put them:
-  /var/data/bin/
+  /var/site/bin/
   /home/site/bin/
   c:\data\bin/
@@ Line 7: / Line 7: @@
 Then, suggested directory structure for the site's themselves:
-  /var/data/$site_name - root directory of the site
+  /var/site/$site_name - root directory of the site
-  /var/data/$research_area/$site_name - another logical location for the root directory of the site.
+  /var/site/$research_area/$site_name - another logical location for the root directory of the site.
-  /var/data/$site_name/config/ - location of the configuration files used by datapro and related utilities for processing the data
+  /var/site/$site_name/config/ - location of the configuration files used by datapro and related utilities for processing the data
-  /var/data/$site_name/raw/ - location for the raw data (I personally think a good practice is to have the original data from LoggerNet / loggerdata etc
+  /var/site/$site_name/raw/ - location for the raw data (I personally think a good practice is to have the original data from LoggerNet / loggerdata etc
-  /var/data/$site_name/outputs/ - location for the automatically processed data: raw data that has been analyzed by datapro
+  /var/site/$site_name/outputs/ - location for the automatically processed data: raw data that has been analyzed by datapro
-  /var/data/$site_name/web/ - location for a shorter time series, like the last 3 weeks suitable for a web page.
+  /var/site/$site_name/web/ - location for a shorter time series, like the last 3 weeks suitable for a web page.
-  /var/data/$site_name/qc/ - location for the automated qa/qc log
+  /var/site/$site_name/qc/ - location for the automated qa/qc log
-  /var/data/$site_name/error/ - location for datapro error logs
+  /var/site/$site_name/error/ - location for datapro error logs
 To run:
-  python /var/data/bin/csv_utilities/datapro.py --key_file=/var/data/$site_name/config/keyfile.txt
+  python /var/site/bin/csv_utilities/datapro.py --key_file=/var/site/$site_name/config/keyfile.txt
 Then, watch for error messages. Depending on your system there may be some python libraries that need to be installed.  If you use windows, enthought python distribution is a nice all-encompassing choice or cygwin for making sure you have all of the right things on the computer.