GP - Revision history

137.229.92.253: /* Overview */

2008-10-02T01:23:14Z

Overview

137.229.92.253: /* Fleshing out more thoughts */

2008-09-10T21:06:20Z

Fleshing out more thoughts

137.229.92.253: /* Functions called from the .kid templates /

2008-09-10T20:53:12Z

Functions called from the *.kid templates

137.229.92.253 at 20:40, 10 September 2008

2008-09-10T20:40:28Z

137.229.92.253 at 19:48, 10 September 2008

2008-09-10T19:48:13Z

137.229.92.253 at 19:47, 10 September 2008

2008-09-10T19:47:40Z

137.229.92.253 at 19:03, 10 September 2008

2008-09-10T19:03:12Z

209.112.196.223 at 03:05, 10 September 2008

2008-09-10T03:05:47Z

137.229.92.253 at 00:03, 4 September 2008

2008-09-04T00:03:21Z

137.229.92.253 at 23:31, 3 September 2008

2008-09-03T23:31:13Z

@@ Line 1: / Line 1: @@
 ==Overview==
-This is the first utility program given to us by the programmers at EE Internet.  Turns out this program just does the html pages and not the graphs.  So, in some ways it is quite similar to the program Crane wrote except without the figure generating capability.
+This is the first utility program given to us by the programmers at EE Internet.  Turns out this program just does the html pages and not the graphs.  So, in some ways it is quite similar to the perl program rawdata2html.
 ==Required Packages==

@@ Line 145: / Line 145: @@
 # the .ini, which controls general directory locations, stations to process, station groupings etc.
 # stationinfo.csv which contains station specific information (including data input file names)
-# the .dat files (listed in stationinfo.csv)
+# the .dat files, station specific (listed in stationinfo.csv under the column ''DataFileSuffixes'')
-# the html templates (''*.kid'').  The html templates are viewable in a web browser they just have variable anchors in place of real numbers like:
+# the html templates (''*.kid'').  The html templates are viewable in a web browser they just have variable anchors in place of real numbers like discussed above and are turned into regular html using the python package kid.  The file names are hard coded into the program (also listed above):
-<pre>
+## stationDiag.kid
-General Station Info:
+## stationInclude.kid
-${si['StationDescriptiveName']}
+## stationPage.kid
-si['timeDiff']
+# another key file that basically contains aliases between variables common to each station which may have different names in the specific .dat  file (like different ways of identifying air temperature for example).  This file is also specified in stationinfo.csv under the column ''TemplateMap''.  File format is mentioned above.
-${format(0, ms2mph(lr.windSpeedAvg))}
+So, changing the way plots are displayed as Emily talked about probably involves:
-${format(0, lr.windDirection)}
+# creating a simple page with the figure, caption and a back navigation button
-${lr.timeStamp}
+# modifiying stationInclude.kid so that the plot link goes to the page rather than the image.
-and quite a few more
+Doing some early QA/QC stuff is even easier.  Just write a new script (or point to datasets QA/QCed from the O.O. based file system Ken's been working on) that does the qaqc on the data and puts it all into a new file that has a first level of processing done to it.  No need to change this program at all.  The things that would change would be the the stationinfoCSV file with the general gp.py framework remaining in place. <br>
-</pre>
+Additionally, to get older cr10x data to work with this system wouldn't be a giant deal.  The julian date & time would need to be changed into a regular format as the table based is and the arrays (met/soil/snow/precip events) would each need to go into their own data file but otherwise, not to much required.

@@ Line 137: / Line 137: @@
 * mb2hg = mb2hg --> a function that takes pressure in millibars and converts to inches of mercury (e.g. mb2hg(lr.pressure))
 * format = format --> a function that sets the numerical format.  Can be used alone or with another function embedded (e.g. format(0, ms2mph(lr.windSpeedAvg) which displays the average winspeed in mile per hour after converting it from meter per second.  There are 0 decimal places shown for the value. Or, format(1, lr.solarPanelVoltageAvg) which displays the solarPanelVoltage with 0.1 precision)
 ==Fleshing out more thoughts==

@@ Line 4: / Line 4: @@
 ==Required Packages==
 To run it you'll need to install two extra python packages:
-* Kid -- A simple and pythonic XML template language, the Kid package is the one that reads through the template pages looking for sensor variables and plugs values in to those locations.  Look at stationInclude.kid for example in a web browser to get a taste for the way Kid does things.
+* '''Kid''' -- A simple and pythonic XML template language, the Kid package is the one that reads through the template pages looking for sensor variables and plugs values in to those locations.  Look at stationInclude.kid for example in a web browser to get a taste for the way Kid does things.
-* Pytz -- A timezone and time helper package
+* '''Pytz''' -- A timezone and time helper package
-* ConfigParser -- A configuration file reader (reads the .ini file and places the values found in it into the equivalent of read only perl hash).  To use it there is a first order sectional variable such as ''[Main]'' and then kind of subvariables in the section such as ''configDir''
+* '''ConfigParser''' -- A configuration file reader (reads the .ini file and places the values found in it into the equivalent of read only perl hash).  To use it there is a first order sectional variable such as ''[Main]'' and then kind of subvariables in the section such as ''configDir''
 ==Hardcoded Filenames==
 These should be located in the configuration directory (variable ''configDir'' in the .ini file):
@@ Line 12: / Line 12: @@
 * '''''stationInclude.kid''''' (case sensitive, this is the html template for the center frame on the current website layout)
 * '''''stationPage.kid''''' (case sensitive, this is a currently unused - at least on the public werc site- alternate html template)
-* '''''TemplateMap-dot.profile''''' (The contents of this file are shown below, maps loggernet data file variables to the variable names found in the template pages)
 ==Program Execution and Background==
@@ Line 27: / Line 27: @@
   /work/python/gws/gp/ADOT/plots
-In addition, for the first run there is some code at the end of gp.py that tries to send an email.  I commented that section out (lines 314 to 358 in revision 143 of gp.py).
+In addition, for the first run there is some code at the end of gp.py that tries to send an email.  I commented that section out (lines 314 to 358 in revision 143 of gp.py) for the understanding how things work part of the process.
 == Sample .ini file==
-Here's a working initialization file (''dot.ini'' from the EEI server).
+Here's a sample working initialization file (''dot.ini'' from the EEI server).
 <pre>
 [Main]
 networkName = ADOT
 # Where the configuration files lie
@@ Line 122: / Line 123: @@
 Soil_Moisture_3 = SoilMoistureThree, 40 cm
 </pre>
-In this template file, Column 1 is found in header at the top of the file downloaded using LoggerNet.  Column 2 is the corresponding variable name found in the template files: ''stationDiag.kid'', ''stationInclude.kid'', ''stationPage.kid''
+This file name isn't hard coded into the program but nearly so. Format is: ''TemplateMap-stationInfoCSV.TemplateMap.profile'' where ''stationInfoCSV.TemplateMap'' in the previous string is a value from the stationInfoCSV file.  Default case seems to be ''dot'' but other ones on the subversion server appear to be ''dot-NoAT'', ''dot-noSoil'', and ''dot-snow'' ... or create your own.<br>
 ==Fleshing out more thoughts==

@@ Line 9: / Line 9: @@
 ==Hardcoded Filenames==
 These should be located in the configuration directory (variable ''configDir'' in the .ini file):
-* ''''stationDiag.kid'''' (case sensitive, this is the html template for the diagnostic page)
+* '''''stationDiag.kid''''' (case sensitive, this is the html template for the diagnostic page)
-* ''''stationInclude.kid'''' (case sensitive, this is the html template for the center frame on the current website layout)
+* '''''stationInclude.kid''''' (case sensitive, this is the html template for the center frame on the current website layout)
-* ''''stationPage.kid'''' (case sensitive, this is a currently unused - at least on the public werc site- alternate html template)
+* '''''stationPage.kid''''' (case sensitive, this is a currently unused - at least on the public werc site- alternate html template)
-* ''''TemplateMap-dot.profile'''' (The contents of this file are shown below, maps loggernet data file variables to the variable names found in the template pages)
+* '''''TemplateMap-dot.profile''''' (The contents of this file are shown below, maps loggernet data file variables to the variable names found in the template pages)
 ==Program Execution and Background==

@@ Line 1: / Line 1: @@
-This is the first utility program given to us by the programmers at EE Internet.  Turns out this program just does the html pages and not the graphs.  To run it you'll need to install two extra python packages:  Kid and Pytz.  Instruction for that are on the [[python]] page.  To run gp from the command line you give python the input script name and also the name/location of an initialization file:
+==Overview==
   $ python gp.py ADOT/dot.ini
@@ Line 8: / Line 18: @@
 This is where all of the source data files from loggernet are contained.  There is no directory structure here, everything is lumped into one directory.
   /work/python/gws/gp/ADOT/data
-It appears this directory, although listed in the configuration file, is not currently used by gp.py.
+It appears this directory, although listed in the configuration file, is not currently used by gp.py... but it may be used by [[Graphing | graph.py]]
   /work/python/gws/gp/ADOT/plots
 In addition, for the first run there is some code at the end of gp.py that tries to send an email.  I commented that section out (lines 314 to 358 in revision 143 of gp.py).
+== Sample .ini file==
-Here's a working initialization file.
+Here's a working initialization file (''dot.ini'' from the EEI server).
 <pre>
 [Main]
@@ Line 51: / Line 61: @@
 </pre>
+== Sample StationInfo.csv File==
-A bit more on this gp business.  It appears to contain main() for this utility but the bulk of the work is done in gputil.py which contains object definitions for the system.  Everything is pretty much done through the objects.  The sneaky catch is that there's a filename hardcoded into the object, ''StationInfo.csv''.  ''StationInfo.csv'' basically contains the metadata for all these sites in csv format (header on the first line).  Cracks me up a bit since they're also using the kid module which I thought was for flexible XML use but whatever I guess.  So, you can get the feeling from reading the ADOT ''StationInfo.csv'' file below that all of this info gets used in various ways in the junk outputted to the html pages:
+So, you can get the feeling from reading the ADOT ''StationInfo.csv'' file below that all of this info gets used in various ways in the junk outputted to the html pages.  Also, line 1 is a list of the keys.  Row 2 Column 1 is another key and then key values march across the columns to the right like a multi-dimensional hash:
 <pre>
 StationName,PakbusID,StationDescriptiveName,StationLocationDescription,Latitude,Longitude,MagneticDeclination(East),ElevationFt,ElevationM,DataFileDirectory,DataFileNameBase,DataFileSuffixes,OutputDir,TemplateMap
@@ Line 78: / Line 88: @@
 </pre>
 <br>
-September 9 after work update<br>
 Will continue fleshing this out but had some more thoughts to add.
 So, there are generally five parts of input:
@@ Line 84: / Line 124: @@
 # stationinfo.csv which contains station specific information (including data input file names)
 # the .dat files (listed in stationinfo.csv)
-# the html templates.  The html templates are turned into regular html using the python package kid.
+# the html templates (''*.kid'').  The html templates are viewable in a web browser they just have variable anchors in place of real numbers like:
 # another key file that basically contains aliases between variables common to each station which may have different names in the specific .dat  file (like different ways of identifying air temperature for example).

← Older revision		Revision as of 03:05, 10 September 2008
Line 77:		Line 77:
	DFR3,613,Shell Pingo Rep,Shell Pingo Top,N 70 1.234,W 147 40.903,23.8,408,124.44,,DFR3_,"HrlyAtms.dat,HrlyDiag.dat",DFR3,dot		DFR3,613,Shell Pingo Rep,Shell Pingo Top,N 70 1.234,W 147 40.903,23.8,408,124.44,,DFR3_,"HrlyAtms.dat,HrlyDiag.dat",DFR3,dot
	</pre>		</pre>
		+	<br>
		+	September 9 after work update<br>
		+	Will continue fleshing this out but had some more thoughts to add.
		+	So, there are generally five parts of input:
		+	# the .ini, which controls general directory locations, stations to process, station groupings etc.
		+	# stationinfo.csv which contains station specific information (including data input file names)
		+	# the .dat files (listed in stationinfo.csv)
		+	# the html templates. The html templates are turned into regular html using the python package kid.
		+	# another key file that basically contains aliases between variables common to each station which may have different names in the specific .dat file (like different ways of identifying air temperature for example).
		+
		+	So, changing the way plots are displays probably involves copying and then modifiying one of html templates. This could also allow annotations to be included in a separate include for example. Doing some early QA/QC stuff is even easier. Just write a new script (or point to datasets QA/QCed from the O.O. based file system Ken's been working on) that does the qaqc on the data and puts it all into a new file that has a first level of processing done to it. No need to change this program at all. The things that would change would be the .ini and stationinfo file with the general gp.py framework remaining in place. <br>
		+	So, for html changes (changing how figures are shown and adding annotation text) there will be at least a minimal amount of coding required to modify gp.py or gp_util.py but for data changes that can all be done through the .ini files.<br>
		+	I guess additionally, to get older cr10x data to work with this system wouldn't be a giant deal. The julian date & time would need to be changed into a regular format as the table based is and the arrays would each need to go into their own data file but otherwise, not to much required.

@@ Line 49: / Line 49: @@
 baseStationName = Kuparuk Foothills Project
 stationOrder = DFM1,DFM2,DFM3,DFM4,DFM5,DFR1,DFR2,DFR3
 </pre>