Difference between revisions of "GP"
(9 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
− | This is the first utility program given to us by the programmers at EE Internet. Turns out this program just does the html pages and not the graphs. To run it you'll need to install two extra python packages: Kid and Pytz. Instruction for | + | ==Overview== |
+ | This is the first utility program given to us by the programmers at EE Internet. Turns out this program just does the html pages and not the graphs. So, in some ways it is quite similar to the perl program rawdata2html. | ||
+ | |||
+ | ==Required Packages== | ||
+ | To run it you'll need to install two extra python packages: | ||
+ | * '''Kid''' -- A simple and pythonic XML template language, the Kid package is the one that reads through the template pages looking for sensor variables and plugs values in to those locations. Look at stationInclude.kid for example in a web browser to get a taste for the way Kid does things. | ||
+ | * '''Pytz''' -- A timezone and time helper package | ||
+ | * '''ConfigParser''' -- A configuration file reader (reads the .ini file and places the values found in it into the equivalent of read only perl hash). To use it there is a first order sectional variable such as ''[Main]'' and then kind of subvariables in the section such as ''configDir'' | ||
+ | ==Hardcoded Filenames== | ||
+ | These should be located in the configuration directory (variable ''configDir'' in the .ini file): | ||
+ | * '''''stationDiag.kid''''' (case sensitive, this is the html template for the diagnostic page) | ||
+ | * '''''stationInclude.kid''''' (case sensitive, this is the html template for the center frame on the current website layout) | ||
+ | * '''''stationPage.kid''''' (case sensitive, this is a currently unused - at least on the public werc site- alternate html template) | ||
+ | |||
+ | |||
+ | ==Program Execution and Background== | ||
+ | Instruction for installation are on the [[python]] page. To run gp from the command line you give python the input script name and also the name/location of an initialization file: | ||
$ python gp.py ADOT/dot.ini | $ python gp.py ADOT/dot.ini | ||
Line 8: | Line 24: | ||
This is where all of the source data files from loggernet are contained. There is no directory structure here, everything is lumped into one directory. | This is where all of the source data files from loggernet are contained. There is no directory structure here, everything is lumped into one directory. | ||
/work/python/gws/gp/ADOT/data | /work/python/gws/gp/ADOT/data | ||
− | It appears this directory, although listed in the configuration file, is not currently used by gp.py. | + | It appears this directory, although listed in the configuration file, is not currently used by gp.py... but it may be used by [[Graphing | graph.py]] |
/work/python/gws/gp/ADOT/plots | /work/python/gws/gp/ADOT/plots | ||
− | In addition, for the first run there is some code at the end of gp.py that tries to send an email. I commented that section out (lines 314 to 358 in revision 143 of gp.py). | + | In addition, for the first run there is some code at the end of gp.py that tries to send an email. I commented that section out (lines 314 to 358 in revision 143 of gp.py) for the understanding how things work part of the process. |
− | + | == Sample .ini file== | |
− | Here's a working initialization file. | + | Here's a sample working initialization file (''dot.ini'' from the EEI server). |
<pre> | <pre> | ||
[Main] | [Main] | ||
+ | # networkName is used on the [http://www.eeinternet.com/dot/html/adot-diag.html|diagnostics page] | ||
networkName = ADOT | networkName = ADOT | ||
# Where the configuration files lie | # Where the configuration files lie | ||
Line 50: | Line 67: | ||
stationOrder = DFM1,DFM2,DFM3,DFM4,DFM5,DFR1,DFR2,DFR3 | stationOrder = DFM1,DFM2,DFM3,DFM4,DFM5,DFR1,DFR2,DFR3 | ||
</pre> | </pre> | ||
+ | |||
+ | == Sample StationInfo.csv File== | ||
+ | So, you can get the feeling from reading the ADOT ''StationInfo.csv'' file below that all of this info gets used in various ways in the junk outputted to the html pages. Also, line 1 is a list of the keys. Row 2 Column 1 is another key and then key values march across the columns to the right like a multi-dimensional hash: | ||
+ | <pre> | ||
+ | StationName,PakbusID,StationDescriptiveName,StationLocationDescription,Latitude,Longitude,MagneticDeclination(East),ElevationFt,ElevationM,DataFileDirectory,DataFileNameBase,DataFileSuffixes,OutputDir,TemplateMap | ||
+ | DBM1,201,Accomplishment Creek Met,"Accomplishment Ck, below lake at pass to Rubicon",N 68 24.696,W 148 8.190,22.6,4833,1474.065,,DBM1_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM1,dot | ||
+ | DBM2,202,Ribdon Met,Upper Rubicon,N 68 38.548,W 147 21.107,23.1,4648,1417.64,,DBM2_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM2,dot | ||
+ | DBM3,203,Juniper Met,Upper Juniper Ck,N 69 4.570,W 146 30.294,23.8,4324,1318.82,,DBM3_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM3,dot | ||
+ | DBM4,204,Sag-Ivishak Met,"Close to snow survey site UP1, aka FT1",N 69 12.933,W 148 33.116,22.9,1414,431.27,,DBM4_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM4,dot | ||
+ | DBM5,205,Upper Kad Met,Kadleroshilik uplands,N 69 32.968,W 147 56.505,23.5,686,209.23,,DBM5_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM5,dot | ||
+ | DBM6,206,Kavik Met,Kavik camp,N 69 40.402,W 146 54.034,24.1,649,197.945,,DBM6_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM6,dot | ||
+ | DBM7,207,Lower Kad Met,"Kadleroshilik River, near the old runway",N 70 4.406,W 147 39.000,23.8,78,23.79,,DBM7_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM7,dot | ||
+ | DBM8,208,Bullen Met,South of Bullen Pt,N 70 4.792,W 146 49.166,24.3,86,26.23,,DBM8_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM8,dot | ||
+ | DBR1,211,TBD,TBD,,,,,,,DBR1_,"",DBR1,dot | ||
+ | DBR2,212,Ribdon Rep,Somewhere around Lupine valley?,N 68 36.337,W 147 23.716,23.2,N/A,N/A,,DBR2_,"HrlyAtms.dat,HrlyDiag.dat",DBR2,dot | ||
+ | DBR3,213,Pogopuk Rep,Between Gilead Ck and Niviak Pass,N 69 17.369,W 146 22.990,23.3,N/A,N/A,,DBR3_,"HrlyAtms.dat,HrlyDiag.dat",DBR3,dot | ||
+ | DBR4,214,Kavik Rep,"South of Kavik Camp, next to winter trail",N 69 37.893,W 146 52.889,23.4,1436,437.98,,DBR4_,"HrlyAtms.dat,HrlyDiag.dat",DBR4,dot | ||
+ | DBR5,215,Franklin Bluffs Rep,"Franklin Bluffs, NE side",N 69 48.633,W 148 19.540,23.1,875,266.875,,DBR5_,"HrlyAtms.dat,HrlyDiag.dat",DBR5,dot | ||
+ | DFM1,601,South White Hills Met,WERC HV5 snowsurvey site,N 69 12.043,W 149 33.508,22.4,962,293.41,,DFM1_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DFM1,dot | ||
+ | DFM2,602,White Hills Met,WERC White Hills previous repeater site,N 69 29.187,W 149 49.284,22.5,,,,DFM2_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DFM2,dot | ||
+ | DFM3,603,North White Hills Met,Between lake and stream to NNE of White Hills,N 69 42.892,W 149 28.227,22.6,276,84.18,,DFM3_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DFM3,dot | ||
+ | DFM4,604,North West Kuparuk Met,WERC H03 snowsurvey site,N 69 56.851,W 149 55.014,22.7,406,123.83,,DFM4_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DFM4,dot | ||
+ | DFM5,605,TBD,TBD,,,,,,,DFM5_,"",DFM5,dot | ||
+ | DFR1,611,Kak Rep,Kakukturat Mtn,N 69 4.357,W 149 30.870,22.3,1667,508.435,,DFR1_,"HrlyAtms.dat,HrlyDiag.dat",DFR1,dot | ||
+ | DFR2,612,Slope Mnt Rep,Slope Mtn Repeater Site (relocated to top),N 68 44.448,W 149 1.989,,3858,1176.69,,DFR2_,"HrlyAtms.dat,HrlyDiag.dat",DFR2,dot | ||
+ | DFR3,613,Shell Pingo Rep,Shell Pingo Top,N 70 1.234,W 147 40.903,23.8,408,124.44,,DFR3_,"HrlyAtms.dat,HrlyDiag.dat",DFR3,dot | ||
+ | </pre> | ||
+ | <br> | ||
+ | |||
+ | == Sample Variable Mapping file == | ||
+ | The contents of ''TemplateMap-dot.profile'': | ||
+ | <pre> | ||
+ | # DATA_FILE_NAME = templateName | ||
+ | TIMESTAMP = timeStamp | ||
+ | WindSpeed_WVc(1) = windSpeedAvg, Avg Wind Speed | ||
+ | WindSpeed_WVc(2) = windDirection, Wind Direction | ||
+ | WindSpeed_Max = windSpeedMax, Max Wind Speed | ||
+ | batt_volt_Avg = batteryVoltage, Battery Voltage | ||
+ | SolarPnlV_Avg = solarPanelVoltageAvg, Avg Solar Panel Voltage | ||
+ | PanelT = panelTemp, Panel Temperature | ||
+ | CRNT_AT_Avg = airTempAvg, Avg Air Temperature | ||
+ | CRNT_RH_Avg = relativeHumidityAvg, Avg Relative Humidity | ||
+ | CRNT_DP_Avg = dewTempAvg, Avg Dew Point | ||
+ | Precip_Tot = totalPrecipitation, Total Precipitation | ||
+ | SnowDepth_Avg = snowDepthAvg, Avg Snow Depth | ||
+ | Therm_degC(1) = soilSurfaceTemp, Soil Surface Temperature | ||
+ | Therm_degC(4) = soilProfileSurface, 0 cm | ||
+ | Therm_degC(5) = soilProfileOne, 5 cm | ||
+ | Therm_degC(6) = soilProfileTwo, 10 cm | ||
+ | Therm_degC(7) = soilProfileThree, 15 cm | ||
+ | Therm_degC(8) = soilProfileFour, 20 cm | ||
+ | Therm_degC(9) = soilProfileFive, 40 cm | ||
+ | Soil_Moisture_1 = SoilMoistureOne, 5 cm | ||
+ | Soil_Moisture_2 = SoilMoistureTwo, 20 | ||
+ | Soil_Moisture_3 = SoilMoistureThree, 40 cm | ||
+ | </pre> | ||
+ | This file name isn't hard coded into the program but nearly so. Format is: ''TemplateMap-stationInfoCSV.TemplateMap.profile'' where ''stationInfoCSV.TemplateMap'' in the previous string is a value from the stationInfoCSV file. Default case seems to be ''dot'' but other ones on the subversion server appear to be ''dot-NoAT'', ''dot-noSoil'', and ''dot-snow'' ... or create your own.<br> | ||
+ | In this template file, Column 1 is found in header at the top of the file downloaded using LoggerNet. Column 2 is the corresponding variable name found in the template files: ''stationDiag.kid'', ''stationInclude.kid'', ''stationPage.kid''<br> | ||
+ | Deviously, the stuff after the comma (e.g. 40 cm) is an optional parameter and should appear on the figures generated by [[Graphing | graph.py]]. | ||
+ | |||
+ | ==Functions called from the *.kid templates== | ||
+ | These functions are defined at the top of gp.py: | ||
+ | * lr = latestReading --> use the latest reading from the .dat file for this variable (e.g. lr.windDirection) | ||
+ | * ld = lastDayReadings --> display the last 24 hours of reading from the .dat file for this variable (e.g. ld.windDirection) | ||
+ | * si = stationInfo --> display a parameter from the stationInfoCSV file (e.g. si.['StationLocationDescription']) | ||
+ | * ms2mph = ms2mph --> a function that takes a speed value in meters per second and converts it to miles per hour (e.g. ms2mph(lr.windSpeedAvg)) | ||
+ | * c2f = c2f --> a function that takes temperature in degrees Celsius and converts it to Fahrenheit (e.g. c2f(lr.panelTemp)) | ||
+ | * f2c = f2c --> a function that takes temperature in degrees Fahrenheit and converts it to Celsius (e.g. f2c(lr.airTempAvg)) | ||
+ | * mb2hg = mb2hg --> a function that takes pressure in millibars and converts to inches of mercury (e.g. mb2hg(lr.pressure)) | ||
+ | * format = format --> a function that sets the numerical format. Can be used alone or with another function embedded (e.g. format(0, ms2mph(lr.windSpeedAvg) which displays the average winspeed in mile per hour after converting it from meter per second. There are 0 decimal places shown for the value. Or, format(1, lr.solarPanelVoltageAvg) which displays the solarPanelVoltage with 0.1 precision) | ||
+ | |||
+ | If you look at the *.kid templates you'll see a mix of stuff like ${si.['StationLocationDescription']} and si.['StationLocationDescription']. If the function is part of an html tag like:<pre><p py:if="si['timeDiff'] > 24"></pre> then it doesn't need ${} however, most times you'll need to put the functions within the ${} marker. | ||
+ | |||
+ | ==Fleshing out more thoughts== | ||
+ | Will continue fleshing this out but had some more thoughts to add. | ||
+ | So, there are generally five parts of input: | ||
+ | # the .ini, which controls general directory locations, stations to process, station groupings etc. | ||
+ | # stationinfo.csv which contains station specific information (including data input file names) | ||
+ | # the .dat files, station specific (listed in stationinfo.csv under the column ''DataFileSuffixes'') | ||
+ | # the html templates (''*.kid''). The html templates are viewable in a web browser they just have variable anchors in place of real numbers like discussed above and are turned into regular html using the python package kid. The file names are hard coded into the program (also listed above): | ||
+ | ## stationDiag.kid | ||
+ | ## stationInclude.kid | ||
+ | ## stationPage.kid | ||
+ | # another key file that basically contains aliases between variables common to each station which may have different names in the specific .dat file (like different ways of identifying air temperature for example). This file is also specified in stationinfo.csv under the column ''TemplateMap''. File format is mentioned above. | ||
+ | |||
+ | So, changing the way plots are displayed as Emily talked about probably involves: | ||
+ | # creating a simple page with the figure, caption and a back navigation button | ||
+ | # modifiying stationInclude.kid so that the plot link goes to the page rather than the image. | ||
+ | Doing some early QA/QC stuff is even easier. Just write a new script (or point to datasets QA/QCed from the O.O. based file system Ken's been working on) that does the qaqc on the data and puts it all into a new file that has a first level of processing done to it. No need to change this program at all. The things that would change would be the the stationinfoCSV file with the general gp.py framework remaining in place. <br> | ||
+ | Additionally, to get older cr10x data to work with this system wouldn't be a giant deal. The julian date & time would need to be changed into a regular format as the table based is and the arrays (met/soil/snow/precip events) would each need to go into their own data file but otherwise, not to much required. |
Latest revision as of 16:23, 1 October 2008
Contents
Overview
This is the first utility program given to us by the programmers at EE Internet. Turns out this program just does the html pages and not the graphs. So, in some ways it is quite similar to the perl program rawdata2html.
Required Packages
To run it you'll need to install two extra python packages:
- Kid -- A simple and pythonic XML template language, the Kid package is the one that reads through the template pages looking for sensor variables and plugs values in to those locations. Look at stationInclude.kid for example in a web browser to get a taste for the way Kid does things.
- Pytz -- A timezone and time helper package
- ConfigParser -- A configuration file reader (reads the .ini file and places the values found in it into the equivalent of read only perl hash). To use it there is a first order sectional variable such as [Main] and then kind of subvariables in the section such as configDir
Hardcoded Filenames
These should be located in the configuration directory (variable configDir in the .ini file):
- stationDiag.kid (case sensitive, this is the html template for the diagnostic page)
- stationInclude.kid (case sensitive, this is the html template for the center frame on the current website layout)
- stationPage.kid (case sensitive, this is a currently unused - at least on the public werc site- alternate html template)
Program Execution and Background
Instruction for installation are on the python page. To run gp from the command line you give python the input script name and also the name/location of an initialization file:
$ python gp.py ADOT/dot.ini
This setup was modified by Bob after the materials we got from the EEI people. For this run I copied everything off the EEI page and put it all into the directory
/work/python/gws/gp
Important directories you'll need full and be filling:
This is where the gp.py created html will go. On your first run directories will be created for each station.
/work/python/gws/gp/ADOT/html
This is where all of the source data files from loggernet are contained. There is no directory structure here, everything is lumped into one directory.
/work/python/gws/gp/ADOT/data
It appears this directory, although listed in the configuration file, is not currently used by gp.py... but it may be used by graph.py
/work/python/gws/gp/ADOT/plots
In addition, for the first run there is some code at the end of gp.py that tries to send an email. I commented that section out (lines 314 to 358 in revision 143 of gp.py) for the understanding how things work part of the process.
Sample .ini file
Here's a sample working initialization file (dot.ini from the EEI server).
[Main] # networkName is used on the [http://www.eeinternet.com/dot/html/adot-diag.html|diagnostics page] networkName = ADOT # Where the configuration files lie configDir = /work/python/gws/gp/ADOT # File with all the station information stationInfoCsv = StationInfo.csv # Directory containing the template files templateDir = /work/python/gws/gp/ADOT # The base dir for rendered files outputDirBase = /work/python/gws/gp/ADOT/html # Base URL of the data files (http://, ftp://, file:///home/data, etc) #dataFileBaseURL = http://werc.engr.uaf.edu/~ken/nslope-adot/ # dataFileBaseURL = http://www.gwscientific.com/remote/adot/ dataFileBaseURL = file:///work/python/gws/gp/ADOT/data/ # Name of the diagnostics files diagOutputName = adot-diag.html # Are we generating the "stripped down" include files? generateInclude = True # Where are the graphs going? graphOutputDirBase = /work/python/gws/gp/ADOT/plots diagURL = http://www.eeinternet.com/dot/html/adot-diag.html baseStationOrder = bullen,foothills [DataRecency] threshold = 0.25 notifyEmail = fnkci@uaf.edu,jkugler@eeinternet.com [bullen] baseStationName = Bullen Point Project stationOrder = DBM1,DBM2,DBM3,DBM4,DBM5,DBM6,DBM7,DBM8,DBR1,DBR2,DBR3,DBR4,DBR5 [foothills] baseStationName = Kuparuk Foothills Project stationOrder = DFM1,DFM2,DFM3,DFM4,DFM5,DFR1,DFR2,DFR3
Sample StationInfo.csv File
So, you can get the feeling from reading the ADOT StationInfo.csv file below that all of this info gets used in various ways in the junk outputted to the html pages. Also, line 1 is a list of the keys. Row 2 Column 1 is another key and then key values march across the columns to the right like a multi-dimensional hash:
StationName,PakbusID,StationDescriptiveName,StationLocationDescription,Latitude,Longitude,MagneticDeclination(East),ElevationFt,ElevationM,DataFileDirectory,DataFileNameBase,DataFileSuffixes,OutputDir,TemplateMap DBM1,201,Accomplishment Creek Met,"Accomplishment Ck, below lake at pass to Rubicon",N 68 24.696,W 148 8.190,22.6,4833,1474.065,,DBM1_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM1,dot DBM2,202,Ribdon Met,Upper Rubicon,N 68 38.548,W 147 21.107,23.1,4648,1417.64,,DBM2_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM2,dot DBM3,203,Juniper Met,Upper Juniper Ck,N 69 4.570,W 146 30.294,23.8,4324,1318.82,,DBM3_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM3,dot DBM4,204,Sag-Ivishak Met,"Close to snow survey site UP1, aka FT1",N 69 12.933,W 148 33.116,22.9,1414,431.27,,DBM4_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM4,dot DBM5,205,Upper Kad Met,Kadleroshilik uplands,N 69 32.968,W 147 56.505,23.5,686,209.23,,DBM5_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM5,dot DBM6,206,Kavik Met,Kavik camp,N 69 40.402,W 146 54.034,24.1,649,197.945,,DBM6_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM6,dot DBM7,207,Lower Kad Met,"Kadleroshilik River, near the old runway",N 70 4.406,W 147 39.000,23.8,78,23.79,,DBM7_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM7,dot DBM8,208,Bullen Met,South of Bullen Pt,N 70 4.792,W 146 49.166,24.3,86,26.23,,DBM8_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DBM8,dot DBR1,211,TBD,TBD,,,,,,,DBR1_,"",DBR1,dot DBR2,212,Ribdon Rep,Somewhere around Lupine valley?,N 68 36.337,W 147 23.716,23.2,N/A,N/A,,DBR2_,"HrlyAtms.dat,HrlyDiag.dat",DBR2,dot DBR3,213,Pogopuk Rep,Between Gilead Ck and Niviak Pass,N 69 17.369,W 146 22.990,23.3,N/A,N/A,,DBR3_,"HrlyAtms.dat,HrlyDiag.dat",DBR3,dot DBR4,214,Kavik Rep,"South of Kavik Camp, next to winter trail",N 69 37.893,W 146 52.889,23.4,1436,437.98,,DBR4_,"HrlyAtms.dat,HrlyDiag.dat",DBR4,dot DBR5,215,Franklin Bluffs Rep,"Franklin Bluffs, NE side",N 69 48.633,W 148 19.540,23.1,875,266.875,,DBR5_,"HrlyAtms.dat,HrlyDiag.dat",DBR5,dot DFM1,601,South White Hills Met,WERC HV5 snowsurvey site,N 69 12.043,W 149 33.508,22.4,962,293.41,,DFM1_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DFM1,dot DFM2,602,White Hills Met,WERC White Hills previous repeater site,N 69 29.187,W 149 49.284,22.5,,,,DFM2_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DFM2,dot DFM3,603,North White Hills Met,Between lake and stream to NNE of White Hills,N 69 42.892,W 149 28.227,22.6,276,84.18,,DFM3_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DFM3,dot DFM4,604,North West Kuparuk Met,WERC H03 snowsurvey site,N 69 56.851,W 149 55.014,22.7,406,123.83,,DFM4_,"HrlyAtms.dat,HrlyDiag.dat,HrlySubs.dat",DFM4,dot DFM5,605,TBD,TBD,,,,,,,DFM5_,"",DFM5,dot DFR1,611,Kak Rep,Kakukturat Mtn,N 69 4.357,W 149 30.870,22.3,1667,508.435,,DFR1_,"HrlyAtms.dat,HrlyDiag.dat",DFR1,dot DFR2,612,Slope Mnt Rep,Slope Mtn Repeater Site (relocated to top),N 68 44.448,W 149 1.989,,3858,1176.69,,DFR2_,"HrlyAtms.dat,HrlyDiag.dat",DFR2,dot DFR3,613,Shell Pingo Rep,Shell Pingo Top,N 70 1.234,W 147 40.903,23.8,408,124.44,,DFR3_,"HrlyAtms.dat,HrlyDiag.dat",DFR3,dot
Sample Variable Mapping file
The contents of TemplateMap-dot.profile:
# DATA_FILE_NAME = templateName TIMESTAMP = timeStamp WindSpeed_WVc(1) = windSpeedAvg, Avg Wind Speed WindSpeed_WVc(2) = windDirection, Wind Direction WindSpeed_Max = windSpeedMax, Max Wind Speed batt_volt_Avg = batteryVoltage, Battery Voltage SolarPnlV_Avg = solarPanelVoltageAvg, Avg Solar Panel Voltage PanelT = panelTemp, Panel Temperature CRNT_AT_Avg = airTempAvg, Avg Air Temperature CRNT_RH_Avg = relativeHumidityAvg, Avg Relative Humidity CRNT_DP_Avg = dewTempAvg, Avg Dew Point Precip_Tot = totalPrecipitation, Total Precipitation SnowDepth_Avg = snowDepthAvg, Avg Snow Depth Therm_degC(1) = soilSurfaceTemp, Soil Surface Temperature Therm_degC(4) = soilProfileSurface, 0 cm Therm_degC(5) = soilProfileOne, 5 cm Therm_degC(6) = soilProfileTwo, 10 cm Therm_degC(7) = soilProfileThree, 15 cm Therm_degC(8) = soilProfileFour, 20 cm Therm_degC(9) = soilProfileFive, 40 cm Soil_Moisture_1 = SoilMoistureOne, 5 cm Soil_Moisture_2 = SoilMoistureTwo, 20 Soil_Moisture_3 = SoilMoistureThree, 40 cm
This file name isn't hard coded into the program but nearly so. Format is: TemplateMap-stationInfoCSV.TemplateMap.profile where stationInfoCSV.TemplateMap in the previous string is a value from the stationInfoCSV file. Default case seems to be dot but other ones on the subversion server appear to be dot-NoAT, dot-noSoil, and dot-snow ... or create your own.
In this template file, Column 1 is found in header at the top of the file downloaded using LoggerNet. Column 2 is the corresponding variable name found in the template files: stationDiag.kid, stationInclude.kid, stationPage.kid
Deviously, the stuff after the comma (e.g. 40 cm) is an optional parameter and should appear on the figures generated by graph.py.
Functions called from the *.kid templates
These functions are defined at the top of gp.py:
- lr = latestReading --> use the latest reading from the .dat file for this variable (e.g. lr.windDirection)
- ld = lastDayReadings --> display the last 24 hours of reading from the .dat file for this variable (e.g. ld.windDirection)
- si = stationInfo --> display a parameter from the stationInfoCSV file (e.g. si.['StationLocationDescription'])
- ms2mph = ms2mph --> a function that takes a speed value in meters per second and converts it to miles per hour (e.g. ms2mph(lr.windSpeedAvg))
- c2f = c2f --> a function that takes temperature in degrees Celsius and converts it to Fahrenheit (e.g. c2f(lr.panelTemp))
- f2c = f2c --> a function that takes temperature in degrees Fahrenheit and converts it to Celsius (e.g. f2c(lr.airTempAvg))
- mb2hg = mb2hg --> a function that takes pressure in millibars and converts to inches of mercury (e.g. mb2hg(lr.pressure))
- format = format --> a function that sets the numerical format. Can be used alone or with another function embedded (e.g. format(0, ms2mph(lr.windSpeedAvg) which displays the average winspeed in mile per hour after converting it from meter per second. There are 0 decimal places shown for the value. Or, format(1, lr.solarPanelVoltageAvg) which displays the solarPanelVoltage with 0.1 precision)
If you look at the *.kid templates you'll see a mix of stuff like ${si.['StationLocationDescription']} and si.['StationLocationDescription']. If the function is part of an html tag like:
<p py:if="si['timeDiff'] > 24">
then it doesn't need ${} however, most times you'll need to put the functions within the ${} marker.
Fleshing out more thoughts
Will continue fleshing this out but had some more thoughts to add. So, there are generally five parts of input:
- the .ini, which controls general directory locations, stations to process, station groupings etc.
- stationinfo.csv which contains station specific information (including data input file names)
- the .dat files, station specific (listed in stationinfo.csv under the column DataFileSuffixes)
- the html templates (*.kid). The html templates are viewable in a web browser they just have variable anchors in place of real numbers like discussed above and are turned into regular html using the python package kid. The file names are hard coded into the program (also listed above):
- stationDiag.kid
- stationInclude.kid
- stationPage.kid
- another key file that basically contains aliases between variables common to each station which may have different names in the specific .dat file (like different ways of identifying air temperature for example). This file is also specified in stationinfo.csv under the column TemplateMap. File format is mentioned above.
So, changing the way plots are displayed as Emily talked about probably involves:
- creating a simple page with the figure, caption and a back navigation button
- modifiying stationInclude.kid so that the plot link goes to the page rather than the image.
Doing some early QA/QC stuff is even easier. Just write a new script (or point to datasets QA/QCed from the O.O. based file system Ken's been working on) that does the qaqc on the data and puts it all into a new file that has a first level of processing done to it. No need to change this program at all. The things that would change would be the the stationinfoCSV file with the general gp.py framework remaining in place.
Additionally, to get older cr10x data to work with this system wouldn't be a giant deal. The julian date & time would need to be changed into a regular format as the table based is and the arrays (met/soil/snow/precip events) would each need to go into their own data file but otherwise, not to much required.