IARC 207 Wiki - User contributions [en]

WERC datasite architecture

2007-07-06T23:16:58Z

127.0.0.1:

= WERC datasite architecture =

WERC uses the Linux operating system for running downloads and
storing the data from the Campbell Scientific CR10 family of
dataloggers.

The following setup and conventions might be viewed as a type of
ad-hoc database, but the implementation is straightforward, using
ordinary directories and (''flat'') files.

==site root==

A '''site root''' directory is established on the data-gathering
host computer, e.g., /home/site/.

Individual sites could be found directly under this location, e.g.,
as:

/home/site/SITE1/
/home/site/SITE2/

but since we have so many different projects and locations, it makes sense to
provide subdirectories for those entities to help keep things organized,
and so the sites are usually defined under "area" directories:

/home/site/AREA1/SITE1/
/home/site/AREA1/SITE2/
/home/site/AREA2/SITE1/
/home/site/AREA2/SITE5/

Here we're showing two "areas" or "projects", but some of the sites contained
therein have the same names. In practice, it might be preferrable to keep the
site names unique, for example to allow utility programs to refer to the sites
without having to specify the area.

In any case, the "area" directories don't necessarily have any assumed
meaning or importance other than organization -- though they could do...

== site directory ==

A ''site'' -- here assumed to be a local or remote location with a data logger
and attached sensors -- is defined by a directory. What that directory contains
depends on how things are configured; a ''site'' could just have files in its
site directory, or it might include other subdirectories, e.g., var/, etc/, web/
tmp/, whatever makes sense for the intended use.

What '''really''' defines a ''site'' for the purposes here is the existence of
a '''site rc''' file.

== site rc file ==

In Unix-speak, ''rc'' files are plain text configuration files, very similar
to INI files found on (older? versions of) the Windows operating system. ('''rc''
itself might stand for "resource configuration", but other definitions can likewise
be found.)

For this WERC datasite system, we chose to use the filename '''siterc''' or it's
dotted (and so, "hidden") version, '''.siterc''', for storing configuration
settings.

Further, we defined a simple syntax for the site rc files:

section:
entry=value
...

where the rc file can contain any number of sections, and any number of
''entry=value'' entries under each section.

In practice, our convention has been to have an rc file section corresponding
to -- and sharing the name of -- the programs that are used to gather, process,
and move data on the site's behalf. Any program settings specific to the site
can then be viewed and configured directly in the rc file, e.g., using a text
editor -- subject to the normal user & group permissions inherent to the
Linux operating system.

Here's an [[WERC-datasite-siterc-example|example '''siterc''' file]] from
one of our active sites (with key settings changed).

WERC-datasite-siterc-example

2007-07-06T22:36:51Z

127.0.0.1:

name=East Headwater
type=CSI CR10
logfile=/home/site/nslope/east_hw/var/log
werc_id=EH

Campbell::Comm: = Campbell::Comm-sagriver:
# Campbell::Comm: = Campbell::Comm-toolik:

Campbell::Comm-sagriver:
host=sagriver.datanet.werc.nondns
lockfile=/var/lock/uaf-sagriver
port=5555
code=ATXC1ATD5551212
READONLY_FLAGS=0
timezone=AKST
timeout=-10
radio_window = 1+60, 10+60
radio_stats=/var/tmp/cc/radio-stats/sagriver

Campbell::Comm-toolik:
host=toolik.datanet.werc.nondns
lockfile=/var/lock/uaf-toolik
port=5555
code=ATXC0ATD5551212
READONLY_FLAGS=0
timezone=AKST
timeout=-10
radio_window = 1+60, 10+60
radio_stats=/var/tmp/cc/radio-stats/toolik

loggerdata:
# use_ascii=0
use_ascii=1
datapointer=/home/site/nslope/east_hw/var/DSP
datafile=/home/site/nslope/east_hw/var/data
newdatafile=/home/site/nslope/east_hw/var/newdata
retries=2
retrydelay=3
ascii_arrays=1
settime=10
program=var/program
getprogram_days=10
flags=var/flags
copylogs=var/
session=var/session

split-csv:
source = var/newdata
append = data/.@1-@n/SPOOL
verbose = 1
mkdir = 1

rawdata2proc:
procfs = proc/raw/
datafile = var/newdata
timestamp-106 = Year_RTM, Day_RTM, Hour_Minute_RTM
timestamp-115 = Year_RTM, Day_RTM, Hour_Minute_RTM

miscellaneous:
radioaddress=555-1212

rawdata2html:
page=web/current-raw.html
title = N.Slope, Kuparuk R, East headwater
data=var/newdata
numberfields=1

guidata2html:
page=web/current.html
title = Upper Kuparuk R., East headwater site
data=var/newdata
template = ../.web/uk_template.html
vars = web/template.var

tabulardata:
datafile=var/data
vars = web/template.var
hours = 168

files2spool4web:
from = web/
to = WEBHOST/NorthSlope/upper_kuparuk/e_headwater/
statefile = var/files2spool4web-state

check-headers:
headers=var/headers

check-program:
program=var/program
annotate=1
text=web/csi-program.txt

push-datafile:
datafile = var/newdata
statefile = var/.push-datafile
pushdir = /home/waterprj/near-real-time/rawdatafiles/

WERC datasite architecture

2007-07-06T22:33:55Z

127.0.0.1:

= WERC datasite architecture =

WERC uses the Linux operating system for running downloads and
storing the data from the Campbell Scientific CR10 family of
dataloggers.

The following setup and conventions might be viewed as a type of
ad-hoc database, but the implementation is straightforward, using
ordinary directories and (''flat'') files.

==site root==

A '''site root''' directory is established on the data-gathering
host computer, e.g., /home/site/.

Individual sites could be found directly under this location, e.g.,
as:

/home/site/SITE1/
/home/site/SITE2/

but since we have so many different projects and locations, it makes sense to
provide subdirectories for those entities to help keep things organized,
and so the sites are usually defined under "area" directories:

/home/site/AREA1/SITE1/
/home/site/AREA1/SITE2/
/home/site/AREA2/SITE1/
/home/site/AREA2/SITE5/

Here we're showing two "areas" or "projects", but some of the sites contained
therein have the same names. In practice, it might be preferrable to keep the
site names unique, for example to allow utility programs to refer to the sites
without having to specify the area.

In any case, the "area" directories don't necessarily have any assumed
meaning or importance other than organization -- though they could do...

== site directory ==

A ''site'' -- here assumed to be a local or remote location with a data logger
and attached sensors -- is defined by a directory. What that directory contains
depends on how things are configured; a ''site'' could just have files in its
site directory, or it might include other subdirectories, e.g., var/, etc/, web/
tmp/, whatever makes sense for the intended use.

What '''really''' defines a ''site'' for the purposes here is the existence of
a '''site rc''' file.

== site rc file ==

In Unix-speak, ''rc'' files are plain text configuration files, very similar
to INI files found on (older? versions of) the Windows operating system. ('''rc''
itself might stand for "resource configuration", but other definitions can likewise
be found.)

For this WERC datasite system, we chose to use the filename '''siterc''' or it's
dotted (and so, "hidden") version, '''.siterc''', for storing configuration
settings.

Further, we defined a simple syntax for the site rc files:

section:
entry=value
...

where the rc file can contain any number of sections, and any number of
''entry=value'' entries under each section.

In practice, our convention has been to have an rc file section corresponding
to -- and sharing the name of -- the programs that are used to gather, process,
and move data on the site's behalf. Any program settings specific to the site
can then be viewed and configured directly in the rc file, e.g., using a text
editor -- subject to the normal user & group permissions inherent to the
Linux operating system.

Here's an [[WERC-datasite-siterc-example|example]] '''siterc''' file from
one of our active sites (with key settings changed).

WERC datasite architecture

2007-07-06T22:21:40Z

127.0.0.1:

Software

2007-07-06T21:39:29Z

127.0.0.1:

Here's a list of software commonly used in the department. You may wish to add tips or questions to the program's wiki page.

== Mac ==
* [[Dreamweaver]] -- HTML/CSS editing program
* [[Aptana]] -- HTML / CSS editing program with a different IDE from dreamweaver (some advantages for each)
* [[MS Office]] -- Excel, Powerpoint etc.
* [[fugu]] -- a nice GUI version of SFTP/SCP for the Mac

== Linux ==
* [[Perl]] -- Handy programming language for all sorts of things
* [[Loggernet]] -- Campbell Scientific's data retrieval program

== Windows ==
* [[Dreamweaver]] -- HTML/CSS editing program
* [[Aptana]] -- HTML / CSS editing program with a different IDE from dreamweaver (some advantages for each)
* [[MS Office]] -- Excel, Powerpoint etc.
* [[Adobe PDF tools]] -- Making PDFs (free alternatives also available)
* [[Adobe Photoshop]] -- Image Manipulation
* [[Adobe Illustrator]] -- Graphics creation
* [[Irfanview]] -- Image viewing program more flexible than the default Microsoft Image viewer
* [[Cygwin]] -- linux utilities for Windows
* [[ArcGIS]] -- ESRI's GIS program
* [[Putty]] -- Small suite of tools for doing SSH/SFTP/SCP
* [[SSH Secure Shell]] -- An alternative tool for doing SSH/SFTP
* [[Origin]] -- A fancy plotting & graph program
* [[Surfer]] -- A fancy plotting & graph program
* [[Perl]] -- Handy programming language for all sorts of things
* [[Loggernet]] -- Campbell Scientific's data retrieval program

==WERC in-house software development==
* data-gathering system, written in Perl:
** [[src-perl-Campbell::Comm-overview|overview]]
** [[src-perl-Campbell::Comm|Campbell::Comm]] -- perl module to interface Campbell CR10X loggers
** [[src-perl-querylogger|querylogger.pl]] -- query remote loggers
** [[src-perl-loggerdata|loggerdata.pl]] -- automatic downloads from remote loggers
** [[WERC datasite architecture]] -- setup & conventions assumed by the above

Campbell::Comm

2007-07-06T21:31:17Z

127.0.0.1:

Campbell::Comm

2007-07-06T21:30:32Z

127.0.0.1:

Campbell::Comm

2007-07-06T20:32:32Z

127.0.0.1:

Campbell Scientific's CR10 family of dataloggers has been heavily used by WERC
and many other research groups for years. Though now superseded by the CR1000
line of loggers, the CR10 line will likely be in service for years to come.

The CR10 loggers were provided with complete documentation, both for developing
the datalogging programs themselves and for interfacing with other systems via
serial communications. At WERC, we used this documentation to develop code in
the Perl programming language for ''talking to'' these datalogging systems, with
connections via serial radios (FreeWave) and the Internet.

Note that Campbell provides the LoggerNet (formerly PC208 and other)
programs to interface dataloggers to PCs; WERC's Campbell::Comm and related
programs serve a more limited function than LoggerNet/PC208, being primarily
concerned with reliably downloading data and checking/setting the time.

WERC's code is posted online at http://www.uaf.edu/water/staff/irving/csi-code/,
including a [http://www.uaf.edu/water/staff/irving/csi-code/readme README] file.
This system has been in use for several years, downloading data from up to 50
remote loggers hourly in several different radio networks.

This code has not been packaged for distribution, or automatic
installation, but it is intended to be available under the GNU Public
License (GPL).

Technical descriptions of the primary components follow...

Note that this Perl code has been written, tested, and run only on Linux
computers; Perl itself is very portable across platforms, but porting
problems (e.g,. to Windows OS family) might be encountered in such areas
as process management (fork, exec), ''alarm'' timing, and possibly others.

==Campbell/Comm.pm==

This module becomes part of a Perl program via the '''use''' or '''require'''
keywords. It provides the machinery to connect and send/receive commands
to/from a remote datalogger using Perl's object-oriented programming
methodology. Individual ''methods'' are provided corresponding to some/most
of the ''telecommunications'' commands documented in the Campbell Scientific
CR10 family manuals.

Another Perl module is needed in order to interact with remote dataloggers,
and that is Net::Telnet, which is included in the standard Perl distribution.
This module allows automating the telnet protocol, but boils down to being
able to send commands to the remote system and then wait for and handle
responses -- or no response, if that's necessary.

Besides the nominal TCP/IP connections, Net::Telnet is capable of supporting
interfaces using a direct serial or modem connection. This usage is
implemented in the (or various...) open() methods.

A ''bug'' in Net::Telnet (this is my view; it may be considered a ''feature''
by the authors...) requires a patch; this should be represented in the
Net::Telnet version included in the above tree. Specifically, Net::Telnet
goes out of its way to handle ''timeouts'' using ''wall'' time (i.e., absolute
time as measured on a wall clock) rather than ''relative'' inactivity time
as used in the underlying select() system function. In our case the latter
form is required due to the many levels and sources of latency in the
communication system.

The Perl Expect module might provide a good alternative to Net::Telnet if
Campbell::Comm were to be rewritten. Both provide the key functionality
of being able to wait for and selectivly match on responses from the
remote system.

Due to the specific need to communicate with dataloggers via FreeWave
radios, the Campbell::Comm module was written with this capability perhaps
too implicitly. A rewrite (anticipated) should remove any code not
directly concerned with the Campbell dataloggers.

==querylogger.pl==
This perl script was written to exercise and test the Campbell::Comm
module, but may be useful in its own right as a way to interface with
dataloggers in batch (i.e., non-interactive) form. Each method in the
Campbell::Comm module should be represented by a querylogger command
as specified on the command line.

The interface is described if run with no arguments:

$ querylogger
syntax: "$ /home/ken/bin/querylogger OPTIONS SITE COMMAND ...
where OPTIONS are
--quiet suppress output to STDOUT
-q same as above
--logfile=f print everything to file f
-l f same as above
--comm=s use rc file section Campbell::Comm-s, or
-c s use rc file section Campbell::Comm-s
--identify identify site in output (default)
--noidentify don't identify site in output
where SITE is one of: site, slope_mtn,
and COMMANDs may include:
status list status strings
time show time (setting not yet supported)
backup=n n=number of output arrays to spew (no limit)
MPTR=loc loc=output storage location (no limit)
data=n n=number of data records to get
binary=n n=number of data locations to get
program print all logger programming
flags optionally toggle specified flag(s), show flags
ports show ports (toggle not yet supported)
memory show mode A memory settings
signatures show mode B signature settings
input=n[,m][,r-s] show input storage locations
capture=[f] end previous capture, capture output to file f
get_time_flags experimental using F command

The querylogger (and loggerdata) scripts assume a specific system
configuration of sites and ''rc'' files; see below...

==loggerdata.pl==

The '''loggerdata.pl''' script is the workhorse of WERC's data
gathering system from CR10X loggers, and besides the '''querylogger.pl'''
script (see above) is the only script we've written around the Campbell::Comm
module.

See below for the [[#data site architecture]] assumed by the system.

Like [[#querylogger.pl]], '''loggerdata.pl''' runs in
a non-interactive or ''batch'' mode. But where the former program is
driven according to the set of commands provided on the command line, the
latter is hard-wired to follow a set sequence of commands.

As with any computer program, the details of what this program does are
spelled out in its source code. The following steps are from memory, and
should be confirmed...

* a connection is established to the datalogger
* several ''escape characters'' are sent in case the datalogger has been left in an edit or other state
* the datalogger command prompt, '''"*"''' is obtained
* the "next data memory location" ('''DSP''') is retrieved from the site state on disk (not on the logger)
* the datalogger's current data memory position ('''MPTR''') is obtained
* the '''G''' logger telecommunications command is used to set the '''MPTR''' to the last '''DSP''' value
* data is requested in either ASCII (the '''D''' command) or binary (the '''F''' command) mode
* incoming data is integrity-checked using checksums or ''signatures''
* confirmed-good data is stored on disk
* update the stored '''DSP''' state information
* repeat the above data query steps until:
** the '''DSP''' value reaches the logger's '''MPTR'''
** the query times out
** the allowed processing time for the session elapses
* query the datalogger's time
** if the logger time is off by more than a specified amount (e.g., 10 seconds), set the time
* if the logger program hasn't been downloaded for greater than a specified time (e.g., 10 days), download the program
** store the logger program to a '''version control system''' (RCS)
* query and store the datalogger ''flags'' settings
* disconnect from the remote logger
* compose a ''log message'' summarizing the session
** print the log message if the session has been run by a user
** append the log message to the site log file

In summary, the '''loggerdata.pl''' program:

* tries to download data from the last available location
** all data retrieved is integrity-checked, then stored
* checks (& maybe sets) the time
* maybe downloads the datalogger's current program

In general, we make every attempt to ensure that data is downloaded as
it exists on the datalogger, and safely stored on the data-gathering
computer. Data is not skipped; in particular, if a remote site has "fallen
behind", the program will patiently and persistently try to bring the
local (on-computer) data store in line with the remote logger.

The datalogger's time setting is of particular importance mainly so that
connection times can be kept ''in synch'', assuming that the radios are
powered on only at certain times. Of course, another benefit in checking/setting
the logger's time is that the stored data timestamps are meaningful in an
absolute sense -- but note that this could also be achieved by only checking
(and not setting) the time.

==data site architecture==

THIS IS A STUB .... NEEDS WORK!

Campbell::Comm

2007-07-06T20:14:53Z

127.0.0.1:

Campbell Scientific's CR10 family of dataloggers has been heavily used by WERC
and many other research groups for years. Though now superseded by the CR1000
line of loggers, the CR10 line will likely be in service for years to come.

The CR10 loggers were provided with complete documentation, both for developing
the datalogging programs themselves and for interfacing with other systems via
serial communications. At WERC, we used this documentation to develop code in
the Perl programming language for ''talking to'' these datalogging systems, with
connections via serial radios (FreeWave) and the Internet.

Note that Campbell provides the LoggerNet (formerly PC208 and other)
programs to interface dataloggers to PCs; WERC's Campbell::Comm and related
programs serve a more limited function than LoggerNet/PC208, being primarily
concerned with reliably downloading data and checking/setting the time.

WERC's code is posted online at http://www.uaf.edu/water/staff/irving/csi-code/,
including a [http://www.uaf.edu/water/staff/irving/csi-code/readme README] file.
This system has been in use for several years, downloading data from up to 50
remote loggers hourly in several different radio networks.

This code has not been packaged for distribution, or automatic
installation, but it is intended to be available under the GNU Public
License (GPL).

Technical descriptions of the primary components follow...

Note that this Perl code has been written, tested, and run only on Linux
computers; Perl itself is very portable across platforms, but porting
problems (e.g,. to Windows OS family) might be encountered in such areas
as process management (fork, exec), ''alarm'' timing, and possibly others.

==Campbell/Comm.pm==

This module becomes part of a Perl program via the '''use''' or '''require'''
keywords. It provides the machinery to connect and send/receive commands
to/from a remote datalogger using Perl's object-oriented programming
methodology. Individual ''methods'' are provided corresponding to some/most
of the ''telecommunications'' commands documented in the Campbell Scientific
CR10 family manuals.

Another Perl module is needed in order to interact with remote dataloggers,
and that is Net::Telnet, which is included in the standard Perl distribution.
This module allows automating the telnet protocol, but boils down to being
able to send commands to the remote system and then wait for and handle
responses -- or no response, if that's necessary.

Besides the nominal TCP/IP connections, Net::Telnet is capable of supporting
interfaces using a direct serial or modem connection. This usage is
implemented in the (or various...) open() methods.

A ''bug'' in Net::Telnet (this is my view; it may be considered a ''feature''
by the authors...) requires a patch; this should be represented in the
Net::Telnet version included in the above tree. Specifically, Net::Telnet
goes out of its way to handle ''timeouts'' using ''wall'' time (i.e., absolute
time as measured on a wall clock) rather than ''relative'' inactivity time
as used in the underlying select() system function. In our case the latter
form is required due to the many levels and sources of latency in the
communication system.

The Perl Expect module might provide a good alternative to Net::Telnet if
Campbell::Comm were to be rewritten. Both provide the key functionality
of being able to wait for and selectivly match on responses from the
remote system.

Due to the specific need to communicate with dataloggers via FreeWave
radios, the Campbell::Comm module was written with this capability perhaps
too implicitly. A rewrite (anticipated) should remove any code not
directly concerned with the Campbell dataloggers.

==querylogger.pl==
This perl script was written to exercise and test the Campbell::Comm
module, but may be useful in its own right as a way to interface with
dataloggers in batch (i.e., non-interactive) form. Each method in the
Campbell::Comm module should be represented by a querylogger command
as specified on the command line.

The interface is described if run with no arguments:

$ querylogger
syntax: "$ /home/ken/bin/querylogger OPTIONS SITE COMMAND ...
where OPTIONS are
--quiet suppress output to STDOUT
-q same as above
--logfile=f print everything to file f
-l f same as above
--comm=s use rc file section Campbell::Comm-s, or
-c s use rc file section Campbell::Comm-s
--identify identify site in output (default)
--noidentify don't identify site in output
where SITE is one of: site, slope_mtn,
and COMMANDs may include:
status list status strings
time show time (setting not yet supported)
backup=n n=number of output arrays to spew (no limit)
MPTR=loc loc=output storage location (no limit)
data=n n=number of data records to get
binary=n n=number of data locations to get
program print all logger programming
flags optionally toggle specified flag(s), show flags
ports show ports (toggle not yet supported)
memory show mode A memory settings
signatures show mode B signature settings
input=n[,m][,r-s] show input storage locations
capture=[f] end previous capture, capture output to file f
get_time_flags experimental using F command

The querylogger (and loggerdata) scripts assume a specific system
configuration of sites and ''rc'' files; see below...

==loggerdata.pl==

The '''loggerdata.pl''' script is the workhorse of WERC's data
gathering system from CR10X loggers, and besides the '''querylogger.pl'''
script (see above) is the only script we've written around the Campbell::Comm
module.

See below for the [[#data site architecture]] assumed by the system.

Like [[#querylogger.pl]], '''loggerdata.pl''' runs in
a non-interactive or ''batch'' mode. But where the former program is
driven according to the set of commands provided on the command line, the
latter is hard-wired to follow a set sequence of commands.

As with any computer program, the details of what this program does are
spelled out in its source code. The following steps are from memory, and
should be confirmed...

* a connection is established to the datalogger
* several ''escape characters'' are sent in case the datalogger has been left in an edit or other state
* the datalogger command prompt, '''"*"''' is obtained
* the "next data memory location" ('''DSP''') is retrieved from the site state on disk (not on the logger)
* the datalogger's current data memory position ('''MPTR''') is obtained
* the '''G''' logger telecommunications command is used to set the '''MPTR''' to the last '''DSP''' value
* data is requested in either ASCII (the '''D''' command) or binary (the '''F''' command) mode
* incoming data is integrity-checked using checksums or ''signatures''
* confirmed-good data is stored on disk
* update the stored '''DSP''' state information
* repeat the above data query steps until:
** the '''DSP''' value reaches the logger's '''MPTR'''
** the query times out
** the allowed processing time for the session elapses
* query the datalogger's time
** if the logger time is off by more than a specified amount (e.g., 10 seconds), set the time
* if the logger program hasn't been downloaded for greater than a specified time (e.g., 10 days), download the program
** store the logger program to a '''version control system''' (RCS)
* query and store the datalogger ''flags'' settings
* disconnect from the remote logger
* compose a ''log message'' summarizing the session
** print the log message if the session has been run by a user
** append the log message to the site log file

==data site architecture==

THIS IS A STUB .... NEEDS WORK!

Campbell::Comm

2007-07-06T20:13:33Z

127.0.0.1:

Campbell Scientific's CR10 family of dataloggers has been heavily used by WERC
and many other research groups for years. Though now superseded by the CR1000
line of loggers, the CR10 line will likely be in service for years to come.

The CR10 loggers were provided with complete documentation, both for developing
the datalogging programs themselves and for interfacing with other systems via
serial communications. At WERC, we used this documentation to develop code in
the Perl programming language for ''talking to'' these datalogging systems, with
connections via serial radios (FreeWave) and the Internet.

Note that Campbell provides the LoggerNet (formerly PC208 and other)
programs to interface dataloggers to PCs; WERC's Campbell::Comm and related
programs serve a more limited function than LoggerNet/PC208, being primarily
concerned with reliably downloading data and checking/setting the time.

WERC's code is posted online at http://www.uaf.edu/water/staff/irving/csi-code/,
including a [http://www.uaf.edu/water/staff/irving/csi-code/readme README] file.
This system has been in use for several years, downloading data from up to 50
remote loggers hourly in several different radio networks.

This code has not been packaged for distribution, or automatic
installation, but it is intended to be available under the GNU Public
License (GPL).

Technical descriptions of the primary components follow...

Note that this Perl code has been written, tested, and run only on Linux
computers; Perl itself is very portable across platforms, but porting
problems (e.g,. to Windows OS family) might be encountered in such areas
as process management (fork, exec), ''alarm'' timing, and possibly others.

==Campbell/Comm.pm==

This module becomes part of a Perl program via the '''use''' or '''require'''
keywords. It provides the machinery to connect and send/receive commands
to/from a remote datalogger using Perl's object-oriented programming
methodology. Individual ''methods'' are provided corresponding to some/most
of the ''telecommunications'' commands documented in the Campbell Scientific
CR10 family manuals.

Another Perl module is needed in order to interact with remote dataloggers,
and that is Net::Telnet, which is included in the standard Perl distribution.
This module allows automating the telnet protocol, but boils down to being
able to send commands to the remote system and then wait for and handle
responses -- or no response, if that's necessary.

Besides the nominal TCP/IP connections, Net::Telnet is capable of supporting
interfaces using a direct serial or modem connection. This usage is
implemented in the (or various...) open() methods.

A ''bug'' in Net::Telnet (this is my view; it may be considered a ''feature''
by the authors...) requires a patch; this should be represented in the
Net::Telnet version included in the above tree. Specifically, Net::Telnet
goes out of its way to handle ''timeouts'' using ''wall'' time (i.e., absolute
time as measured on a wall clock) rather than ''relative'' inactivity time
as used in the underlying select() system function. In our case the latter
form is required due to the many levels and sources of latency in the
communication system.

The Perl Expect module might provide a good alternative to Net::Telnet if
Campbell::Comm were to be rewritten. Both provide the key functionality
of being able to wait for and selectivly match on responses from the
remote system.

Due to the specific need to communicate with dataloggers via FreeWave
radios, the Campbell::Comm module was written with this capability perhaps
too implicitly. A rewrite (anticipated) should remove any code not
directly concerned with the Campbell dataloggers.

==querylogger.pl==

This perl script was written to exercise and test the Campbell::Comm
module, but may be useful in its own right as a way to interface with
dataloggers in batch (i.e., non-interactive) form. Each method in the
Campbell::Comm module should be represented by a querylogger command
as specified on the command line.

The interface is described if run with no arguments:

$ querylogger
syntax: "$ /home/ken/bin/querylogger OPTIONS SITE COMMAND ...
where OPTIONS are
--quiet suppress output to STDOUT
-q same as above
--logfile=f print everything to file f
-l f same as above
--comm=s use rc file section Campbell::Comm-s, or
-c s use rc file section Campbell::Comm-s
--identify identify site in output (default)
--noidentify don't identify site in output
where SITE is one of: site, slope_mtn,
and COMMANDs may include:
status list status strings
time show time (setting not yet supported)
backup=n n=number of output arrays to spew (no limit)
MPTR=loc loc=output storage location (no limit)
data=n n=number of data records to get
binary=n n=number of data locations to get
program print all logger programming
flags optionally toggle specified flag(s), show flags
ports show ports (toggle not yet supported)
memory show mode A memory settings
signatures show mode B signature settings
input=n[,m][,r-s] show input storage locations
capture=[f] end previous capture, capture output to file f
get_time_flags experimental using F command

The querylogger (and loggerdata) scripts assume a specific system
configuration of sites and ''rc'' files; see below...

==loggerdata.pl==

The '''loggerdata.pl''' script is the workhorse of WERC's data
gathering system from CR10X loggers, and besides the '''querylogger.pl'''
script (see above) is the only script we've written around the Campbell::Comm
module.

See below for the [[data site architecture]] assumed by the system.

Like [[#querylogger.pl]], '''loggerdata.pl''' runs in
a non-interactive or ''batch'' mode. But where the former program is
driven according to the set of commands provided on the command line, the
latter is hard-wired to follow a set sequence of commands.

As with any computer program, the details of what this program does are
spelled out in its source code. The following steps are from memory, and
should be confirmed...

* a connection is established to the datalogger
* several ''escape characters'' are sent in case the datalogger has been left in an edit or other state
* the datalogger command prompt, '''"*"''' is obtained
* the "next data memory location" ('''DSP''') is retrieved from the site state on disk (not on the logger)
* the datalogger's current data memory position ('''MPTR''') is obtained
* the '''G''' logger telecommunications command is used to set the '''MPTR''' to the last '''DSP''' value
* data is requested in either ASCII (the '''D''' command) or binary (the '''F''' command) mode
* incoming data is integrity-checked using checksums or ''signatures''
* confirmed-good data is stored on disk
* update the stored '''DSP''' state information
* repeat the above data query steps until:
** the '''DSP''' value reaches the logger's '''MPTR'''
** the query times out
** the allowed processing time for the session elapses
* query the datalogger's time
** if the logger time is off by more than a specified amount (e.g., 10 seconds), set the time
* if the logger program hasn't been downloaded for greater than a specified time (e.g., 10 days), download the program
** store the logger program to a '''version control system''' (RCS)
* query and store the datalogger ''flags'' settings
* disconnect from the remote logger
* compose a ''log message'' summarizing the session
** print the log message if the session has been run by a user
** append the log message to the site log file

==data site architecture==

THIS IS A STUB .... NEEDS WORK!

Campbell::Comm

2007-07-06T20:12:31Z

127.0.0.1:

Campbell Scientific's CR10 family of dataloggers has been heavily used by WERC
and many other research groups for years. Though now superseded by the CR1000
line of loggers, the CR10 line will likely be in service for years to come.

The CR10 loggers were provided with complete documentation, both for developing
the datalogging programs themselves and for interfacing with other systems via
serial communications. At WERC, we used this documentation to develop code in
the Perl programming language for ''talking to'' these datalogging systems, with
connections via serial radios (FreeWave) and the Internet.

Note that Campbell provides the LoggerNet (formerly PC208 and other)
programs to interface dataloggers to PCs; WERC's Campbell::Comm and related
programs serve a more limited function than LoggerNet/PC208, being primarily
concerned with reliably downloading data and checking/setting the time.

WERC's code is posted online at http://www.uaf.edu/water/staff/irving/csi-code/,
including a [http://www.uaf.edu/water/staff/irving/csi-code/readme README] file.
This system has been in use for several years, downloading data from up to 50
remote loggers hourly in several different radio networks.

This code has not been packaged for distribution, or automatic
installation, but it is intended to be available under the GNU Public
License (GPL).

Technical descriptions of the primary components follow...

Note that this Perl code has been written, tested, and run only on Linux
computers; Perl itself is very portable across platforms, but porting
problems (e.g,. to Windows OS family) might be encountered in such areas
as process management (fork, exec), ''alarm'' timing, and possibly others.

==Campbell/Comm.pm==

This module becomes part of a Perl program via the '''use''' or '''require'''
keywords. It provides the machinery to connect and send/receive commands
to/from a remote datalogger using Perl's object-oriented programming
methodology. Individual ''methods'' are provided corresponding to some/most
of the ''telecommunications'' commands documented in the Campbell Scientific
CR10 family manuals.

Another Perl module is needed in order to interact with remote dataloggers,
and that is Net::Telnet, which is included in the standard Perl distribution.
This module allows automating the telnet protocol, but boils down to being
able to send commands to the remote system and then wait for and handle
responses -- or no response, if that's necessary.

Besides the nominal TCP/IP connections, Net::Telnet is capable of supporting
interfaces using a direct serial or modem connection. This usage is
implemented in the (or various...) open() methods.

A ''bug'' in Net::Telnet (this is my view; it may be considered a ''feature''
by the authors...) requires a patch; this should be represented in the
Net::Telnet version included in the above tree. Specifically, Net::Telnet
goes out of its way to handle ''timeouts'' using ''wall'' time (i.e., absolute
time as measured on a wall clock) rather than ''relative'' inactivity time
as used in the underlying select() system function. In our case the latter
form is required due to the many levels and sources of latency in the
communication system.

The Perl Expect module might provide a good alternative to Net::Telnet if
Campbell::Comm were to be rewritten. Both provide the key functionality
of being able to wait for and selectivly match on responses from the
remote system.

Due to the specific need to communicate with dataloggers via FreeWave
radios, the Campbell::Comm module was written with this capability perhaps
too implicitly. A rewrite (anticipated) should remove any code not
directly concerned with the Campbell dataloggers.

==querylogger.pl==

This perl script was written to exercise and test the Campbell::Comm
module, but may be useful in its own right as a way to interface with
dataloggers in batch (i.e., non-interactive) form. Each method in the
Campbell::Comm module should be represented by a querylogger command
as specified on the command line.

The interface is described if run with no arguments:

$ querylogger
syntax: "$ /home/ken/bin/querylogger OPTIONS SITE COMMAND ...
where OPTIONS are
--quiet suppress output to STDOUT
-q same as above
--logfile=f print everything to file f
-l f same as above
--comm=s use rc file section Campbell::Comm-s, or
-c s use rc file section Campbell::Comm-s
--identify identify site in output (default)
--noidentify don't identify site in output
where SITE is one of: site, slope_mtn,
and COMMANDs may include:
status list status strings
time show time (setting not yet supported)
backup=n n=number of output arrays to spew (no limit)
MPTR=loc loc=output storage location (no limit)
data=n n=number of data records to get
binary=n n=number of data locations to get
program print all logger programming
flags optionally toggle specified flag(s), show flags
ports show ports (toggle not yet supported)
memory show mode A memory settings
signatures show mode B signature settings
input=n[,m][,r-s] show input storage locations
capture=[f] end previous capture, capture output to file f
get_time_flags experimental using F command

The querylogger (and loggerdata) scripts assume a specific system
configuration of sites and ''rc'' files; see below...

==loggerdata.pl==

The '''loggerdata.pl''' script is the workhorse of WERC's data
gathering system from CR10X loggers, and besides the '''querylogger.pl'''
script (see above) is the only script we've written around the Campbell::Comm
module.

See below for the [[data site architecture]] assumed by the system.

Like [[#querylogger.pl]], or [#querylogger.pl], '''loggerdata.pl''' runs in
a non-interactive or ''batch'' mode. But where the former program is
driven according to the set of commands provided on the command line, the
latter is hard-wired to follow a set sequence of commands.

As with any computer program, the details of what this program does are
spelled out in its source code. The following steps are from memory, and
should be confirmed...

* a connection is established to the datalogger
* several ''escape characters'' are sent in case the datalogger has been left in an edit or other state
* the datalogger command prompt, '''"*"''' is obtained
* the "next data memory location" ('''DSP''') is retrieved from the site state on disk (not on the logger)
* the datalogger's current data memory position ('''MPTR''') is obtained
* the '''G''' logger telecommunications command is used to set the '''MPTR''' to the last '''DSP''' value
* data is requested in either ASCII (the '''D''' command) or binary (the '''F''' command) mode
* incoming data is integrity-checked using checksums or ''signatures''
* confirmed-good data is stored on disk
* update the stored '''DSP''' state information
* repeat the above data query steps until:
** the '''DSP''' value reaches the logger's '''MPTR'''
** the query times out
** the allowed processing time for the session elapses
* query the datalogger's time
** if the logger time is off by more than a specified amount (e.g., 10 seconds), set the time
* if the logger program hasn't been downloaded for greater than a specified time (e.g., 10 days), download the program
** store the logger program to a '''version control system''' (RCS)
* query and store the datalogger ''flags'' settings
* disconnect from the remote logger
* compose a ''log message'' summarizing the session
** print the log message if the session has been run by a user
** append the log message to the site log file

==data site architecture==

THIS IS A STUB .... NEEDS WORK!

Campbell::Comm

2007-07-06T19:20:27Z

127.0.0.1:

Campbell::Comm

2007-07-06T17:40:43Z

127.0.0.1:

Campbell::Comm

2007-07-06T17:15:27Z

127.0.0.1:

Campbell::Comm

2007-07-06T17:02:01Z

127.0.0.1:

Campbell::Comm

2007-07-06T16:38:57Z

127.0.0.1:

Software

2007-07-06T15:50:05Z

127.0.0.1:

Here's a list of software commonly used in the department. You may wish to add tips or questions to the program's wiki page.
== Mac ==
* [[Dreamweaver]] -- HTML/CSS editing program
* [[Aptana]] -- HTML / CSS editing program with a different IDE from dreamweaver (some advantages for each)
* [[MS Office]] -- Excel, Powerpoint etc.
* [[fugu]] -- a nice GUI version of SFTP/SCP for the Mac
== Linux ==
* [[Perl]] -- Handy programming language for all sorts of things
** [[Campbell::Comm]] -- perl module to interface Campbell CR10X loggers
* [[Loggernet]] -- Campbell Scientific's data retrieval program
== Windows ==
* [[Dreamweaver]] -- HTML/CSS editing program
* [[Aptana]] -- HTML / CSS editing program with a different IDE from dreamweaver (some advantages for each)
* [[MS Office]] -- Excel, Powerpoint etc.
* [[Adobe PDF tools]] -- Making PDFs (free alternatives also available)
* [[Adobe Photoshop]] -- Image Manipulation
* [[Adobe Illustrator]] -- Graphics creation
* [[Irfanview]] -- Image viewing program more flexible than the default Microsoft Image viewer
* [[Cygwin]] -- linux utilities for Windows
* [[ArcGIS]] -- ESRI's GIS program
* [[Putty]] -- Small suite of tools for doing SSH/SFTP/SCP
* [[SSH Secure Shell]] -- An alternative tool for doing SSH/SFTP
* [[Origin]] -- A fancy plotting & graph program
* [[Surfer]] -- A fancy plotting & graph program
* [[Perl]] -- Handy programming language for all sorts of things
* [[Loggernet]] -- Campbell Scientific's data retrieval program

All projects

2007-07-05T20:04:58Z

127.0.0.1:

= WERC Research Stations -- Site Visit Logs & ToDo Lists =

== North Slope Kuparuk (Kane/NSF) ==
* main met sites:
** [[ukmet|Upper Kuparuk met]]
** [[fbmet|Franklin Bluffs met]]
** [[ibmet|Imnavait Basin met]]
** [[shmet|Sagwon Hill met]]
** [[wkmet|West Kuparuk met]]
** [[bpmet|Betty Pingo met]]
** [[wdmet|West Dock met]]
* precip sites in the headwaters:
* [[uh|Upper Headwaters]]
* [[eh|East Headwaters]]
* [[wh|West Headwaters]]
* [[nh|North Headwaters]]
* [[gcl|Green Cabin Lake]]
* radio base stations:
** [[sagriver|SagRiver Base]]
** [[toolik|ToolikLake Base]]
** [[deadhorse|Deadhorse Base]]
* radio repeater stations:
** [[smrep|Slope Mountain repeater]]
** [[ssrep|Sagwon Summit repeater]]
** [[ukrep|Upper Kuparuk Ridge repeater]]

== North Slope Bullen Project ==
* main met sites:
** [[dbm1|Accomplishment Creek met]] (DBM1)
** [[dbm2|Ribdon River met]] (DBM2)
** [[dbm3|Juniper Creek met]] (DBM3)
** [[dbm4|Sag-Ivishak met]] (DBM4)
** [[dbm5|Upper Kadleroshilik River met]] (DBM5)
** [[dbm6|Kavik Camp met]] (DBM6)
** [[dbm7|Lower Kadleroshilik River met]] (DBM7)
** [[dbm8|Bullen Point met]] (DBM8)
* radio repeater stations:
** [[dfr2|Slope Mtn2 repeater]] (DFR2)
** [[dbr2|Ribdon River repeater]] (DBR2)
** [[dbr3|Pogopuk Creek repeater]] (DBR3)
** [[dbr4|Kavik Ridge repeater]] (DBR4)
** [[dbr5|Franklin Bluffs repeater]] (DBR5)
** [[dfr3|Shell Pingo repeater]] (DFR3)

== North Slope Foothills Project ==
* main met sites:
** [[dfm1|South White Hills met]] (DFM1)
** [[dfm2|White Hills met]] (DFM2)
** [[dfm3|North White Hills met]] (DFM3)
** [[dfm4|Northwest Kuparuk met]] (DFM4)
* radio repeater stations:
** [[dfr1|Kakukturat Mountain repeater]] (DFR1)

All projects

2007-07-05T20:01:31Z

127.0.0.1:

= WERC Research Stations -- Site Visit Logs & ToDo Lists =

== North Slope Kuparuk (Kane/NSF) ==
* main met sites:
** [[ukmet|Upper Kuparuk met]]
** [[fbmet|Franklin Bluffs met]]
** [[ibmet|Imnavait Basin met]]
** [[shmet|Sagwon Hill met]]
** [[wkmet|West Kuparuk met]]
** [[bpmet|Betty Pingo met]]
** [[wdmet|West Dock met]]
* precip sites in the headwaters:
* [[uh|Upper Headwaters]]
* [[eh|East Headwaters]]
* [[wh|West Headwaters]]
* [[nh|North Headwaters]]
* [[gcl|Green Cabin Lake]]
* radio base stations:
** [[sagriver|SagRiver Base]]
** [[toolik|ToolikLake Base]]
** [[deadhorse|Deadhorse Base]]
* radio repeater stations:
** [[smrep|SlopeMountainRepeater]]
** [[ssrep|SagwonSummitRepeater]]
** [[ukrep|UpperKuparukRidgeRepeater]]

== North Slope Bullen Project ==
* main met sites:
** [[dbm1|AccomplishmentCreek]] (DBM1)
** [[dbm2|RibdonRiver]] (DBM2)
** [[dbm3|JuniperCreek]] (DBM3)
** [[dbm4|Sag-Ivishak met]] (DBM4)
** [[dbm5|UpperKadleroshilikRiver]] (DBM5)
** [[dbm6|KavikCamp]] (DBM6)
** [[dbm7|LowerKadleroshilikRiver]] (DBM7)
** [[dbm8|BullenPoint]] (DBM8)
* radio repeater stations:
** [[dfr2|SlopeMtn2Repeater]] (DFR2)
** [[dbr2|RibdonRiverRepeater]] (DBR2)
** [[dbr3|PogopukCreekRepeater]] (DBR3)
** [[dbr4|KavikRidgeRepeater]] (DBR4)
** [[dbr5|FranklinBluffsRepeater]] (DBR5)
** [[dfr3|ShellPingoRepeater]] (DFR3)

== North Slope Foothills Project ==
* main met sites:
** [[dfm1|SouthWhiteHills]] (DFM1)
** [[dfm2|WhiteHills]] (DFM2)
** [[dfm3|NorthWhiteHills]] (DFM3)
** [[dfm4|NorthWestKuparuk]] (DFM4)
* radio repeater stations:
** [[dfr1|KakukturatMountainRepeater]] (DFR1)