NGEE Download ARM Data

Procedure for downloading Data

• Probably easiest to do this from linux. I found lftp is a simple ftp program that I've been using for retrieving some Japanese data over FTP.
• once installed... change to the directory you wish to download into:

  $ cd /home/bob/Documents/barrow_arm/sensor1/

• start up lftp:

  bob@elephant ~/Documents/barrow_arm/sensor1 $ lftp ftp.archive.arm.gov

• change to the proper remote directory:

  lftp ftp.archive.arm.gov:~> cd youchae1/174468_extractedData

• download the data:

  lftp ftp.archive.arm.gov:/youchae1/174468_extractedData> mget *

Reformatting the Data

Ross has made a Python tool to get from netCDF files to csv. It's part of the standard DataPro tool suite.

• https://github.com/rwspicer/csv_utilities
  • From a command line... where you wish these files to appear:

  • git clone https://github.com/rwspicer/csv_utilities.git

  • Then, the file for this one is rad_decoder.py
• The program reads through the NetCDF files with one minute out put for each day... compute an hourly average and then outputs to the standard DataPro formatted csv file. Here is an example bash script for making the magic happen (one note -- be sure to leave the trailing forward slash on the directory as it isn't added by the python program.)

$ cat ~/bin/arm_rad
#!/bin/bash
#
# This uses rad_decoder.py from the csv_utilities
# to convert ARM netCDF files to hourly averages in csv format.
# last run 2/10/2016

# Set some variables:
# GND is upwelling radiation, the downward facing sensors
radd="/home/bob/lib/csv_utilities/rad_decoder.py"
indir="/home/bob/Documents/barrow_arm/gndrad60/"
outdir="/home/bob/Documents/barrow_arm/gndrad60csv/"
python $radd --in_directory=$indir --out_directory=$outdir --sitename="barrow"

#SKY RAD is downwelling radiation, the sensors facing upwards.
indir="/home/bob/Documents/barrow_arm/skyrad60/"
outdir="/home/bob/Documents/barrow_arm/skyrad60csv/"
python $radd --in_directory=$indir --out_directory=$outdir --sitename="barrow"