http://ocotal.iarc.uaf.edu/index.php?action=history&feed=atom&title=NGEE_Download_ARM_Data 2026-08-13T08:18:19Z Revision history for this page on the wiki MediaWiki 1.34.2 http://ocotal.iarc.uaf.edu/index.php?title=NGEE_Download_ARM_Data&diff=2526&oldid=prev 2016-02-11T01:07:37Z

<p><span dir="auto"><span class="autocomment">Reformatting the Data</span></span></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 01:07, 11 February 2016</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l13" >Line 13:</td> <td colspan="2" class="diff-lineno">Line 13:</td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>===Reformatting the Data===</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>===Reformatting the Data===</div></td></tr> <tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>Ross has made a Python tool to get from netCDF files to csv.  It's part of the standard DataPro tool suite.   </div></td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>Ross has made a Python tool to get from netCDF files to csv.  It's part of the standard <ins class="diffchange diffchange-inline">[[</ins>DataPro<ins class="diffchange diffchange-inline">]] </ins>tool suite.   </div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* https://github.com/rwspicer/csv_utilities</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* https://github.com/rwspicer/csv_utilities</div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>** From a command line... where you wish these files to appear:</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>** From a command line... where you wish these files to appear:</div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>** &lt;pre&gt;git clone https://github.com/rwspicer/csv_utilities.git&lt;/pre&gt;</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>** &lt;pre&gt;git clone https://github.com/rwspicer/csv_utilities.git&lt;/pre&gt;</div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>** Then, the file for this one is ''rad_decoder.py''</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>** Then, the file for this one is ''rad_decoder.py''</div></td></tr> <tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>*<del class="diffchange diffchange-inline">* more </del>to <del class="diffchange diffchange-inline">come once I hash this out</del>...</div></td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>* <ins class="diffchange diffchange-inline">The program reads through the NetCDF files with one minute out put for each day... compute an hourly average and then outputs </ins>to <ins class="diffchange diffchange-inline">the standard DataPro formatted csv file.  Here is an example bash script for making the magic happen (one note -- ''be sure to leave the trailing forward slash on the directory as it isn't added by the python program.'')</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline">&lt;pre&gt;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline">$ cat ~/bin/arm_rad</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline">#!/bin/bash</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline">#</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline"># This uses rad_decoder.py from the csv_utilities</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline"># to convert ARM netCDF files to hourly averages in csv format</ins>.</div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline"># last run 2/10/2016</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div> </div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline"># Set some variables:</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline"># GND is upwelling radiation, the downward facing sensors</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline">radd=&quot;/home/bob/lib/csv_utilities/rad_decoder</ins>.<ins class="diffchange diffchange-inline">py&quot;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline">indir=&quot;/home/bob/Documents/barrow_arm/gndrad60/&quot;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline">outdir=&quot;/home/bob/Documents/barrow_arm/gndrad60csv/&quot;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline">python $radd --in_directory=$indir --out_directory=$outdir --sitename=&quot;barrow&quot;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div> </div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline">#SKY RAD is downwelling radiation, the sensors facing upwards</ins>.</div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline">indir=&quot;/home/bob/Documents/barrow_arm/skyrad60/&quot;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline">outdir=&quot;/home/bob/Documents/barrow_arm/skyrad60csv/&quot;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline">python $radd --in_directory=$indir --out_directory=$outdir --sitename=&quot;barrow&quot;</ins></div></td></tr> </table>

imported>Bob http://ocotal.iarc.uaf.edu/index.php?title=NGEE_Download_ARM_Data&diff=2525&oldid=prev 2016-02-10T21:06:53Z

<p>Created page with &quot;===Procedure for Selecting Data=== ===Procedure for downloading Data=== * Probably easiest to do this from linux. I found &#039;&#039;&#039;lftp&#039;&#039;&#039; is a simple ftp program that I&#039;ve be...&quot;</p> <p><b>New page</b></p><div>===Procedure for Selecting Data===<br /> <br /> <br /> <br /> <br /> <br /> ===Procedure for downloading Data===<br /> * Probably easiest to do this from linux. I found '''lftp''' is a simple ftp program that I've been using for retrieving some Japanese data over FTP.<br /> * once installed... change to the directory you wish to download into:&lt;pre&gt;$ cd /home/bob/Documents/barrow_arm/sensor1/&lt;/pre&gt;<br /> * start up lftp: &lt;pre&gt;bob@elephant ~/Documents/barrow_arm/sensor1 $ lftp ftp.archive.arm.gov&lt;/pre&gt;<br /> * change to the proper remote directory: &lt;pre&gt;lftp ftp.archive.arm.gov:~&gt; cd youchae1/174468_extractedData&lt;/pre&gt;<br /> * download the data: &lt;pre&gt;lftp ftp.archive.arm.gov:/youchae1/174468_extractedData&gt; mget *&lt;/pre&gt;<br /> <br /> ===Reformatting the Data===<br /> Ross has made a Python tool to get from netCDF files to csv. It's part of the standard DataPro tool suite. <br /> * https://github.com/rwspicer/csv_utilities<br /> ** From a command line... where you wish these files to appear:<br /> ** &lt;pre&gt;git clone https://github.com/rwspicer/csv_utilities.git&lt;/pre&gt;<br /> ** Then, the file for this one is ''rad_decoder.py''<br /> ** more to come once I hash this out...</div>

imported>Bob