The em27gi software suite is meant to process from interferograms to retrieved DMF using i2s and ggg by Geoff Toon in a simple one-line manner.

BASIC DESCRIPTION
This suite simplifies running GGG and I2S with the same settings each time.  It has been designed to work specifically with the Caltech EM27 but can be easily modified for other EM27s or for the advanced user probably different instruments as well.  This suite requires that raw data is divided into daily folders named YYYMMDD and requires interferograms to be in opus format.  Spectra are written (and looked for) in monthly folders.
It also requires weather data in a plain text file with headers of UTCDate (format YY/MM/DD) & UTCTime (format HH:MM:SS) or CompSrlDate (example 735854.84046).  Also requires headers (and data) of Temp (or Tout), RH, Pout, WSPD, and WDIR.
Currenly only works for spectra acquired on ground (gnd).


REQUIRED FUNCTIONALITY
This relies on a lot of functionality, and thus may not work for many systems.
GGG must be installed (see the TCCON wiki)
I2S must be installed (see the TCCON wiki)
  Additionally the paths $gggpath and $i2spath need to be set your GGG and I2S paths respectivly.  See https://tccon-wiki.caltech.edu/Software/GGG/Download/GGG_2012_Release_Notes#Installation_Instructions for details on how to.  Ex. b-shell
  export GGGPATH=/home/toon/ggg         (then   source ~/.bashrc)
  Ex. c-shell
  setenv GGGPATH /home/toon/ggg         (then   source ~/.cshrc)
  MATLAB must be installed
System needs to be running c-shell (this is necessary for how scripts like multiggg.sh and post-processing.sh are called in.  This likely can be changed or modified to fit/find given system but I am relatively new to Linux and I don't know how).
System must be capable of running bash scripts (that's how most of egi is written)
Must be connected to the internet to download NOAA NCEP files.
grep
awk
sed
cut



SETUP INSTRUCTIONS
1)  All files need to be copied to a working directory that you have control of.

2)  You will need to set your file locations in a file_loc.lst file, following the format in the example file.  
An example entry for a filepath is:
/oco2-data/tccon/data/caltech_em27/cn_sun/  		    : (2) raw data directory (interferograms, seperated in daily folders)
Note full filepath, and it ends with a forward slash.  There must be a space at the end, and everything after the space is ignored.  I entered the colons just to make things look nice.
file_loc.lst needs to be changed for different instruments

3)  You will need to copy mod_maker9b into your idl folder ($GGGPATH/src/idl)
The difference between this and mod_maker9 is this function will allow inputs when it is called in (all in one line)

4)  You will need to have the correct windows for your instrument already set (if you've been running GGG already this should be done).


HOW TO RUN
I suggest looking into the screen utility before running large files.
The main file is egi_full.  The syntax is to enter (after navigating to folder):
./egi_full
Followed by the start date (format YYYYMMDD) end date (formate YYYYMMDD) and maximum number of processors to be used (default is 1).
That's it!  To get daily output files...


ADVANCED DESCRIPTION
egi_full
The script egi_full runs all the 6 sub-scripts (and checks there is weather data avalible in the .in files).  If you only want to run some of the scripts, you can comment out the ones you don't want with # or you can use exit to stop the code prematurely.
The 6 sub-scripts can each be run independently as well.

datein_sub 
Takes your inputs of start and end-dates and processor numbers and makes any fixes if necessary (e.g. start/end date backwards, or only one date given, or no date given).  This script also prevents you from using more than 40 processors (this can easily be changed).

(1) egiprepdaily and egi_prepdaily_multi
egiprepdaily has the purpose of creating .in files with given weather files, GPS files (optional, just enter location in file_loc.lst), and opus interferogram files.  Creates lists of spectra, and lists that include number of scans+time stamp+interferogram peak.  Uses a matlab script which will average up to 5 of the nearest weather points as long as they are within a minute's time, and also creates a file for arbitrary solar intensity based on the interferogram peaks.  Requires you to have already set a .top file to the .in.  
egi_prepdaily_multi is the multi-processor version (both scripts required).

(2) egi_i2sdaily
Runs i2s using .in files created by egiprepdaily

(3) egirunlogdaily and egi_runlogdaily_multi
Creates sunruns, runlogs, and lse files for each day from spectra created after running egi_i2sdaily.  This also includes an OSDS correction (gets written to a different, unused runlog file).  It also includes a static correction to the delta_nu value that is likely specific to the Caltech EM27 only (changes 0.24105889405 to 0.24105490224, which was found empirically from 3 months of data).
egi_runlogdaily_multi is the multi-processor version (both scripts required).


(4) egi_modmaker
This script redownloads all of the NOAA NCEP data for the given years (may be slow if system has a slow connection).  Then creates (or attempts to create) .mod files for every day.


(5) egigsetup and egi_gsetup_multi
Creates .gsin files (inputs to gsetup) and runs gsetup to create .ggg files for everyday for every window.  
egi_gsetup_multi is the multi-processor version (both scripts required).

(6) egi_gfitdaily
First creates a master multiggg_*.sh file and then runs gfit on multiple processors.  The master multiggg_*.sh file is optimized so that O2 will run first, then CO2, then CH4.  Finally CO, H2O, HDO, N2O, and air are run in a mixed up order.  Note: these are currently all the species allowed, but it could be easily modified to include other species. 
After gfit is run, post-processing is run on a daily basis.


Other scripts:
egi_cumulative
No inputs are used on this one, rather it just looks for all folders that contain data that has gone through processing.  Combines all the runlog, lse, mav, ray, and col files.  Then runs post-processing on combined files.  Note: for the EM27 outputs can get huge.
This script also requires a cumulative_exclude.lst that includes dates and specific spectra (or rather line-numbers) to exclude.  For example
20140528
20140529:1,2,30-50
Excludes any data from May 28, 2014 and excludes spectra (rather line-numbers) 1, 2, and 30 through 50 on the 29th.

pfit_jkh
Same as pfit, but allows user to specify spt locations, enter in new runlogs while running, and change file-save names.  Note: any enter settings are still reset upon closing pfit


POSSIBLE FUTURE UPDATES
*Include making NetCDF files in egi_cumulative, Debra already has a (Python?) script for this I think  (increasing need with increasing .eof file sizes).
*Stop redownloading entire NCEP files.  Only download a part.  - Unsure how to do this right now.  I've gotten errors everytime I've tried this.
*Also calculate SIA based on pre-amp.  I can't decide if I want this or not...
*Make an active delta_nu correction calculation.  -  May be needed, using static right now.
*Add in all other wildcard/leftover species in egi_gfitdaily. - Not a current need for this, our species we can measure never change.
*Allow code to run on non-csh/tcsh systems.  -  Not currently a need for this, so it's not planned for.
*Remove last dependence on MATLAB.  - Also not a current need for this, so it's not planned for.




