saltfast – Notify the PI that data were observed
saltftp pinames scamobslog rssobslog datapath server username password cleanup (clobber) logfile (verbose) (status)
saltftp is called within the SALT pipeline after raw image files have been cleaned and reduced (saltclean), separated into individual programs (saltobsid) and documentation for each program has been generated (salthtml). The task can also be executed standalone to update existing data on the FTP server or process individual programs rather than a full night of programs.
saltftp uses the RSS and SALTICAM observation logs created with saltlog as the basis for it’s tasks. In sequential order, saltftp will read the pre-prepared observation logs and compile a list of PIs who own data contained in the log. If pinames=’all’, data for every PI in the observation log will have tar files transferred to the FTP server. Otherwise pinames must contain a subset of the PIs in the observation log. saltlog will ensure that the subset is fully-contained in the observation log files or stop with an error.
The task will then ensure that each PI in the piname list has an identically named sub-directory under the path defined by the datapath argument. The matching is case-insensitve. If a mis-match is found then the task will stop with an error. Each sub-directory will normally contain data organized: datapath/PI/raw/ (raw data), datapath/PI/product/ (reduced data) and datapath/PI/doc (observation and reduction documentation).
For each matching sub-directory, all files contained within it will be tarred and compressed using bzip2 compression into an archive called YYYYMMDD.tar.bz2, where YYYY is the numerical year, MM the numerical month and DD the numerical day at sunset of the night of observations. The archive will be stored within the PI’s sub-directory, i.e. if the PI’s surname is Dopey, the archive will be stored in the file datapath/DOPEY/YYYYMMDD.tar.bz2.
Tar files are transferred to the public server using FTP. If there is no directory on the FTP server with a name identical to the PI a new one will be created. The tar file will be deposited in the PI-named directory.
If cleanup=’yes’ the tar files created on the local machine will be deleted before the task ends.
1. To create three bzipped2 tar files of PI-collated data called BILL.tar.bz2, BEN.tar.bz2 and WEED.tar.bz2 in the existing directories called /volumes/data2/BILL, /Volumes/data2/BEN and /Volumes/data2/WEED and ftp the tar files to the server ftp.xyz.ac.za, after which the tar files on the local machine will be deleted:
--> saltftp pinames='Bill,Ben,Weed' scamobslog='/Volumes/data1/Sobslog.fits'
rssobslog='/Volumes/data1/Pobslog.fits' datapath='/Volumes/data2'
server='ftp.xyz.ac.za' username='someone' password='!@#$%^&*'
logfile='salt.log' verbose='yes'
2. To create three bzipped2 tar files of PI-collated data called
BILL.tar.bz2, BEN.tar.bz2 and WEED.tar.bz2 in the existing
directories called /volumes/data2/BILL, /Volumes/data2/BEN and
/Volumes/data2/WEED and ftp the tar files to the server
ftp.xyz.ac.za, without deleting the tar files on the local machine::
--> saltftp pinames='Bill,Ben,Weed' scamobslog='/Volumes/data1/Sobslog.fits'
rssobslog='/Volumes/data1/Pobslog.fits' datapath='/Volumes/data2'
server='ftp.xyz.ac.za' username='someone' password='!@#$%^&*'
cleanup='no' logfile='salt.log' verbose='yes'
saltftp creates large tar files, often containing many GB each. Space for these files are required both on the working machine and the detination server. Tarring GB of data may often require an hour or more of processing.
Until the SALT database is released, data files will not contain an observation ID within keywords. Before then, saltftp tar and transfer data according to PI name.
Send feedback and bug reports to salthelp@saao.ac.za