saltpipe – SALT data reduction pipeline
saltpipe obsdate archive email ftp emserver emuser empasswd qcpcuser qcpcpasswd ftpserver ftpuser ftppasswd median function order rej_lo rej_hi niter interp (clobber) logfile verbose (status)
String. The functional form of the fit intended to characterize the the bias structure in the overscan region. The user has variety of function options to choose from:
chebyshev - Chebyshev polynomial
polynomial - standard polynomial
legendre - Legendre polynomial
spline1 - linear splines
spline3 - cubic splines
If the chebyshev, legendre of spline functions are called then saltbias will use the IRAF task colbias to subtract the overscan bias from science frames and trim the images.
String. The interpolation scheme used to rebin pixel values during the process of mosaicing CCD amplifier images into one single detector image. This requires both translations and rotations. The choices are:
linear -- linear function
nearest -- nearest pixel center
poly3 -- 3rd order polynomial function
poly5 -- 5th order polynomial function
spline3 -- cubic spline function
sinc -- sinc function
‘nearest’ is the least expensive for CPU processing but the least accurate, the ‘sinc’ function is the most expensive.
By nature, the saltpipe task is a linear series of individual pipeline tasks. To understand the principles of the underlying sub-tasks, the user is referred to the individual help documents for saltfixsec, saltedtky, saltlog, saltclean, salthtml, saltobsid, saltftp and saltemail. The user can run all of these steps individually using the sub-tasks if required or re-perform parts of the pipeline manually. The processing sequence is as follows:
1. Create a working direcrtory below the current directory with a name defined by obsdate, e.g. YYYYMMDD.
2. Test for the existence of both RSS and SALTICAM data directories on ctfileserver, consistent with obsdate. The absense of one instrument directory does not results in an error, but the absence of both does and the pipeline will stop, sending the user a warning email.
3. Check that RSS and SALTICAM data directories are complete. If they are not, the pipeline will stop, sending a warning email to the user.
4. Notify the user by email that the pipeline has started (if email=’yes’).
6. Convert any SLOT mode data to FITS format using the task saltbin2fits. FITS files are written to the raw directories in the temporary workspace.
7. If data were obtained before 12/08/06, fix the erroneous SEC keywords in raw data using the task saltfixsec. The procedure overwrites raw files in the temporary workspace.
8. Identify a file containing header keyword changes requested by the duty Astronomer at the telescope. These files will be of the name /home/sa/newheadfiles/list_newhead_YYYYMMDD. If an appropriate file does not exist, saltpipe will attempt to identify and FTP a suitable file from the machine at the telescope qcpc.salt. If that operation is unsuccessful, the pipeline will continue without keyword edits. At the current phase of the project, the most critical information that is missing from keywords is PI names. Without these edits, data cannot be coupled to PIs by the pipeline.
9. Perform header keyword edits using the task saltedtky. This procedure overwrites raw files in the temporary workspace. FITS tables containining a log of the keyword changes is stored in the product directory in the temporary workspace.
10. Create observation logs for the RSS and SALTICAM data. These logs are FITS tables stored in the product directory in the temporary workspace. The logs contain one row for each data file and one column for each primary header keyword. This step is probably temporary and the functions of the log file in data processing will be replaced by the SALT database once it is online.
11. Process and clean the raw images. The steps are keyword preparation, gain correction, crosstalk correction, bias subtraction and amplifier mosaicing. Tasks for flat fielding, fringe subtraction and cosmic ray rejection are pending. All steps are performed by calling the task saltclean, which internally calls, saltprepare, saltgain, saltxtalk, saltbias, saltmosaic and saltslot. Cleanded data are stored in the product directory of the temporary workspace.
12. From the observation logs, identify all PIs associated with the data. Collate data files, both raw and cleaned, according to PI, create new directories on the temporry workspace named after the PIs and populate them with symbolic links to the appropriate raw and cleaned data files. All steps are performed by the task saltobsid.
13. Generate HTML documentation containing details, of the night log, exposure sequence, pipeline log etc. Each PI receives a copy of the documentation which is deposited in the doc/ directory. This procedure is performed by the task salthtml.
14. For each PI, archive the data in a bzipped tar file and tranfer it to the public FTP server for retrieval by the PI. The tar file is deleted from the temporary workspace directly after transfer to the FTP server. The procedure is performed by the task saltftp.
15. Using saltemail, if email=’yes’, email each PI notfication that they have data to retrive.
16. If archive=’yes’, move the temporary working directory to ctfileserver for permanent storage.
17. If email=’yes’, email the user notification of the successful/unsuccessful completion of the pipeline with procesing statistics.
1. To execute the full pipeline on all observations from the night starting 16 Aug 2006:
--> saltpipe obsdate=20060816 archive=yes email=yes ftp=yes
emserver=smtp.saao.ac.za emuser=**** empasswd=****
qcpcuser=**** qcpcpasswd=**** ftpserver=www.saao.ac.za
ftpuser=**** ftppasswd=**** median=n function=polynomial
order=3 rej_lo=3.0 rej_hi=3.0 niter=10 interp=linear
clobber=y logfile=saltpipe.log verbose=y
The user and password arguments have been swapped for the string
'****' in this example to avoid security issues.
Individual unbinned full frame RSS image files can be 112MB in size. It is recommended to use workstations with a minimum of 512MB RAM. Depending on file numbers and sizes, the pipeline may take many hours to complete. The goal is to keep the running times shorter than real time observing so that the pipeline never becomes backlogged.
Ideally data should be reduced using data extracted from the SALT database. Until the SALT database is released, data files will not contain an observation ID within keywords. Before then, saltpipe will operate upon PI names supplied by the observer.
See individual tools below for individual bugs and limitations.
Send feedback and bug reports to salthelp@saao.ac.za