UVES_headsort
Sort ESO/VLT UVES exposures based on header-card values

---

By: Michael T. Murphy

Usage: UVES_headsort [OPTIONS] [FITS file or list]

Brief description:
When one wishes to reduce ESO/VLT UVES spectra, typically extracted from the science archive, one must go through all the FITS files and decide which ones are the relevant science exposures and which calibration exposures are best to use. This can take a long time. If one has many objects to reduce, it can also be very tedious and you are prone to making mistakes. This can be true even with nice programs provided by ESO, like Gasgano.

UVES_headsort is a C code designed to automate this organisational task. In principle, it is very simple: for each science exposure that it identifies in a given list, it searches for calibration exposures taken soon before or soon after. It creates directories which are named according to the science object names in the FITS headers and then creates links to all the necessary science and calibration files within that directory. The links are named in an obvious way such that the ESO MIDAS and CPL VLT/UVES pipeline software can be used. It also goes one step further and writes reduction scripts which the pipeline understands. The only input required (for default behaviour) is a list of FITS files.

Non-standard requirements:
CFITSIO library.

Copyright:
Copyright 2003-2016 Michael T Murphy

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

---

Contents

1. Download and installation
2. System architectures tested
3. Update information
4. Basic Operation
   1. The input FITS file list: ABSOLUTE PATH NAMES!
   2. Debug mode
5. Options
   • -a [Bef.] [Aft.]
   • -c [Bef.] [Aft.]
   • -bias [N]
   • -flat [N]
   • -wav [N]
   • -fmt [N]
   • -ord [N]
   • -std [N]
   • -redscr
   • -redstd
   • -tharfile [FILE]
   • -atmofile [FILE]
   • -info [opt. FILE]
   • -list
   • -d
   • -h, -help
6. Calibration exposure selection
7. Output
   • Science object directories
   • Links to FITS files
   • MIDAS and CPL reduction scripts

1. Download and installation:
The best, and now only, way to grab the code is to get it from GitHub. You can grab a zip file from that site or, better, you can just use the"git" to "pull" the code to your computer, i.e.

prompt> git clone https://github.com/MTMurphy77/UVES_headsort.git

After grabbing the code (and unzipping the zip file, if that's what you downloaded), change to the source code directory:

prompt> cd UVES_headsort/

You will need to edit the Makefile to reflect your system's architecture, where your CFITSIO libraries are and where you'd like the final executables to be put. For the latter, change the TARGET keyword in the Makefile header. You should then just be able to type

prompt> make

and hopefully things will just compile nicely. Let me know if you have problems compiling the code.

Then, to install the various programs compiled and some additional shell scripts (used by the reduction scripts written out when one runs UVES_headsort), you should type

prompt> make install

Afterwards you might like to do a

prompt> make clean

to remove the *.o and temporary *~ files.


2. System architectures tested:
UVES_headsort is designed to run on UNIX and Linux systems. So far, it has been tested on a wide variety of linux and UNIX platforms, including linux-based Mac OS versions. UVES_headsort should work on any UNIX/Linux installation since the libraries it uses are fairly standard. Let me know if you have problems getting it to work on a specific platform.


3. Update information:
UVES_headsort is a new program and it's written by a C-code amateur. I will try to update the available version (as well as this page) when I make changes. Please let me know if you find bugs or you wish UVES_headsort to do something differently or something new.

I also maintain a list of interested parties to which I send emails regarding update information. If you would like to receive such emails, send an email to me with the Subject heading "Subscribe: UVES_headsort". If you want to stop receiving such emails, send an email to me with the Subject heading "Unsubscribe: UVES_headsort".


4. Basic operation:
To execute the program for the first time, it's best to use the debugging mode so that you can look for errors associated with your input FITS files:

prompt> UVES_headsort -d file_list.txt

You will no doubt receive either an error message or several warning messages. These will probably be associated with your input file (file_list.txt) or the FITS files pointed to by that file.

1. The input FITS file list: ABSOLUTE PATH NAMES!: Getting the input FITS file list correct is crucial to the workings of UVES_headsort. This file should be text only and should only contain one file name per line. The crucial thing is that each FITS file should be listed with its full absolute path name. For example, the following file list is NOT valid:
   
   25169/2001-07-13/UVES.2001-07-14T10:55:34.466.fits
   25169/2001-07-13/UVES.2001-07-14T10:56:17.006.fits
   25169/2001-07-13/UVES.2001-07-14T10:56:59.554.fits
   
   because each line doesn't begin with a "/". It's also a good idea to have the FITS files (pointed to by file_list.txt) somewhere other than the present working directory (PWD).
   
   
2. Debug mode: Using the "-d" option causes the program to enter debug mode in which no changes are made to your system, i.e. the program executes normally up to the point where directories and files/links need to be created. This allows you to quickly check whether the list of files you have supplied in file_list.txt (i) is a list of valid FITS files and (ii) contains all the required calibration frames for each science frame. Any error or warning messages returned should prompt you to change your file list in some way, whether that be (i) correcting errors in file_list.txt or (ii) including/excluding more files from the list. It may also prompt you to change the defaults using the other options (by, for example, increasing the calibration period).

5. Options:

• -a : Attached calibration period. The att. cal. prd. is the time-span - "centered" on the observation time of a given science exposure - over which UVES_headsort will search for a wavelength calibration exposure (i.e. ThAr wavelength cals) which has the same grating encoder value as the corresponding science exposure. This means the calibration has a high chance of having been "attached" to the science exposure, though it doesn't guarantee it. To make this more likely, UVES_popler will select as a first priority the first such cal. after the science exposure that is within 1/5th of the attached cal. period. Failing that, it will select any such cal. within the attached cal. period. If no such cal.s are found, UVES_popler will silently revert to the normal calibration period (the -c option below) for selecting an appropriate ThAr calibration, without any requirement on the grating encoder value.
  
  There are 2 arguments to this option, (i) the time in hours BEFORE and (ii) the time AFTER the science exposure. Thus, one can define (and indeed the default is) an "asymmetric" cal. prd.. For example, one may wish to seach for "attached" calibrations taken within 2 hours after the science exposure but not more than 1 hour before it. One would use
  
  prompt> UVES_headsort -a 1.0 2.0 file_list.txt
  
  UVES_headsort will search for only a single "attached" calibration; this is not affected by using the -wav option.
  
  
• -c : Calibration period. The cal. prd. is the time-span - "centered" on the observation time of a given science exposure - over which UVES_headsort will search for associated calibration exposures (i.e. biases, flats, ThAr wavelength cal.s, ThAr order-definitions, ThAr format checks and standard-stars).
  
  There are 2 arguments to this option, (i) the time in hours BEFORE and (ii) the time AFTER the science exposure. Thus, one can define an "asymmetric" cal. prd.. For example, one may wish to seach for calibrations taken not before but within 5 hours after the science exposures. One would use
  
  prompt> UVES_headsort -c 0.0 5.0 file_list.txt
  
  UVES_headsort will search for the requested number of each type of calibration file (see below) within the cal. prd., selecting those which are closest in time to the science frame.
  
  Default: -c 7.0 12.0
  
  
• -bias : Number of biases. This is the number of biases to be selected for calibration of each science frame.
  
  There is 1 argument to this option, the maximum number of biases to select if all of them are within the calibration period. So, for example,
  
  prompt> UVES_headsort -c 0.0 5.0 -bias 3 file_list.txt
  
  would select 3 biases which lie closest in time but within 5 hours after each science exposure in file_list.txt. A warning is issued if less than 3 biases are in file_list.txt within 5 hours after the science exposure.
  
  Default: -bias 5
  
  
• -flat : Number of flats. Same as for -bias but for the flatfield exposures.
  
  Default: -flat 5
  
  
• -wav : Number of wavelength calibrations. Same as for -bias but for the wav exposures.
  
  At the moment, the UVES pipeline is such that reducing only 1 ThAr wavelength calibration exposure per science object is possible. Therefore, specifying the argument to be greater than 1 has no meaning. However, setting the argument to "2" might at least give you a hint that the wavelength calibration might be poor for a given science object since you may see more than 1 wav file in the final object directory.
  
  Default: -wav 1
  
  
• -ord : Number of order-definition exposures. Same as for -wav but for the ord exposures.
  
  Default: -ord 1
  
  
• -fmt : Number of format check exposures. Same as for -wav but for the fmt exposures.
  
  Default: -fmt 1
  
  
• -std : Number of standard star exposures. Same as for -wav but for the std exposures.
  
  You may specify "-std 0" here if you wish and this will avoid any warning messages you might receive if you know a prioi (or from a previous run through in debug mode) that there are no standard stars included in file_list.txt. This will also automatically hit the -redstd switch as well.
  
  Default: -std 0
  
  
• -redscr : Turn off reduction script writing. By default UVES_headsort will attempt to write out some MIDAS and CPL UVES pipeline reduction scripts in each science object directory. It will try to write a reduction script for each object, one script used to prepare the directory for the reduction process and a master script to be used if you want to reduce all the science frames all at once. Using the -redscr option causes this NOT to happen - i.e. no reduction scripts are written.
  
  
• -redstd : Include standards in reduction scripts. This option simply causes the reduction scripts to be written so as to include reference to standard stars. Thus, even is you use "-std 1", for example, using the -redstd switch in addition will make sure that these standard star frames are not reduced. The links to the standard star frames will be made, however.
  
  
• -tharfile : Full path name of ThAr laboratory reference frame. This is the location of the ThAr reference frame that is included as part of the MIDAS or CPL UVES pipeline software. UVES_headsort only needs to know where this file is if it is to write reduction scripts. You may also specify where this file is by setting the environment variable UVES_HEADSORT_THARFILE in your shell. The default value is set in UVES_headsort.h and so can be changed if you are installing UVES_headsort system-wide.
  
  Default: -tharfile /data/software/local/midas/uves/calib/thargood_2.tfits
  
  
• -atmofile : Full path name of Atmospheric line reference frame. This is the location of the atmospheric line reference frame that is included as part of the MIDAS or CPL UVES pipeline software. UVES_headsort only needs to know where this file is if it is to write reduction scripts. You may also specify where this file is by setting the environment variable UVES_HEADSORT_ATMOFILE in your shell. The default value is set in UVES_headsort.h and so can be changed if you are installing UVES_headsort system-wide.
  
  Default: -atmofile /data/software/local/midas/uves/calib/atmoexan.tfits
  
  
• -info : Write header information to a file. An ASCII file is written out to the working directory which contains a list of the files in file_list.txt with some associated header information. After running UVES_headsort and receiving some warning messages, this file is very useful when trying to determine which calibration files you do or don't have in file_list.txt. By default, the name of the output file will be UVES_headsort.info but you can specify a different file on the command line using an (optional) argument to the "-info" option. For example,
  
  
• -list : Write lists of relevant files for each science exposure. For each object found in file_list.txt an ASCII file is written to the working directory containing a list of the files in file_list.txt which are relevant to that object. This will include all science, standard and calibration files. This is very useful for breaking down a long file-list containing many objects and redundant calibration files into object-specific file-lists with only the relevant files shown. The names of the list files will be Object_name.list where Object_name is the name of the object as specified in the header files.
  
  
• -d : Debug mode. Description is above
  
  
• -h, -help : Help message. A possibly-useful help message is written out containing the list of options and a brief description of each.

6. Calibration exposure selection:
There are several criteria which a calibration exposure must satisfy if it is to be used to calibrate a given science exposure. These are listed below for each calibration type. One selection criterion common to all is that each calibration exposure must be within the specified calibration period of the science exposure.

• Biases: Each science exposure is taken with a certain central wavelength. Generally speaking, if this is below 500nm then the blue CCD of UVES is used to receive the dispersed light (i.e. the blue arm of UVES is used). If it's above 500nm then the red CCDs are used. Thus, a suitable bias must firstly be of the correct CCD(s). Secondly, the biases must be binned in the same way as the science exposure. For example, if the science exposure is binned 2x2 then the bias cannot be binned 2x1 or 1x1.
  
  
• Flats: Flats must have the same central wavelength as the science exposure. They must also have the same binning, just as with biases. Additionally, they must also have been taken using the same slit-width (future versions may allow you to relax this constraint).
  
  
• ThAr wavelength calibrations (wavs): Same as for flats.
  
  
• ThAr order definitions (ords): Order def.s must have the same central wavelength and binning as the science exposure.
  
  
• ThAr format checks (fmts): Same as for ords.

7. Output:
Apart from warning and error messages, the optional header information output and object-specific file-list files, UVES_headsort is designed to make science object directories, appropriately named links to the raw FITS files within those directories and MIDAS and CPL reduction scripts to ease the overall pain of doing the pipeline reduction. These are described below.

• Science object directories:
  The input FITS file list, file_list.txt, will generally contain many exposures of many different science objects (e.g. different QSOs). FITS header cards should distinguish between science objects and standard stars. UVES_hedsort will find all the different science exposures and create directories (beneath the directory where UVES_headsort was executed) which are named after the object names in the FITS headers.
  
  NOTE: Existing directories will not be modified and UVES_headsort will exit with an error message if it sees an existing directory with the same name as one it wants to create.
  
  NOTE: It is crucial that the object names in the FITS headers are consistent. For example, if the same object, say Q1234-5678 was observed with two exposures and the names in the FITS headers are "Q1234-5678" and "Q1234" then two different object directories will be created.
  
  NOTE: Special symbols in the FITS header object names (such as "+", "-" and "_") will be changed to other appropriate letters ("p", "m" and "u" respectively) when creating the object directories. This is necessary to avoid problems when writing using the MIDAS and CPL reduction scripts for the UVES pipeline.
  
  
• Links to FITS files:
  Within each science object directory, UVES_headsort will attempt to create soft links to each of the science exposures and each calibration file that is required to reduce them using the UVES pipeline. Each link is given a unique name will the following format:
  
  Object_Type_CenWavlen_SciInd_CalInd.fits
  
  
  • Object: For science exposures and standard star exposures this is the name in the FITS header (subject to the naming conditions above). For bias and flat exposures this is simply "bias" and "flat". For wavelength calibrations, order definitions and format check exposures this is "thar" because all three are exposures of the ThAr lamp.
  • Type: For science exposures this is "sci" and for standards it is "std". For biases and flats this is "cal". For wavelength cal.s, ord. def.s and fmt. checks this is "wav", "ord" and "fmt".
  • CenWavlen: Aside from biases, this is the central wavelength in nanometers taken directly from the FITS header. For biases, the central wavelength of the corresponding science exposure is used so that you can tell which biases go with which science exposures (biases themselves do not have a central wavelength header card because they do not involve the light-path of UVES - they are dark frames).
  • SciInd: This is the index number of the science frame. Up to 99 science frames (with the same central wavelength) are allowed. Calibration frames (including standards) are given the index number of the science exposure to which they correspond.
  • CalInd: Science frames do not have this index added. There may be many calibration frames of the same type per science exposure (e.g. several flats are usually taken and applied to each science exposure) and this index (again, between 0 and 99) enumerates them. Even if only 1 calibration exposure of a given type is requested (e.g. 1 standard star exposure with the same central wavelength as the science exposure), this index is still added.
  
  
  
• MIDAS and CPL UVES pipeline reduction scripts:
  Since UVES_headsort gathers all the information needed to decide which calibration frames are best to use and then actually names the links, it seems appropriate for it to also write out MIDAS and CPL pipeline reduction script so that, in the ideal case, you can immediately reduce your UVES data once it's been UVES_headsort'ed. To this end, the following files are written into each science object directory:
  
  
  • reduce_CenWavelen_SciInd.[prg or cpl]: For each science exposure (denoted by the science index) of each central wavelength, a reduction script is written out which should fully reduce that science exposure using the MIDAS and CPL UVES pipeline. However, if problems occur during the reduction, no attempt is made to check what's happened or even if something out-of-the-ordinary has happened. Some error checks may be included in future.
  • reduce_prep.[prg or cpl]: This script can be run within MIDAS or CPL in order to prepare the current science object directory for the UVES pipeline. It should be run before any of the science object reduction scripts above.
  • reduce_master.[prg or cpl] If you're confident that no problems in the reduction will occur then this script will simply run all the above scripts in sequence, thereby reducing all of your data with one command. The CPL scripts are designed to run entirely automatically and so doing using the reduce_master.cpl (i.e. doing a "source reduce_master.cpl" in a terminal) is highly recommended. Some errors may occur, but you can just check to see if all the expected pipeline products are produced to check whether that's the case. In my experience, 90-95% of exposures will be reduced well using this option when using the CPL pipeline. Neat, eh?

---

Last updated: 09th April 2008 by Michael Murphy.