ESO/VLT UVES post-pipeline echelle reduction
By: Michael T. Murphy
Usage: UVES_popler [OPTIONS] [FITS file or list]
PGPLOT and CPGPLOT libraries.
Copyright 2003-2015 Michael T. Murphy
This program is free software; you can redistribute it and/or modify it
under the terms of the GNU
General Public License as published by the Free Software Foundation;
either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
- Download and installation
- System architectures tested
- Update information
- Basic Operation
- The input FITS file list: ABSOLUTE PATH NAMES!
- The files pointed to by
file_list.txt: Flat-fielded, bias-corrected, extracted 2D spectra
General spectrum specifications:
Initial sigma-clipping of echelle orders (not done when using -thar):
- -combmeth = 0 : Comb. method, order-scaling (0), order-fitting (1)
- -disp = TBD : Dispersion in km/s or Angstroms (for lin. disp.)
- -filetype : File origin: UVES=0, IRAF=1, MAKEE=2, IRAFLSS=3, mixed=-1
- -zem = 0.00000 : Emission redshift of QSO (if required for cont. fit)
- -linear = 0 : Linear (1) or log-linear (0) dispersion
- -thar = 0 : Reduce and combine ThAr calibration spectra
Spectrum/order combination parameters:
- -nordclip = 3 : No. most positive pixels removed from clip window
- -nordsig = 35 : No. pixels in order sig.-clip window
- -ordsig = 5.00 : Sigma-clipping level for orders [rms in window]
- -ordsignbr = 3.00 : Sigma-clipping level for neighbouring pixels
- -ordsigzero = 3.00 : Only clip when mean is this many sigma above zero
Order continuum fitting method parameters:
- -clipsig = 3.00 : Sigma-clipping level for spectra combination
- -lrgerr = 4.00 : Large error rejection ratio for combination
- -rankspec = 0 : (0) Autoscale; (1) Hi-SNR ord of 1st spec unscaled
- -scalclip = 3.00 : Sigma-clipping level for inter-order scale-factor
- -scalerr = 0.20 : Limit for relative error on inter-order scale-factor
Combined spectrum continuum fitting method parameters:
- -contftyp = 2 : Type of fit (1=polynomial, 2=Chebyshev, 3=Legendre)
- -contord = 4 : Order of fitted continuum (1=const.)
- -contpctl = 0.10 : Lower percentile to remove for initial fit [%/100]
- -contsigl = 1.40 : Lower rej. level for order continuum fitting [sigma]
- -contsigu = 3.00 : Upper rej. level for order continuum fitting [sigma]
- -contwgt = 500 : Weighting scale in pixels for ends of orders
Data file options:
- -cordlya = 4 : Order of fitted continuum below Lyman-alpha
- -cordred = 6 : Order of fitted continuum above Lyman-alpha
- -ftyplya = 2 : Type of fit below Lyman-alpha (1=pol, 2=Che, 3=Leg)
- -ftypred = 2 : Type of fit above Lyman-alpha (1=pol, 2=Che, 3=Leg)
- -pctllya = 0.30 : Lower percentile to remove for initial Lya fit [%/100]
- -pctlred = 0.10 : Lower percentile to remove for initial red fit [%/100]
- -rsiglyal = 1.20 : Lower rej. level for Lyman-alpha continuum fit [sigma]
- -rsiglyau = 3.00 : Upper rej. level for Lyman-alpha continuum fit [sigma]
- -rsigredl = 1.40 : Lower rej. level for cont. fit redwards of Lya [sigma]
- -rsigredu = 3.00 : Upper rej. level for cont. fit redwards of Lya [sigma]
- -vclya = 20000.00 : Velocity scale for continuum fit below Lya [km/s]
- -vcred = 2500.00 : Velocity scale for continuum fit above Lya [km/s]
- -vlya = 5000.00 : Velocity below Lya where red cont. fit begins [km/s]
- -dat = 0 : Write out data file with normalized spectrum
- -save = 0 : Force UPL and FITS save (1) when quitting program
- User's manual
1. Download and installation:
The best, and now only, way to grab the code is to get it from
You can grab a zip file from that site or, better, you can install"github" program and then pull the code to your computer, i.e.
prompt> git clone https://github.com/MTMurphy77/UVES_popler.git
After grabbing the code (and unzipping the zip file, if that's what you downloaded), change to the source code directory:
prompt> cd UVES_popler/
You will need to edit the Makefile to reflect your
system's architecture, where your PGPLOT,
CPGPLOT and CFITSIO libraries are and where you'd like the final
executable to be put. For the latter, change the TARGET keyword in the
Makefile header. You should then just be able to type
prompt> make depend
and hopefully things will just compile nicely. Let me
know if you have problems compiling the code. Afterwards you might like
to do a
prompt> make clean
to remove the *.o and temporary *~ files.
2. System architectures tested:
UVES_popler is designed to run on UNIX and Linux systems. So far, it
has been tested on a wide variety of linux and UNIX platforms,
including linux-based Mac OS versions. UVES_popler should work
on any UNIX/Linux installation since the libraries it uses are fairly
standard. Let me know if you have problems
getting it to work on a specific platform.
3. Update information:
UVES_popler is a new program and it's written by a C-code amateur. I
will try to update the available version (as well as this page) when I
make changes. Please let me know if you find
bugs or you wish UVES_popler to do something differently or
To stay up to date with new versions and changes to UVES_popler, it is
best to simply follow it on
4. Basic operation:
To execute the program for the first time:
prompt> UVES_popler file_list.txt
You may receive either an error message or several warning
messages. These will probably be associated with your input file
(file_list.txt) or the FITS files pointed to by that file.
- The input FITS file list: ABSOLUTE PATH
NAMES!: Getting the input FITS file list correct
is crucial to the workings of UVES_popler. This file should be text only
and should only contain one file name per line. The crucial thing is that
each FITS file should be listed with its full absolute path
name. For example, the following file list is NOT valid:
because each line doesn't begin with a "/". It's also a good idea to have
the FITS files (pointed to by file_list.txt) somewhere other than
the present working directory (PWD).
- The files pointed to by
file_list.txt: Flat-fielded, bias-corrected, extracted 2D
spectra: The files pointed to by file_list.txt must be
FITS files output by the UVES pipeline. Various files are produced by
the pipeline but the ones pointed to by file_list.txt must be
the flat-fielded, bias-corrected, extracted 2D spectra of the science
object. These spectra have been binned in the spatial direction by the
optimal extraction procedure but HAVE NOT been redispersed
(i.e. re-binned in the dispersion direction).
The flux files should be labelled with a "fxb_" prefix. The associated
error files should have a "err_" prefix and the wavelength calibration
files should have a "wpol_" prefix. The "err_" and "wpol_" files must reside in the same directory as their associated "fxb_" files.
If you used the MIDAS UVES pipeline (now well out of date), the
"flx_" and "err_" prefixes are automatically added to the flux and
error files. The wavelength calibation information is stored as
wavelength calibration polynomial coefficients in tabular form in
files named lCWLchipBINXxBINY_2.tbl. Here CWL is the central
wavelength of the exposure, chip is "_" for blue settings and "L" or
"U" for the bluer and redder of the red UVES chips, BINX and BINY are
the spectral and spatial CCD binning factors. For example
l390_2x2_2.tbl, l580L1x1_2.tbl and l580U1x1_2.tbl. These files must be
converted to FITS format within the pipeline and given the same name
as the "fxb_" files but with the "fxb_" replaced by "wpol_". If you
used UVES_headsort to sort the raw
UVES spectra and write the MIDAS reduction scripts then the naming
conventions above are automatically produced via the reduction
If you used the modern CPL UVES pipeline using ESOREX routines
directly, you will need to have added the "--debug" flag to the
"uves_obs_scired" command. This will produce the required "fxb_" flux
file and associated "errfxb_" error file. The latter must be renamed
to have "err_" as the prefix. If you used UVES_headsort to sort the raw UVES
spectra and write the CPL reduction scripts then the naming
conventions above are automatically produced via the reduction
If you used ESO's Reflex interface to the CPL pipeline, following these instructions (courtesy of Trystyn Berg at ESO) will produce the required files:
- Open the spectrum extraction actor: i.e. Right-click → Open actor.
- Configure the uves_obs_scired actor (double-click), and set the option "debug=true" (all lower case).
- Run the UVES reduction workflow as normal. The relevant "fxb_" and "errfxb_" files appear within the reflex_book_keeping/uves/uves_obs_scired1// directory. The latter must be renamed with the "err_" prefix. The relevant LINE_TABLE file is listed within the sof.dat file in the same directory; it should be remamed to have the "wpol_" prefix.
- -vlya : Velocity below Lya where red
cont. fit begins [km/s].
Use the redshift of the object input into UVES_popler to compute the
predicted observed wavelength of the Lya emission line, w_Lya, and
then select a wavelength were you'd like the Lya continuum to stop and
the red continuum to start, w_red. The value of vlya =
c*(1-w_red/w_Lya) is the value to input into UVES_popler. Note that
sometimes there can be quite substantial shifts between the Lya line
and the CIV or SiIV lines which are more usually used to determine the
emission redshift of QSOs.
- -h, -help : Help message. A
possibly-useful help message is written
out containing the list of options, a
brief description of each and the default
value of the parameter associated with the
6. User's manual:
6.2 Initial order cosmic ray cleaning:
For each order that is read in, a (nordclip)-pixel window is slid
along it and the mean and variance of the pixels in the window
(without including the nordclip highest flux pixel) is
calculated. Then if the middle pixel is more than (ordsig)-sigma [i.e.
sqrt(variance)] above the mean then it is regarded as a cosmic ray and
removed. Also, if the pixel immediately to the left of the removed
pixel is more than (ordsignbr)-sigma above the mean then it is also
removed. The same is true for the pixel to the right of the first
6.5 Status symbol meanings:
- Star: Pixel is invalid in UVES pipeline product.
- Asterisk: Pixel was flagged as a cosmic ray in initial order cosmic ray cleaning.
- Circle: Pixel had a large error
bar compared to others at the same wavelength and was not
considered during the spectral combination.
- Square: Pixel rejected by sigma-clipping algorithm during combination.
- Ex (i.e. "x"): There were no valid raw pixels contributing to this combined pixel.
- Triangle: Pixel has been cosmetically clipped by the user.
- Cross: Pixel clipped from continuum re-fit.
6.6 Action IDs:
- 1 : Clip pixels from an individual order
- 2 : Clip pixels from the combined spectrum
- 3 : Unclip pixels from an individual order
- 4 : Unclip pixels from the combined spectrum
- 5 : Fit a new continuum to part of an individual order
- 6 : Fit a new continuum to part of the combined spectrum
- 7 : Draw/spline a new continuum to part of an individual order
- 8 : Draw/spline a new continuum to part of the combined continuum
- 9 : Scale the flux and error array of an individual order
- 10: Auto-rescale orders after user has manually scaled some of them
- 11: Auto-reset continuum fit to combined spectrum
6.8 Step-by-step approach to a good reduction:
A summary of the six steps I normally take to get a reliable reduction
and a UPL file which is easier to follow and understand is as follows:
- 1 : Play around with command-line parameters to reduce
number of later manual actions.
- 2 : Remove large spikes generally found at blue edges of
different wavelength settings; recombine.
- 3 : Search spectrum for really bad ghosts and problematic
orders which need to be entirely removed.
- 4 : Search for and rescale any orders which have not been
automatically rescaled properly (the auto-rescale feature
comes in handy here).
- 5 : Search for and correct any pixels that need clipping
and order-edges that are bent with respect to others and need
to be fitted out or clipped.
- 6 : Reset the combined continuum automatically and then
adjust continuum in places where automatic continuum is not