Pipeline Steps — AutoWISP documentation

# Pipeline Steps

The pipeline operations can be broken down into 10 big steps each of which is described in more detail below. The results of each step are saved in three types of files:

calibrated images:

  These are the individual images of the night sky, corrected for bias, dark current and flat fielding (see below). Those are stored using the FITS format with three extensions:

  > - The values of the image pixels

  > - Formal estimate of the standard deviation of each pixel value

  > - A pixel quality mask, flagging things like saturated pixels, pixels which may have received charge from overflowing neighboring pixels, etc.

data reduction files:

  These are the files that contain information about the stars in each image extracted from the calibrated images. Including things like star positions, point spread function information, flux measurements, etc. They are generally in FITS format and can be used for further analysis. Those are in HDF5 format.

lightcurves:

  These are the files that contain the time series of all available measurements for a given star star in each image. They are generally in HDF5 format, and include things like several different version of flux measurements, coordinates of the star in the image, and many others.

## Split Raw Frames by Type:

Before you begin processing the images you have accumulated, a bit of preparation is needed. It is very convenient and helps avoid mistakes if you split your images inte separate directories by type:

> bias

>   Images with near-zero exposure intended to measure the behavior of the analog-to-digital converter(s) of your camera. This shows up in the final image as a value that each pixel starts at, even if there is no signal.

> dark

>   Images with no light falling on the detector, but exposure similar (ideally equal) to the exposure used for science images. These are intended to measure the rate of accumulation of charge in the detector pixels in the absence of light.

> flat

>   Images of something with uniform brightness (or as close to it as one can manage). There are intended to measure the sensitivity to light of the system coming from different directions.

> object

>   Images of the night sky from which photometry is to be extracted. Those can further be split into sub-groups from which independent lightcurves need to be generated. For example if several different exposure times were used, or there could be a number of filters or other chages in the optical system between frames which may produce better results if processed independently.

## 1. Image Calibration

Commands: `wisp-calibrate`, `wisp-stack-to-master`, `wisp-stack-to-master-flat`

### Important Parameters

The most important parameters to set for the calibration are:

- `exposure-start-utc` or `exposure-start-jd`

- `exposure-seconds`

- `saturation-threshold`

- `fnum`

- `image-area`: If your camera generates images with overscan or other areas that are not part of the image.

- `split-channels` or `raw-hdu`: If using a color detector

- `gain`: If known. Used for accurate error estimates.

### Description

Before raw images are used, they need to be calibrated. The sequence of steps is:

> calibrate the raw bias images

>   Accomplished using `wisp-calibrate` command with no master files specified with raw bias images as input.

> generate master bias image

>   Use `wisp-stack-to-master` command with the calibrated bias images as input.

> calibrate raw dark images using the master bias

>   `wisp-calibrate` with the `master-bias` option set to the master bias file generated in the step above.

> generate master dark image

>   Just like generating master bias but using the calibrated dark images.

> calibrate the raw flat images

>   `wisp-calibrate` with the `master-bias` option set to the master bias file and the `master-dark` option set to the master dark.

> generate master flat image(s)

>   `wisp-stack-to-master-flat` command. This is different from how master bias and dark are created, because AutoWISP is designed to allow using sky flats, which may be affected by clouds, or the sky is not perfectly uniformly bright.

> calibrate raw object images

>   `wisp-calibrate` with `master-bias`, `master-dark`, and `master-flat` all specified.

### No Calibration Data?

In case calibration data is not available, only the last of these steps needs to be performed, with no masters specified (see below). Even though in this case the pixels values will not be corrected for any of the effects described above, this step is still needed. It will add change the data format (float point instead of integer), add required information in the headher, split the different colors if using a color camera etc.

### Overscan Corrections

In many instances, the imaging device provides extra areas that attempt to measure bias level and dark current, e.g. by continuing to read pixels past the physical number of pixels in the device, thus measuring the bias or by having an area of pixels which are somehow shielded from light, thus measuring the dark level in real time. Such corrections can be supierior to the master frames in that they measure the instantaneous bias and dark level, which may vary over time due to for example the temperature of the detector varying. However, bias level and dark current in particular can vary from pixel to pixel, which is not captured by these real-time areas. Hence, the best strategy is a combination of both, and is different for different detectors.

AutoWISP allows (but does not require) such areas to be used to estimate some smooth function of image position to subtract from each raw image, and then the masters are applied to the result. This works mathematically, because the masters will also have their values corrected for the bias and dark measured by these areas from the individual frames that were used to construct them. In this scheme, the master frames are used only to capture the pixel to pixel differences in bias and dark current. We refer to these areas as “overscan”, although that term really means only one type of such area.

Overscan area(s) can be specified using the `overscans` option of the `wisp-calibrate` command.

## 2. Find Stars

Command: `wisp-find-stars`

### Important parameters

- `brightness-threshold`.

- `filter-sources` (if needed)

- `srcextract-max-sources` (if needed)

### Description

Before AutoWISP can measure the brightnesses of the star in images, it needs to know the coordinates of the stars in the image. Finding a reliable sets of coordinates consists of two steps: running a source extraction algorithm to find an initial (not extremely precise) list of stars and their coordinates, and then matching these stars to an external high-precision and high-accuracy in order to find a function that can convert the coordinates of the stars in the catalogue to coordinates of the stars in the image.

The first of these steps is performed using the `wisp-find-stars` command. Behind the scenes, this command runs the source extraction algorithm from the FITSH package.

The browser interface provides a sandbox where source extraction can be tested interactively on any calibrated image with different values of `brightness-threshold`, `filter-sources`, and `srcextract-max-sources`.

## 3. Astrometry

Commnd: `wisp-solve-astrometry`

### Important Parameters

- `astrometry-catalog-max-magnitude`

- `max-srcmatch-distance`

- `astrometry-order`

- `min-match-fraction`

- `max-rms-distance`

- `frame-center-estimate`

- `frame-fov-estimate`

- `anet-api-key` if using the online astrometry.net service.

### Description

The `wisp-solve-astrometry` command finds the transformation that converts sky coordinates (right ascention and declination) into image coorditanes (x, y). This allows the use of external catalogue data for more precise positions of the sources than can be extracted from survey images and also the use of auxiliary data provided in the catalogue about each source, in the subsequent processing steps of the pipeline.

Astrometry is accomplished in 2 steps. First, AutoWISP uses astrometry.net to find an initial match between the few tens (to few hundred) brightest extracted source to the Gaia catalog, and then iteratively refines this match and the transformation parameters to match almost every single extracted source to its catalog counterpart. For wide-field images, this means thousands of matches that are used to find the transfomation parameters, allowing AutoWISP to model transformation involving signifiacant image distortion, which often occurs in wide-field images.

## 4. PRF/PSF Fitting

Command: `wisp-fit-star-shape`

### Important Parameters

- `photometry-catalog-max-magnitude`

- `subpixmap`: if processing images collected with a color camera

- `background-annulus`

- `shapefit-smoothing`

- `shape-grid`

- `shape-mode`

- `shape-terms-expression`

- `map-variables`

### Description

The first flux measurements AutoWISP generates for each image is one based on fitting for the shape of stars in the image and the flux (as the proportionality constant before the shape and the actual pixels belonging to that star on the image), called PSF or PRF fitting. This is accomplished by the `wisp-fit-star-shape` command.

Each point source once it is imaged by our observing system produces a particular distribution of light on the detector. The idea of PRF and PSF fitting is to model that distribution as some smooth parametric function centered on the projected source position that has an integral of one. For each star AutoWISP also fits for the amplitude that best scales this shape to the observed pixel values. The amplitude of course is then a measure of the flux of the source, while the parameters of the function specify its shape.

To review the terms:

> - Point Spread Function or PSF: PSF(dx, dy) is the amount of light that hits the surface of the detector offset by (dx, dy) from the projected position of the source. In order to actually predict what a particular detector pixel will measure, one computes the integral of the PSF times a sub-pixel sensitivity map over the area of the pixel.

> - Pixel Response Function or PRF: PRF(dx, dy) is the value that a pixel with a center offset by (dx, dy) from the projected source position will register. Note that dx and dy can be arbitrary real values and not just integers. The PRF already folds in its definition the sub-pixel sensitivity map, and other detector characteristics. Further, since the PRF is the PSF convolved with the sub-pixel sensitiity map it is generally smoother than the PSF and thus easier to model.

In this pipeline we use AstroWISP to perform PSF and PRF fitting. For the details of how this is done, see the AstroWISP documentation. Briefly, the PSF and PRF are modeled as piecewise bi-cubic functions with a number of free parameters. These parameters are in turn forced to vary smoothly as a function of source and image properties across sources and across images.

The information from PSF fitting is then used in the next step (Aperture Photometry)

## 5. Aperture Photometry

Command: `wisp-measure-aperture-photometry`

### Important Parameters

- `apertures`

- `subpixmap`: if processing images collected with a color camera

### Description

For each source, `wisp-measure-aperture-photometry` sums-up the flux in the image within a series of concentric circles centered on the projected source position. In order to properly handle the inevitable pixels that are partiallly within an aperture, knowledge of the distribution of light accross these pixels as well as the sub-pixel sensitivy map is required.

## 6. Fit PSF Map to Extracted Stars

Command: `wisp-fit-source-extracted-psf-map`

### Important Parameters

- `srcextract-psfmap-terms`

The various systematics removal steps in the pipeline can make use of information about the shape of stars in the images (a.k.a. PRF). That can come from two sources: the find stars step as part of the process fits for a PRF model, and obviously the PSF/PRF fitting step does as well. At the moment, the latter does not produce summary information (e.g. full-width at half maximum, ellpticity, etc.). Furthermore, in many applications, the PSF/PRF fitting step is too computationally expensive, and so it is replaced with a dummy fit. The result is that presently we rely on information from the find stars step.

Fitting the PRF of each individual star (as the find stars step does) results in quite noisy estimated parameters. In order to reduce that noise, this step of the pipeline fits a smooth function of image position and possibly stellar properties (brightness, color, etc. from the catalog) to each of the relevant parameters. Because the fit is based on a large number of stars, the noise gets averaged out to a large extent. The map is then evaluated for each star to produce higher-quality estimates of the PRF parameters.

The step is accomplished by the `wisp-fit-source-extracted-psf-map` command.

## 7. Magnitude fitting

Command: `wisp-fit-magnitudes`

### Important Parameters

- `single-photref-dr-fname`

- `correction-parametrization`

- `magfit-catalog-max-magnitude`

### Description

In ground based applications, the night sky is imaged through variable amount of atmosphere, which itself is subject to changes (i.e. clouds, humidity, etc.). In addition various instrumental effects are generally present. The purpose of the magnitude fitting step is to eliminate as much as possible effects that modify the measured source brightness within an image in a manner that depends smoothly on the properties of the source.

In short, a reference frame is selected (and later generated). Then for each individual frame (target frame from now on) a smooth multiplicative correction is derived that when applied to the brightness measurements in the target frame matches the brightness measurements in the reference frame as closely as possible.

In the pipeline this is actually done multiple times. The first time, a single frame which appears to be of very high quality (sharp PSF, high atmospheric transparency, dark sky etc.) is used as the reference frame. The corrected brightness measurements of the individua frames are then stacked to produce a much highe signal to noise “master reference frame”, which is then used in a second iteration of the magnitude fitting process and so on.

## 8. Creating Lightcurves

Command: `wisp-create-lightcurves`

### Important Parameters

- `lc-catalog-max-magnitude`

- `max-memory`

- `latitude-deg`

- `longitude-deg`

- `altitude-meters`

### Description

This is a simple transpose operation. In all previous steps, the photometry is extracted simultaneously for all sources in a given image or in a short series of images. In order to study each source’s individual variability, the measurements from all frames for that source must be collected together. This step simply performs that reorganization. For each catalogue source, all available measurements from the individual frames are collected in a file, possibly combined with earlier measurements from say a different but overlapping pointing of the telescope or with another instrumental set-up.

## 9. External Parameter Decorrelation (EPD)

Commands: `wisp-epd`, `wisp-generate-epd-statistics`

### Important Parameters

- `variables`

- `epd-datasets` (automatically set by BUI but not on the command line)

- `epd-terms-expression`

### Description

The `wisp-epd` command removes from each individual lightcurve the linear combintion of user specified instrumental and other time variable parameters that explain the most variance. Clearly care must be taken when selecting the parameters to decorrelate against, lest they vary on similar timescales as the target signal. If this happens, this step will highly distort if not eliminate the target signal.

The `wisp-generate-epd-statistics` command is used to calculate summary statistics for each lightcurve, showing how much scatter remains in it. This information is useb by the TFA step to select stars that would serve as good templates.

## 10. Trend Filtering Algorithm (TFA)

Commands: `wisp-tfa`, `wisp-generate-tfa-statistics`

### Important Parameters

- `tfa-datasets` (automatically set by BUI but not on the command line)

- `tfa-faint-mag-limit`

- `tfa-sqrt-num-templates`

In this step signals which are shared by mulitple stars are removed from each star’s lightcurve. The idea is that most instrumental effects will affect multiple stars in a similar way, and thus signals common to several sources are suspected of being instrumental, rather than real astrophysical variability. Again this steps has the potential to distort or eliminate target signals, so it should be used with care. If the shape of the target signal is known, there are versions of this procedure which tend to preserve it.