autowisp.image_calibration.master_flat_maker module — AutoWISP documentation

# autowisp.image_calibration.master_flat_maker module

## Class Inheritance Diagram

Define classes for creating master flat frames.

class autowisp.image_calibration.master_flat_maker.MasterFlatMaker(*, stamp_statistics_config=None, stamp_select_config=None, large_scale_smoother=None, cloud_check_smoother=None, master_stack_config=None)

  Bases: `MasterMaker`

  Specialize MasterMaker for making master flat frames.

  stamp_statistics_config

    Dictionary configuring how stamps statistics for stamp-based selection are extracted from the frames.

  stamp_select_config

    Dictionary configuring how stamp-based selection is performed. See keyword only arguments of `configure_stamp_selection()` for details.

  large_scale_smoother

    A `autowisp.image_smoothing.ImageSmoother` instance applied to the ratio of a frame to the reference large scale structure before applying it to the frame.

  cloud_check_smoother

    `autowisp.image_smoothing.ImageSmoother` instance used for cloud detection performed on the full flat frames after smoothing to the master large scale structure.

  master_stack_config

    Dictionary configuring how to stack the individual frames into a master.

  _master_large_scale

    The large scale structure imposed on all master flats. Dictionary with keys `values`, `stdev`, `mask` and `header`. If empty dictionary, nothing is imposed.

  Examples

      >>> import scipy.ndimage.filters
      >>> from autowisp.image_smoothing import\
      >>>     PolynomialImageSmoother,\
      >>>     SplineImageSmoother,\
      >>>     ChainSmoother,\
      >>>     WrapFilterAsSmoother
      >>>
      >>> #Stamp statistics configuration:
      >>> #  * stamps span half the frame along each dimension
      >>> #  * stamps are detrended by a bi-quadratic polynomial with at most
      >>> #    one rejection iteration, discarding more than 3-sigma outliers.
      >>> #  * for each stamp a iterative rejection mean and variance are
      >>> #    calculated with up to 3 iterations rejecting three or more
      >>> #    sigma outliers.
      >>> stamp_statistics_config = dict(
      >>>     fraction=0.5,
      >>>     smoother=PolynomialImageSmoother(num_x_terms=3,
      >>>                                      num_y_terms=3,
      >>>                                      outlier_threshold=3.0,
      >>>                                      max_iterations=3),
      >>>     average='mean',
      >>>     outlier_threshold=3.0,
      >>>     max_iter=3
      >>> )
      >>>
      >>> #Stamp statistics based selection configuration:
      >>> #  * Stamps with more than 0.1% of their pixels saturated are
      >>> #    discarded
      >>> #  * if variance vs mean quadratic fit has residual of more than 5k
      >>> #    ADU^2, the entire night is considered cloudy.
      >>> #  * individual frames with stamp mean and variance deviating more
      >>> #    than 2*(fit_residual) are discarded as cloudy.
      >>> #  * high master flat will be generated from frames with stamp mean
      >>> #    > 25 kADU, and low master flat from frames with stamp mean < 15
      >>> #    kADU (intermediate frames are discarded).
      >>> stamp_select_config = dict(max_saturated_fraction=1e-4,
      >>>                            var_mean_fit_threshold=2.0,
      >>>                            var_mean_fit_iterations=2,
      >>>                            cloudy_night_threshold=5e3,
      >>>                            cloudy_frame_threshold=2.0,
      >>>                            min_high_mean=2.5e4,
      >>>                            max_low_mean=1.5e4)
      >>>
      >>> #Large scale structure smoothing configuration. For each frame, the
      >>> #large scale struture is corrected by taking the ratio of the frame
      >>> #to the reference (median of all input frames), smoothing this ratio
      >>> #and then dividing by it. The following defines how the smoothing is
      >>> #performed:
      >>> #  * shrink by a factor of 4 in each direction (16 pixels gets
      >>> #    averaged to one).
      >>> #  * Performa a box-filtering with a half-size of 6-pixels using
      >>> #    median averaging
      >>> #  * Perform a bi-cubic spline interpolation smoothing of the
      >>> #    box-filtered image.
      >>> #  * Discard more than 5-sigma outliers if any and re-smooth (no
      >>> #    further iterations allowed)
      >>> #  * Re-interpolate the image back to its original size, using
      >>> #    bicubic interpolation (see zoom_image()).
      >>> #  * The resulting image is scaled to have a mean of 1 (no
      >>> #    configuration for that).
      >>> large_scale_smoother = ChainSmoother(
      >>>     WrapFilterAsSmoother(scipy.ndimage.filters.median_filter,
      >>>                          size=12),
      >>>     SplineImageSmoother(num_x_nodes=3,
      >>>                         num_y_nodes=3,
      >>>                         outlier_threshold=5.0,
      >>>                         max_iter=1),
      >>>     bin_factor=4,
      >>>     zoom_interp_order=3
      >>> )
      >>>
      >>> #Configuration for smoothnig when checking for clouds.
      >>> #After smoothing to the master large scale structure:
      >>> #  * extract a central stamp is extracted from each flat covering
      >>> #    3/4 of the frame along each dimension
      >>> #  * shrink the fractional deviation of that stamp from the master
      >>> #    by a factor of 4 in each dimension
      >>> #  * smooth by median box-filtering with half size of 4 shrunk
      >>> #    pixels
      >>> #  * zoom the frame back out by a factor of 4 in each dimension
      >>> #    (same factor as shrinking, no separater config), using
      >>> #    bi-quadratic interpolation.
      >>> cloud_check_smoother = WrapFilterAsSmoother(
      >>>     scipy.ndimage.filters.median_filter,
      >>>     size=8,
      >>>     bin_factor=4,
      >>>     zoom_interp_order=3
      >>> )
      >>>
      >>> #When stacking masters require:
      >>> #  * At least 10 input frames for a high intensity master and at
      >>> #    least 5 for a low intensity one.
      >>> #  * When creating the stack used to match large-scale structure,
      >>> #    use median averaging with outlier rejection of more than
      >>> #    4-sigma outliers with at most one reject/re-fit iteration.
      >>> #  * When creating the final master use median averaging with
      >>> #    outlier rejection of more than 2-sigma outliers in the positive
      >>> #    and more than 3-sigma in the negative direction with at most 2
      >>> #    reject/re-fit iterations. Create the master compressed and
      >>> #    raise an exception if a file with that name already exists.
      >>> master_stack_config = dict(
      >>>     min_pointing_separation=150.0,
      >>>     large_scale_deviation_threshold=0.05,
      >>>     min_high_combine=10,
      >>>     min_low_combine=5,
      >>>     large_scale_stack_options=dict(
      >>>         outlier_threshold=4,
      >>>         average_func=numpy.nanmedian,
      >>>         min_valid_values=3,
      >>>         max_iter=1,
      >>>         exclude_mask=MasterMaker.default_exclude_mask
      >>>     ),
      >>>     master_stack_options=dict(
      >>>         outlier_threshold=(2, -3),
      >>>         average_func=numpy.nanmedian,
      >>>         min_valid_values=3,
      >>>         max_iter=2,
      >>>         exclude_mask=MasterMaker.default_exclude_mask,
      >>>         compress=True,
      >>>         allow_overwrite=False
      >>>     )
      >>> )
      >>>
      >>> #Create an object for stacking calibrated flat frames to master
      >>> #flats. In addition to the stamp-based rejections:
      >>> #  * reject flats that point within 40 arcsec of each other on the
      >>> #    sky.
      >>> #  * Require at least 10 frames to be combined into a high master
      >>> #    and at least 5 for a low master.
      >>> #  * if the smoothed cloud-check image contains pixels with absolute
      >>> #    value > 5% the frame is discarded as cloudy.
      >>> make_master_flat = MasterFlatMaker(
      >>>     stamp_statistics_config=stamp_statistics_config,
      >>>     stamp_select_config=stamp_select_config,
      >>>     large_scale_smoother=large_scale_smoother,
      >>>     cloud_check_smoother=cloud_check_smoother,
      >>>     master_stack_config=master_stack_config
      >>> )
      >>>
      >>> #Create master flat(s) from the given raw flat frames. Note that
      >>> #zero, one or two master flat frames can be created, depending on
      >>> #the input images. Assume that the raw flat frames have names like
      >>> #10-<fnum>_2.fits.fz, with fnum ranging from 1 to 30 inclusive.
      >>> make_master_flat(
      >>>     ['10-%d_2.fits.fz' % fnum for fnum in range(1, 31)],
      >>>     high_master_fname='high_master_flat.fits.fz',
      >>>     low_master_fname='low_master_flat.fits.fz'
      >>> )

  __call__(frame_list, high_master_fname, low_master_fname, *, compress=True, allow_overwrite=False, stamp_statistics_config=None, stamp_select_config=None, master_stack_config=None, custom_header=None)

    Attempt to create high & low master flat from the given frames.

    Parameters:

      - frame_list – A list of the frames to create the masters from (FITS filenames).

      - high_master_fname – The filename to save the generated high intensity master flat if one is successfully created.

      - low_master_fname – The filename to save the generated low intensity master flat if one is successfully created.

      - compress – Should the final result be compressed?

      - allow_overwrite – See same name argument to `autowisp.image_calibration.fits_util.create_result()` .

      - stamp_statistics_config (dict) – Overwrite the configuration for extracting stamp statistics for this set of frames only.

      - stamp_select_config (dict) – Overwrite the stamp selection configuration for this set of frames only.

      - master_stack_config (dict) – Overwrite the configuration for how to stack frames to a master for this set of frames only.

      - custom_header – See same name argument to `autowisp.image_calibration.master_maker.MasterMaker.__call__()` .

    Returns:

      A dictionary splitting the input list of frames into

      > high:

      >   All entries from `frame_list` which were deemed suitable for inclusion in a master high flat.

      > low:

      >   All entries from `frame_list` which were deemed suitable for inclusion in a master low flat.

      > medium:

      >   All entries from `frame_list` which were of intermediate intensity and thus not included in any master, but for which no issues were detected.

      > colocated:

      >   All entries from `frame_list` which were excluded because they were not sufficiently isolated from their closest neighbor to guarantee that stars do not overlap.

      > cloudy:

      >   All entries from `frame_list` which were flagged as cloudy either based on their stamps or on the final full-frame cloud check.

    Return type:

      dict

  __init__(*, stamp_statistics_config=None, stamp_select_config=None, large_scale_smoother=None, cloud_check_smoother=None, master_stack_config=None)

    Create object for creating master flats out of calibrated flat frames.

    Parameters:

      - stamp_statistics_config – A dictionary mith arguments to pass to `configure_stamp_statistics()`.

      - stamp_select_cofig – A dictionary with arguments to pass to `configure_stamp_selection()`.

      - large_scale_smoother – An ImageSmoother instance used when matching large scale structure of individual flats to master.

      - cloud_check_smoother – An ImageSmoother instance used when checkng the full frames for clouds (after stamps are checked).

      - master_stack_config – Configuration of how to stack frames to a master after matching their large scale structure. See example in class doc-string for details.

    Returns:

      None

  _classify_from_stamps(frame_list, stamp_statistics_config, stamp_select_config)

    Classify frames by intensity: (high/low/ntermediate), or flag as cloudy.

    Parameters:

      frame_list – The list of frames to create masters from.

    Returns:

      filename list:

        The list of high intensity non-cloudy frames.

      filename list:

        The list of low intensity non-cloudy frames.

      filename list:

        The list of medium intensity non-cloudy frames.

      filename list:

        The list of frames suspected of containing clouds in their central stamps.

    Return type:

      (tuple)

  _find_colocated(frame_list)

    Split the list of frames into well isolated ones and co-located ones.

    Parameters:

      frame_list – A list of the frames to create the masters from (FITS filenames).

    Returns:

      filename list:

        The ist of frames sufficiently far in pointing from all other frames.

      filename list:

        The list of frames which have at least one other frame too close in pointing to them.

    Return type:

      (tuple)

  _get_stamp_statistics(frame_list, **stamp_statistics_config)

    Get relevant information from the stamp of a single input flat frame.

    Parameters:

      frame_list – The list of frames for which to extract stamp statistics.

    Returns:

      An array with fields called `mean`, `variance` and `num_averaged` with the obvious meanings.

    Return type:

      numpy field array

  configure_stamp_selection(*, max_saturated_fraction=None, var_mean_fit_threshold=None, var_mean_fit_iterations=None, cloudy_night_threshold=None, cloudy_frame_threshold=None, min_high_mean=None, max_low_mean=None)

    Configure stamp-based frame selection and high/low split.

    Parameters:

      - max_saturated_fraction – The maximum fraction of stamp pixels allowed to be saturated before discarding the frame.

      - cloudy_night_threshold – The maximum residual of the variance vs mean quadratic fit before a night is declared cloudy. If None, this check is disabled.

      - cloudy_frame_threshold – The maximum deviation in units of the RMS residual of the fit an individual frame’s variance vs mean from the var(mean) quadratic fit before the frame is declared cloudy.

      - min_high_mean – The minimum mean of the stamp pixels in order to consider the frame high intensity.

      - max_low_mean – The maximum mean of the stamp pixels in order to consider the frame low intensity. Must not overlap with min_high_mean.

    Returns:

      None

  configure_stamp_statistics(*, fraction=None, smoother=None, outlier_threshold=None, max_iter=None, average=None)

    Configure extraction of stamp satistics for rejection & high/low split.

    Any arguments left as None are not updated.

    Parameters:

      - fraction – The fraction of the frame size that is included in the stamp along each dimension (i.e. fraction=0.5 means 1/4 of all frame pixels will be incruded in the stamp).

      - smoother – An ImageSmoother instance used used for de-trending the stamps before extracting statistics

      - outlier_threshold – The threshold in units of RMS deviation from the average above which pixels are considered outliers from the de-trending function and discarded from its fit.

      - max_iter – The maximum number of fit/reject iterations to perform before declaring the de-trending function final.

      - average – How to compute the average. Should be either `'mean'` or `'median'`.

    Returns:

      None

  prepare_for_stacking(image)

    Match image large scale to self._master_large_scale if not empty.

    Parameters:

      image – The image to transform large scale structure of.

    Returns:

      If `_master_large_scale` is an empty dictionary, this is just `image`. Otherwise, `image` is transformed to have the same large scale structure as `_master_large_scale`, while the small scale structure is preserved. `image` is also checked for clouds, and discarded if cloudy.

    Return type:

      2-D array