Numina Pipeline Concepts — Numina 0.20.dev0 documentation

# Numina Pipeline Concepts

## Instrument

## Observing Modes

Each Instrument has a list of predefined types of observations that can be carried out with it. Each Observing Mode is defined by:

> - The configuration of the Telescope
> - The configuration of the Instrument
> - The type of processing required by the images obtained durinf the observation

Some of the observing modes of a Instrument are Scientific, that is, modes devoted to obtain data to perform scientific analysis. Other modes are devoted to Calibration; these modes produce data required to correct the scientific images from the effects of the Instrument, the Telescope and the atmosphere.

## Recipes

A recipe is a method to process the images obtained in a particular observing mode. Recipes in general require (as inputs) the list of raw images obtained during the observation. Recipes can require other inputs (calibrations), and those inputs can be the outputs of other recipes.

Images obtained in a particular mode are processed by one recipe.

## Pipelines

A pipeline represents a particular mapping between the observing modes and the reduction algorithms that process each mode. Each instrument has at least one pipeline called default. It may have other pipelines for specific purposes.

## Products, Requirements and Data Types

A recipe announces its required inputs as `Requirement` and its outputs as `Result`.

Both Results and Requirements have a name and a type. Types can be plain Python types or defined by the developer.

## Format of the input files

The default format of the input and output files is YAML, a data serialization language.

### Format of the Observation Result file

This file contains the result of an observation. It represents an `ObservationResult` object.

The contents of the object are serialized as a dictionary with the following keys:

id: not required, integer, defaults to 1
  Unique identifier of the observing block
instrument: required, string
  Name of the instrument, as it appears in the instrument file (see below)
mode: required, string
  Name of the observing mode
children: not required, list of integers, defaults to empty list
  Identifications of nested observing blocks
frames: required, list of file names
  List of raw images

    id: 21
    instrument: EMIR
    mode: nb_image
    children: []
    frames:
    - r0121.fits
    - r0122.fits
    - r0123.fits
    - r0124.fits
    - r0125.fits
    - r0126.fits
    - r0127.fits
    - r0128.fits
    - r0129.fits
    - r0130.fits
    - r0131.fits
    - r0132.fits

### Format of the requirement file (version 1)

    version: 1
    products:
      EMIR:
       - {id: 1, content: 'file1.fits', type: 'MasterFlat', tags: {'filter': 'J'}, ob: 200}
       - {id: 4, content: 'file4.fits', type: 'MasterBias', tags: {'readmode': 'cds'}, ob: 400}
      MEGARA:
       - {id: 1, content: 'file1.fits', type: 'MasterFlat', tags: {'vph': 'LR1'}, ob: 1200}
       - {id: 2, content: 'file2.yml', type: 'TraceMap', tags: {'vph': 'LR2', 'readmode': 'fast'}, ob: 1203}
    requirements:
      EMIR:
        default:
           TEST6:
              pinhole_nominal_positions: [ [0, 1], [0 , 1]]
              box_half_size: 5
           TEST9:
              median_filter_size: 5
    MEGARA:
        default:
           mos_image: {}

### Format of the requirement file

Warning

This section documents a deprecated format

Deprecated since version 0.14.0.

This file contains configuration parameters for the recipes that are not related to the particular instrument used.

The contents of the file are serialized as a dictionary with the following keys:

requirements: required, dictionary
  A dictionary of parameter names and values.
logger: optional, dictionary
  A dictionary used to configure the custom file logger

    requirements:
      master_bias: master_bias-1.fits
      master_bpm: bpm.fits
      master_dark: master_dark-1.fits
      master_intensity_ff: master_flat.fits
      nonlinearity: [1.0, 0.0]
      subpixelization: 4
      window:
      - [800, 1500]
      - [800, 1500]
    logger:
      logfile: processing.log
      format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
      enabled: true