========================================
STIS Pixel-Based CTI-Correction Tutorial
========================================
Thank you for trying out this **beta** release.  Please note that STScI is not responsible for any damages resulting from the use of this software.

Please send any feedback to biretta@stsci.edu.

Introduction
============
The ``stis_cti`` package implements the Anderson & Bedin (`PASP 2010, 122: 1035-1064 <http://adsabs.harvard.edu//abs/2010PASP..122.1035A>`_) pixel-based CTI-correction on HST/STIS CCD data.  It performs bias corrections on the data, calls ``StisPixCteCorr()`` on the data, and finishes running the remaining ``CalSTIS`` processing steps.

The code also creates a CTI-corrected super-dark reference file to use in this processing from component dark files.  When run, the user will be prompted to retrieve these component darks from the MAST archive if they have not already been retrieved.  The code makes use of the ``refstis`` package to generate super-darks from the component darks.

Note that not all observing modes are currently supported.  The code will not reprocess data taken in subarray or binned modes, with a non-D amplifier, in ACQ mode, or from before the STIS recovery of May 2009.

System Requirements
===================
This script is designed to run on top of the Ureka software suite on the UNIX/Linux and MacOS X operating systems.

Some components in older Ureka installations may be out of date and require updating (including ``astropy``), or a reinstallation of the Ureka package.

Installation
============
First, launch Ureka (available via http://ssb.stsci.edu/ureka/):

::
   
   ur_setup
   
::

Then, install the requiered packages within the Ureka environment:

::
   
   pip install stis-cti --pre
   
::

To optionally install a local CRDS cache in a common location on your system, please see:

https://hst-crds.stsci.edu/docs/cmdline_bestrefs/

Directory Structure
===================
To run the correction, you will need directories to hold dark files, pipeline reference files, reference files generated by the script, and science data.  Many of these directories may be shared between projects, thus reducing redundant computations.

Here's one possible layout:

* base_dir/  
   - science/ — contains at least the pipeline's "uncalibrated" products that are to be corrected  
   - darks/ — component darks (``_flt.fits``); their CTI-corrected counterparts (``_cte.fits``) will be placed in here  
   - ref/ — location for CTI-corrected super-darks produced by the ``stis_cti`` script  
   - oref/ — pipeline reference files (specifying the ``--crds_update`` option creates this directory at ref/references/hst/oref/)

For example::

  mkdir stis_cti_corrected
  mkdir stis_cti_corrected/my_science
  mkdir stis_cti_corrected/darks
  mkdir stis_cti_corrected/ref

By default, the command-line interface assumes that there are ``ref/`` and ``darks/`` directories parallel to the ``science/`` directory (of any name), though alternate configurations may be specified.

Output files in the ``science/`` directory from a previous run of stis_cti will need to be deleted or moved, unless the ``--clean`` option is specified.  
Specifying ``--clean_all`` will force the script to recreate super-darks and re-run the CTI-correction on all needed component darks.

If the ``--crds_update`` option is not selected, then the ``$oref`` shell variable must be set before launching the script (including the trailing ``'/'``)::

  In tcsh:  setenv oref /path/to/oref/  
  In bash:  export oref='/path/to/oref/'

This directory must contain the needed reference files specified in the science and component dark headers, as is typical for running ``CalSTIS``.

Command-line Usage
==================
The easiest way to invoke the correction is from the ``UNIX`` shell.

Note that running this script requires an internet connection to the MAST archive site.

::

  ur_setup
  stis_cti --help

::

  usage: stis_cti [-h] [-d DARK_DIR] [-r REF_DIR] [-n NUM_PROCESSES]
                  [-p PCTETAB] [--crds_update] [--clean] [--clean_all]
                  [-v VERBOSE_LEVEL]
                  [SCIENCE_DIR]
  
  Run STIS/CCD pixel-based CTI-correction on data specified in SCIENCE_DIR.
  Uncorrected component darks are read from DARK_DIR, and corrected component
  darks are written there too. Corrected super-darks are read from and stored to
  REF_DIR.
  
  positional arguments:
    SCIENCE_DIR       directory containing RAW science data (default="./")
  
  optional arguments:
    -h, --help        show this help message and exit
    -d DARK_DIR       directory of dark FLT data
                      (default="[SCIENCE_DIR]/../darks/")
    -r REF_DIR        directory of CTI-corrected reference files
                      (default="[SCIENCE_DIR]/../ref/")
    -n NUM_PROCESSES  maximum number of parallel processes to run (default=15);
                      number of available CPU cores on your system = 40
    -p PCTETAB        name of PCTETAB to use in pixel-based correction
                      (default="[REF_DIR]/[MOST_RECENT]_pcte.fits" or package's
                      default PCTETAB)
    --crds_update     update and download $oref files
    --clean           remove intermediate and final products from previous runs
                      of this script ('*.txt' files are skipped and clobbered)
    --clean_all       '--clean' + remove previous super-darks and CTI-corrected
                      component darks
    -v VERBOSE_LEVEL  verbosity ({0,1,2}; default=1)
  
  Author: Sean Lockwood; Version: 0.4_beta6

The script is designed to run the pixel-based correction in parallel on the component darks, and in parallel on the science files.  The maximum number of processes may be specified via the '``-n #``' option.

A typical call looks like::

  stis_cti -vv -n 15 --crds_update my_science_dir/

(Assuming that there are ``ref/`` and ``darks/`` directories parallel to ``my_science_dir/``)

---------------------------------------------

While ``stis_cti`` will tell you what component darks are needed, you can also determine this ahead of time:

.. parsed-literal::

  archive_dark_query data/\*_raw.fits
  
  Querying MAST archive for dark and anneal program IDs...
  Querying MAST archive for darks...
  Parsing archive results...
  
  Download darks via this link:
  
  http://archive.stsci.edu/hst/search.php?sci_instrume=STIS&sci_instrument_config=STIS%2FCCD&sci_targname=DARK&sci_aec=C&resolve=don%27tresolve&sci_data_set_name=OC4W6XH3Q%2COC4W6YHBQ%2COC4W6ZP2Q%2COC4W70PCQ%2COC4W71TEQ%2COC4W72TOQ%2COC4W73X8Q%2COC4W74XJQ%2COC4W75D0Q%2COC4W76DCQ%2COC4W77HHQ%2COC4W78I0Q%2COC4W79A5Q%2COC4W7AADQ%2COC4W7BFGQ%2COC4W7CF9Q%2COC4W7DJNQ%2COC4W7EJRQ%2COC4W7FOAQ%2COC4W7GO4Q%2COC4W7HSNQ%2COC4W7ISUQ%2COC4W7JXEQ%2COC4W7KXAQ%2COC4W7LGRQ%2COC4W7MGWQ%2COC4W7NA1Q%2COC4W7OA8Q%2COC4W7PM6Q%2COC4W7QMDQ%2COC4W7RTJQ%2COC4W7STNQ%2COC4W7TX4Q%2COC4W7UXDQ%2COC4W7VIKQ%2COC4W7WIRQ%2COC4W7XNJQ%2COC4W7YNRQ%2COC4W7ZSZQ%2COC4W80TMQ%2COC4W81A4Q%2COC4W82AGQ%2COC4W83NMQ%2COC4W84O1Q%2COC4W85SRQ%2COC4W86SZQ%2COC4W87XWQ%2COC4W88YHQ%2COC4W89D6Q%2COC4W8ADJQ%2COC4W8BHWQ%2COC4W8CI2Q%2COC4W8DNUQ%2COC4W8EOAQ%2COC4W8FBPQ%2COC4W8GBTQ&max_records=50000&max_rpp=5000&ordercolumn1=sci_start_time&action=Search

Python Usage
============
If you wish to run the ``stis_cti`` script from within Python (say within another script), it may be called via:

.. code:: python

  import stis_cti
  
  # Explicit directories must be specified when run through the Python interface.
  stis_cti.stis_cti('science_dir/', 'dark_dir/', 'ref_dir/', 5, verbose=True)
      # where 5 is num_processes

Note that there are other options available via the Python interface, but these have not yet been fully vetted.

Output Products
===============
The following data products are output by ``stis_cti``:

+----------------------------------+------------------+-----------------------+
| **Product**                      | **Original Ext** | **CTI-Corrected Ext** |
+==================================+==================+=======================+
| Bias- and CTI-corrected science  | n/a              | CTE [#cte]_           |
| (intermediate product)           |                  |                       |
+----------------------------------+------------------+-----------------------+
| Cosmic ray rejected, flat-       | CRJ              | CRC                   |
| fielded science                  |                  |                       |
+----------------------------------+------------------+-----------------------+
| Flat-fielded science             | FLT              | FLC                   |
+----------------------------------+------------------+-----------------------+
| 1-D extracted spectra for        | X1D              | X1C                   |
| individual imsets:               |                  |                       |
|                                  |                  |                       |
| * Aperture extracted, background |                  |                       |
|   subtracted, flux and           |                  |                       |
|   wavelength calibrated spectra  |                  |                       |
+----------------------------------+------------------+-----------------------+
| 2-D spectral and direct images   | X2D              | X2C                   |
| for individual imsets:           |                  |                       |
|                                  |                  |                       |
| * Rectified, wavelength and      |                  |                       |
|   flux calibrated first order    |                  |                       |
|   spectra or                     |                  |                       |
| * Geometrically corrected        |                  |                       |
|   imaging data.                  |                  |                       |
+----------------------------------+------------------+-----------------------+
| 1-D extracted spectra from       | SX2              | S2C                   |
| from summed (REPEATOBS) or       |                  |                       |
| cosmic ray rejected (CRSPLIT)    |                  |                       |
| images.                          |                  |                       |
+----------------------------------+------------------+-----------------------+
| 2-D rectified direct or spectral | SX1              | S1C                   |
| images from summed (REPEATOBS)   |                  |                       |
| or cosmic ray rejected (CRSPLIT) |                  |                       |
| images.                          |                  |                       |
+----------------------------------+------------------+-----------------------+

.. [#cte] Where ``CTE`` files are bias- and CTI-corrected intermediate products.

.. figure:: obr101010_comparison.png
   :width: 100 %
   :alt: obr101010_crj comparison
   :align: center
   
   A comparison of part of a STIS cosmic-ray rejected image.  Note the CTI trails are removed in both the science and super-dark data used to generate the ``_crc`` file.

Advanced Topics
===============

Custom Super-Darks
------------------
The ``stis_cti`` script first determines if the ``DARKFILE`` specified in each science file's header is already CTI-corrected (assuming it exists) by checking that the ``ext=0`` header keyword ``PCTECORR=='COMPLETE'``.  If it is, then the script will not attempt to replace it.  This allows users the flexibility to create their own super-darks via the ``refstis`` package with their own parameters and/or input data (e.g. herringbone-corrected_ data files).

.. _herringbone-corrected: http://stis2.sese.asu.edu/

.. code:: python

  import refstis
  import glob
  from astropy.io import fits
  
  # First, run stis_cti.StisPixCteCorr() on _flt.fits dark files 
  # to produce _cte.fits files.
  month_files = glob.glob('annealing_month/*_cte.fits')
             # Assuming only the annealing month's darks are selected
  refstis.basedark.make_basedark(month_files, refdark_name='basedark_drk.fits')
  week_files = glob.glob('my_week/*_cte.fits')
  refstis.weekdark.make_weekdark(week_files, refdark_name='weekdark_drk.fits',
      thebasedark='basedark_drk.fits')
  
  # You must mark the new weekdarks as being CTI-corrected, for example:
  fits.setval('weekdark_drk.fits', 'PCTECORR', value='COMPLETE')
  
  # Point the science files at the new weekdark:
  # Define $stisref to point to the directory containing the weekdark in the shell.
  #   In tcsh:  setenv stisref /path/to/my_dir/
  #   In bash:  export stisref='/path/to/my_dir/'
  
  # On each science file:
  fits.setval('science/filename_raw.fits', 'DARKFILE', value='stisref$weekdark_drk.fits')
  
Now when ``stis_cti`` is run on the science directory, it won't try to recreate the super-dark, 
but will CTI-correct the science images.

CRDS Updates
------------
Oftentimes, the reference files specified in a dataset get replaced within the pipeline.  This is especially true of super-biases and super-darks produced in the months following the execution of an observation.  When new reference files are available, you may re-retrieve the data from MAST.  Alternatively, the ``crds`` script now supports updating header keywords and downloading required reference files automatically.

When run with the ``--crds_update`` option, the script will update header keywords and download reference files to the ``$oref`` directory, if it is writable.  Otherwise, one will be created within the ``ref/`` directory.

Note that this option is not currently compatible with specifying one's own super-dark, as the user-specified ``DARKFILE`` keyword will be over-written.  As a workaround, you may run the CRDS bestref script manually and then override the ``DARKFILE`` keyword as desired.

To run CRDS bestref manually, see https://hst-crds.stsci.edu/docs/cmdline_bestrefs/ or re-retrieve your data and best reference files from MAST.  You may also wish to setup a common local CRDS cache of reference files to avoid redundancy.

PCTETAB Updates
---------------
The ``stis_cti`` package includes the ``v0.1_beta`` ``PCTETAB`` reference file, which specifies the parameters necessary to run the pixel-based correction on STIS data.  If this file is updated, or if an advanced user wishes to modify the file to run the correction differently, the new version may be placed in the ``ref/`` directory.  (If multiple ``PCTETAB`` files are present, the one with the last alphabetical name will be used.)

To completely re-run the correction, you can delete any needed basedarks/weekdarks in the ``ref/`` directory and any needed ``_cte.fits`` files in the ``darks/`` directory by specifying the ``--clean_all`` option.
