------
Frames
------

The basic unit of data processing is the frame. Each frame is a free-form mapping from strings to data of a type derived from G3FrameObject. In general, they behave as Python dictionaries. Wrappers are provided for Python plain-old-data types (numbers, strings, etc.) so that they can be used directly.

Frames have fast serialization to and from disk (see the G3Reader and G3Writer modules in :doc:`fileio`) with built-in data integrity protection. This serialization is accessible programmatically in Python through the Python pickle libraries, which are overridden to use the internal serialization.

Each frame has a type defined under the :py:class:`~spt3g.core.G3FrameType` namespace. These are meant to indicate different types of data (calibration vs. scan data, for example) and, in general, many types of frames will be interleaved in the same data stream. A good general rule for whether data should be in a different frame type is to consider the rates at which the data change: data that change at the same speed (e.g. bolometer data and pointing) should share a frame, while data that change at different speeds (e.g. calibration constants and bolometer data) should be separated.

Each frame also has a ``._filename`` attribute, which is set when frames are read in using the :py:class:`~spt3g.core.G3Reader` pipeline module with the ``track_filename`` option set to ``True``. This attribute is not serialized with the frame, so will be an empty string when used with pickling or multiprocessing.  This attribute is fragile and should not be relied upon for production code.

A brief description of the intention of each frame type follows along with a table containing a representative minimal set of data likely to be contained by a frame of a given type. The list is neither exhaustive nor truly minimal: other data can and will be present and some of the data listed here may have been removed or renamed. Neither is this an exhaustive list of frame types: any single character code can be [ab]used as a frame type for special purpose tools.

.. contents:: Contents
   :local:

Scan
====

Scan frames are the most common unit of data for analysis. They contain the data from a single left-to-right (or right-to-left) scan of the telescope. Where quantities are bolometer-indexed, the names match those in the Wiring map.

===================	====================	===========
Key			Type			Description
===================	====================	===========
ACUStatus		ACUStatusVector		Sequences of gcp.ACUStatus objects containing the ACU state readout during the scan
SourceName		G3String		Name of the source being observed
BoresightAz		G3Timestream		Telescope azimuth as a function of time
BoresightEl		G3Timestream		Telescope elevation as a function of time
RawTimestreams_I	G3TimestreamMap		Raw (counts) timestreams from detectors, indexed by bolometer, I modulation
RawTimestreams_Q	G3TimestreamMap		Raw (counts) timestreams from detectors, indexed by bolometer, Q modulation
CalTimestreams		G3TimestreamMap		Bolometer timestreams with some useful calibration applied (not present in raw data)
TimestreamWeights	G3MapDouble		Floating point weights for each timestream, indexed by bolometer (not present in raw data)
ScanNumber		G3Int			Sequence number of the current scan within an observation
Flags			G3MapVectorString	List of detectors with exceptional conditions. Indexed by bolometer ID. Each element is a list of strings that provides a description of the condition[s] that caused the detector to be flagged (not present in raw data).
Turnaround		G3Bool			Present and set to True if scan is a turnaround or otherwise is not a constant velocity scan. Absent otherwise.
TrakerStatus		TrackerStatus		Composite tracker status information at all points during the scan.
DfMuxHousekeeping	DfMuxHousekeepingMap		Tree containing housekeeping data and configuration from the boards connected to the readout system, indexed by board IP address
CalibratorOn		G3Timestream		Timestream of the calibrator sync signal. Set to 1 if the sync line is high and 0 otherwise. NaN if no readout.
GCPFeatureBits		G3VectorString		Strings describing the GCP flags field.
===================	====================	===========

PipelineInfo
============

PipelineInfo frames contain information on the processing applied to make the remainder of the data. They typically occur at the beginning of the data stream and are automatically inserted by G3Pipeline, if not previously present. The frame will contain at least one element of type G3PipelineInfo that gives information on the version of the software that made the data and the configuration of all modules and/or segments added to the pipeline. Other data (G3PipelineInfo from previous scripts, configuration information added by other modules) may also be present.

==============================		===============	===========
Key					Type		Description
==============================		===============	===========
22-May-2019:18:42:15.335969000		G3PipelineInfo	Configuration information of the pipeline that produced an object. Added automatically. The key name is a timestamp indicating the time at which the software began running.
==============================		===============	===========

Timepoint
=========

Timepoint frames include sample-by-sample data from the mux system. They are emitted directly by the DfMuxBuilder as part of primary data acquisition and are generally not seen outside the South Pole or the lab. These are transformed into Scan_ frames by DfMuxCollator.

===================	===============	===========
Key			Type		Description
===================	===============	===========
EventHeader		G3Time		Time of the sample
DfMux			DfMuxMetaSample	Tree containing data from the boards connected to the readout system, indexed by board IP address
===================	===============	===========

Housekeeping
============

Contains housekeeping data. Issued periodically when housekeeping data is taken. Like Timepoint_ frames, these are rolled into Scan_ frames during processing and do not appear in general in stored data.

===================	====================		===========
Key			Type				Description
===================	====================		===========
DfMuxHousekeeping	DfMuxHousekeepingMap		Tree containing housekeeping data and configuration from the boards connected to the readout system, indexed by board IP address
===================	====================		===========

Map
===

Contains either the result of the map maker or the input to simulation.

==========================	======================	===========
Key				Type			Description
==========================	======================	===========
Id				G3String		A string identifying the map for the various processing steps
T				G3SkyMap		A map storing the intensity information (could be sky intensity or sky intensity x weight).  In the case of maps that store abstract information like apodization masks or point source masks, the data will also be stored under the T key.  The motivation being it makes it easy to have G3Modules operating on maps also work on these.
Q				G3SkyMap		A map storing the Q polarization information (could be sky Q or sky Q x weight)
U				G3SkyMap		A map storing the u polarization information (could be sky U or sky U x weight)
Wpol				G3SkyMapWeights		If the frame contains polarized information, this stores the t/q/u covariances scaled by the individual detector weights
Wunpol				G3SkyMapWeights		This stores the unpolarized weight information
==========================	======================	===========



Calibration
===========

This frame contains all measured calibration information (pointing, response, etc.) that may change when remeasured. It does *not* include static information describing how the instrument is set up (see Wiring_ below).

==========================	======================	===========
Key				Type			Description
==========================	======================	===========
BolometerProperties		BolometerPropertiesMap	Measured non-configuration-dependent calibration properties of the instrument (pointing, pol efficiency, etc.), indexed by bolometer
NominalBolometerProperties	BolometerPropertiesMap	As above, but what those properties were meant to be.
TimeConst			G3MapDouble		Time constants of the detectors. These can change with the bias point. Should perhaps be moved to the InstrumentStatus frame.
RCW38FluxCalibration		G3MapDouble		Observed flux of RCW38 per detector as a fraction of the calibrator response.
CalibratorResponse		G3MapDouble		Observed response to the most recent calibrator observation for each detector in Watts.
CalibratorResponseSN		G3MapDouble		Signal to noise (in sigma) of the most recent calibrator observation in sigma.
==========================	======================	===========

Observation
===========

Indicates global observation parameters. Changes at the beginning of a new observation, though, as with all metadata, repeat observation frames may appear during processing.

=========================	======================	===========
Key				Type			Description
=========================	======================	===========
SourceName			G3String		Name of the source being observed
ObservationNumber		G3Int			Sequence number of the current observation since we started recording such things
=========================	======================	===========

Wiring
======

Gives the description of how the system is wired: notably, the connection between board serial number, module, channel and a bolometer ID.

=============	==============	======================================
Key		Type		Description
=============	==============	======================================
WiringMap	DfMuxWiringMap	Digest of the pydfmux channel mappings
ReadoutSystem	G3String	Description of the type of readout system employed. Set to "DfMux" for SPTpol-style readout and "ICE" for 3G-style readout.
=============	==============	======================================

GcpSlow
=======

Holds all the GCP data sampled once per second. Like Timepoint_ and Housekeeping_ frames, these data are consolidated in the Scan frames and do not appear in final data products. The data stored here begins as a strict copy of the GCP register map (see the GCP documentation for details, a few notable entries are summarized below). A few other keys are added transiently in the course of generating Scan_ frames.

=========	================	======================================
Key		Type			Description
=========	================	======================================
array		G3MapFrameObject	Most of the data stored by GCP
antenna0	G3MapFrameObject	Telescope pointing information
=========	================	======================================



EndProcessing
=============

EndProcessing is a special-purpose frame emitted implicitly by G3Pipeline at the end of processing. No further frames will occur after this and reception of an EndProcessing frame is intended as a signal to modules that they should clear any internally buffered data and clean up.

EndProcessing frames should, in general, contain no data.

