Gridded Data Format
===================

This is a pre-release of version 1.0 of this format.  Lots of formats have come
before, but this one is simple and will work with yt; the idea is to create an
import and export function in yt that will read this, so that other codes (such
as ZEUS-MP) can export directly to it or convert their data to it, and so that
yt can export to it from any format it recognizes and reads.

Caveats and Notes
-----------------

#. We avoid having many attributes on many nodes, as access can be quite slow
#. Cartesian data only for now
#. All grids must have the same number of ghost zones.
#. If “/grid_parent” does not exist, parentage relationships will be
   reconstructed and assumed to allow multiple grids
#. No parentage can skip levels
#. All grids are at the same time
#. This format is designed for single-fluid calculations (with color fields)
   but it should be viewed as extensible to multiple-fluids.
#. All fluid quantities are assumed to be in every grid, filling every zone.  Inside
   a given grid, for a given particle type, all the affiliated fields must be the
   same length.  (i.e., dark matter's velocity must be the same in all dimensions.)
#. Everything is in a single file; for extremely large datasets, the user may
   utilize HDF5 external links to link to files other than the primary.  (This
   enables, for instance, Enzo datasets to have only a thin wrapper that creates
   this format.)
#. All fluid fields in this version of the format are assumed to have the
   dimensionality of the grid they reside in plus any ghost zones, plus any
   additionaly dimensionality required by the staggering property.
#. Particles may have dataspaces affiliated with them.  (See Enzo's
   OutputParticleTypeGrouping for more information.)  This enables a light
   wrapper around data formats with interspersed particle types.
#. Boundary conditions are very simply specified -- future revisions
   will feature more complicated and rich specifications for the boundary.

Furthermore, we make a distinction between fluid quantities and particle
quantities.  Particles remain affiliated with grid nodes.  Positions of
particles are global, but this will change with future versions of this
document.

Format Declaration
------------------

The file type is HDF5.  We require version 1.8 or greater.  At the root level,
this group must exist: ::

   /gridded_data_format

This must contain the (float) attribute ``format_version``.  This document
describes version 1.0.  Optional attributes may exist:

``data_software``
   string, references the application creating the file, not the
   author of the data
``data_software_version``
   string, should reference a unique version number
``data_author``
   string, references the person or persons who created the data,
   should include an email address
``data_comment``
   string, anything about the data

Top Level Nodes
---------------

At least five top-level groups must exist, although some may be empty. ::

   /gridded_data_format
   /data
   /simulation_parameters
   /field_types
   /particle_types

Additionally, the grid structure elements must exist.  The 0-indexed index into this array
defines a unique "Grid ID".

``/grid_left_index``
   (int64, Nx3): global, relative to current level, and only the active region
``/grid_dimensions``
   (int64, Nx3): only the active regions
``/grid_level``
   (int64, N): level, indexed by zero
``/grid_particle_count``
   (int64, N): total number of particles.  (May change in subsequent versions.)
``/grid_parent_id``
   (int64, N): optional, may only reference a single parent

Grid Fields
-----------

Underneath ``/data/`` there must be entries for every grid, of the format
``/data/grid_%010i``.  These grids need no attributes, and underneath them
datasets live.

Fluid Fields
++++++++++++

For every grid we then define ``/data/grid_%010i/%(field)s``.

Where ``%(field)s`` draws from all of the fields defined.  We define no
standard for which fields must be present, only the names and units.  Units
should always be ''proper'' cgs (or conversion factors should be supplied, below), and
field names should be drawn from this list, with these names.  Not all fields
must be represented.  Field must extend beyond the active region if ghost zones
are included.  All pre-defined fields are assumed to be cell-centered unless this
is overridden in ``field_types``.

  * ``density`` (g/cc)
  * ``temperature`` (K)
  * ``specific_thermal_energy`` (erg/g)
  * ``specific_energy`` (erg/g, includes kinetic and magnetic)
  * ``magnetic_energy`` (erg/g)
  * ``velocity_x`` (cm/s)
  * ``velocity_y`` (cm/s)
  * ``velocity_z`` (cm/s)
  * ``species_density_%s`` (g/cc) where %s is the species name including ionization
    state, such as H2I, HI, HII, CO, "elec" for electron
  * ``mag_field_x``
  * ``mag_field_y``
  * ``mag_field_z``

Particle Fields
+++++++++++++++

Particles are more expensive to sort and identify based on "type" -- for
instance, dark matter versus star particles.  The particles should be separated
based on type, under the group ``/data/grid_%010i/particles/``.

The particles group will have sub-groups, each of which will be named after the
type of particle it represents.  We only specify "dark_matter" as a type;
anything else must be specified as described below.

Each node, for instance ``/data/grid_%010i/particles/dark_matter/``, must
contain the following fields:

  * ``mass`` (g)
  * ``id``
  * ``position_x`` (in physical units)
  * ``position_y`` (in physical units)
  * ``position_z`` (in physical units)
  * ``velocity_x`` (cm/s)
  * ``velocity_y`` (cm/s)
  * ``velocity_z`` (cm/s)
  * ``dataspace`` (optional) an HDF5 dataspace to be used when opening
    all affiliated fields.   If this is to be used, it must be appropriately set in
    the particle type definition.  This is of type ``H5T_STD_REF_DSETREG``.
    (See Enzo's OutputParticleTypeGrouping for an example.)

Additional Fields
+++++++++++++++++

Any additional fields from the data can be added, but must have a corresponding
entry in the root field table (described below.)  The naming scheme is to be as
explicit as possible, with units in cgs (or a conversion factor to the standard
cgs unit, in the field table.)

Attribute Table
---------------

In the root node, we define several groups which contain attributes.

Simulation Parameters
+++++++++++++++++++++

These attributes will all be associated with ``/simulation_parameters``.

``refine_by``
   relative global refinement
``dimensionality``
   1-, 2- or 3-D data
``domain_dimensions``
   dimensions in the top grid
``current_time``
   current time in simulation, in seconds, from “start” of simulation
``domain_left_edge``
   the left edge of the domain, in cm
``domain_right_edge``
   the right edge of the domain, in cm
``unique_identifier``
   regarded as a string, but can be anything
``cosmological_simulation``
   0 or 1
``num_ghost_zones``
   integer
``field_ordering``
   integer: 0 for C, 1 for Fortran
``boundary_conditions``
   integer (6): 0 for periodic, 1 for mirrored, 2 for outflow.  Needs one for each face
   of the cube.  Any past the dimensionality should be set to -1.  The order of specification
   goes left in 0th dimension, right in 0th dimension, left in 1st dimension, right in 1st dimensions,
   left in 2nd dimension, right in 2nd dimension.  Note also that yt does not currently support non-periodic
   boundary conditions, and that the assumption of periodicity shows up primarily in plots and
   covering grids.

Optionally, attributes for cosmological simulations can be provided, if
cosmological_simulation above is set to 1:

  * current_redshift
  * omega_matter (at z=0)
  * omega_lambda (at z=0)
  * hubble_constant (h100)

Fluid Field Attributes
++++++++++++++++++++++

Every field that is included that is not both in CGS already and in the list
above requires parameters.  If a field is in the above list but is not in CGS,
only the field_to_cgs attribute is necessary.  These will be stored under
``/field_types`` and each must possess the following attributes:

``field_name``
   a string that will be used to describe the field; can contain spaces.
``field_to_cgs``
   a float that will be used to convert the field to cgs units, if necessary.
   Set to 1.0 if no conversion necessary.  Note that if non-CGS units are desired
   this field should simply be viewed as the value by which field values are
   multiplied to get to some internally consistent unit system.
``field_units``
   a string that names the units.
``staggering``
   an integer: 0 for cell-centered, 1 for face-centered, 2 for vertex-centered.
   Non-cellcentered data will be linearly-interpolated; more complicated
   reconstruction will be defined in a future version of this standard; for 1.0
   we only allow for simple definitions.

Particle Types
++++++++++++++

Every particle type that is not recognized (i.e., all non-Dark Matter types)
needs to have an entry under ``/particle_types``.  Each entry must possess the
following attributes:

``particle_type_name``
   a string that will be used to describe the field; can contain spaces.
``particle_use_dataspace``
   (optional) if 1, the dataspace (see particle field definition above) will be used
   for all particle fields for this type of particle.  Useful if a given type of particle
   is embedded inside a larger list of different types of particle.
``particle_type_num``
   an integer giving the total number of particles of this type.

For instance, to define a particle of type ``accreting_black_hole``, the file
must contain ``/particle_types/accreting_black_hole``, with the
``particle_type_name`` attribute of "Accreting Black Hole".

Particle Field Attributes
+++++++++++++++++++++++++

Every particle type that contains a new field (for instance, ``accretion_rate``)
needs to have an entry under ``/particle_types/{particle_type_name}/{field_name}``
containing the following attributes:

``field_name``
   a string that will be used to describe the field; can contain spaces.
``field_to_cgs``
   a float that will be used to convert the field to cgs units, if necessary.
   Set to 1.0 if no conversion necessary.
``field_units``
   a string that names the units.

Role of YT
----------

yt will provide a reader for this data, so that any data in this format can be
used by the code.  Additionally, the names and specifications in this code
reflect the internal yt data structures.

yt will also provide a writer for this data, which will operate on any existing
data format.  Provided that a simulation code can read this data, this will
enable cross-platform comparison.  Furthermore, any external piece of software
(i.e., Stranger) that implements reading this format will be able to read any
format of data tha yt understands.

Example File
------------

An example file constructed from the ``RD0005-mine`` dataset is available
at http://yt.enzotools.org/files/RD0005.gdf .  It is not yet a complete
conversion, but it is a working proof of concept.  Readers and writers are
forthcoming.
