.. _inputs:

########
 Inputs
########

The input methods listed below are used to fetch the initial conditions
(prognostic variables) of a model run. They will also be the source of
constant and dynamic forcings unless ``constant_forcings`` or
``dynamic_forcings`` entries are specified in the configuration (see
:ref:`forcings`).

**********
 Datasets
**********

You can use the dataset that was used during training as input. It can
be ``test``, ``training`` or ``validation``, corresponding to the
entries given during training as ``dataloader.test``,
``dataloader.training`` and ``dataloader.validation`` respectively.

.. literalinclude:: yaml/inputs_1.yaml
   :language: yaml

``test`` is the default input if no input is specified.

If the training happened on a different computer and the datasets files
are not available on the current computer, you can use
`anemoi-datasets's` :ref:`anemoi-datasets:configuration` to define a
search path to the datasets. To enable this, you have set the
``use_original_paths`` option to ``false``.

.. literalinclude:: yaml/inputs_2.yaml
   :language: yaml

You can also provide a full dataset specification as follows:

.. literalinclude:: yaml/inputs_3.yaml
   :language: yaml

See :ref:`anemoi-datasets:opening-datasets` in the documentation of the
`anemoi-datasets` package for more information on how to open datasets.

******
 grib
******

You can specify the input as ``grib`` to read the data from a GRIB file.

.. literalinclude:: yaml/inputs_4.yaml
   :language: yaml

For more options, see :ref:`grib-input`.

********
 netcdf
********

You can specify the input as ``netcdf`` to read the data from a NetCDF
file.

.. literalinclude:: yaml/inputs_5.yaml
   :language: yaml

.. note::

   The netcdf input expects the data to be stored identically to the
   GRIB input, i.e. with a flattened values dimension (..., values), and
   variable names that match what the checkpoint was trained on.

   ```text
   <xarray.Dataset> Size: 165MB
   Dimensions:    (date: X, values: 542080)
   Coordinates:
     * date       (date) ....
       latitude   (values) float64 4MB ...
       longitude  (values) float64 4MB ...
   Dimensions without coordinates: values
   Data variables:
       10u        (date, values) float64 39MB ...
       10v        (date, values) float64 39MB ...
       2t         (date, values) float64 39MB ...
       ...
   ```


   For context, using ``earthkit-data`` a grib file can be exported as
   netcdf with the following and then read back in using the netcdf
   input, note how the names are created:

   .. code:: python

      import earthkit.data as ekd

      ds = ekd.from_source("file", "input.grib")

      ds_xr = ds.to_xarray(
          variable_key="p_l",
          remapping={"p_l": "{param}_{levelist}"},
          add_earthkit_attrs=False,
      )
      ds_xr.to_netcdf("input.nc")

******
 mars
******

You can also specify the input as ``mars`` to read the data from ECMWF's
MARS archive. This requires the `ecmwf-api-client` package to be
installed, and the user to have an ECMWF account.

.. literalinclude:: yaml/inputs_6.yaml
   :language: yaml

You can also specify some of the MARS keywords as options. The default
is to retrieve the data from the operational analysis (``class=od``).
You can change that to use ERA5 reanalysis data (``class=ea``).

.. literalinclude:: yaml/inputs_7.yaml
   :language: yaml

The ``mars`` input also accepts the ``namer`` parameter of the GRIB
input.

*****
 cds
*****

You can also specify the input as ``cds`` to read the data from the
`Climate Data Store <https://cds.climate.copernicus.eu/>`_. This
requires the `cdsapi` package to be installed, and the user to have a
CDS account.

.. literalinclude:: yaml/inputs_8.yaml
   :language: yaml

As the CDS contains a plethora of `datasets
<https://cds.climate.copernicus.eu/datasets>`_, you can specify the
dataset you want to use with the key `dataset`.

This can be a str in which case the dataset is used for all requests, or
a dict of any number of levels which will be descended based on the
key/values for each request.

You can use `*` to represent any not given value for a key, i.e. set a
dataset for `param: 2t`. and `param: *` to represent any other param.

.. literalinclude:: yaml/inputs_9.yaml
   :language: yaml

In the above example, the dataset `reanalysis-era5-pressure-levels` is
used for all with `levtype: pl` and `reanalysis-era5-single-levels` used
for all with `levtype: sfc`.

Additionally, any kwarg can be passed to be added to all requests, i.e.
for ERA5 data, `product_type: 'reanalysis'` is needed.

.. literalinclude:: yaml/inputs_10.yaml
   :language: yaml

********
 cutout
********

``cutout`` is a special type of input that combines one or more Limited
Area Model (LAM) sources into a global source using a nested cutout
approach. This is also known as the "stretched-grid" method, see Nipen
et al. (2024). The ``cutout`` input contains multiple sources, each with
its own input type (e.g. 'grib', 'mars', etc.), and the order of the
sources determines the nesting order. The first source is the innermost
domain, and the last source is the outermost, global domain,
consistently with what is done in ``anemoi-datasets``, see `here
<https://anemoi.readthedocs.io/projects/datasets/en/latest/using/combining.html#cutout>`_.

.. warning::

   The ``cutout`` input nests the ``private_attributes`` of the sources
   states so may prevent usage of some keys. To restore these, use the
   ``extract_from_state`` postprocessor.

To extract regions from different sources within the ``cutout`` input,
your checkpoint must contain the cutout masks as supporting arrays. You
can check this by running the ``anemoi-inference metadata
--supporting-arrays <your_checkpoint>`` command. You should be able to
see some cutout masks in the output:

.. code:: console

   lam_0/cutout_mask: shape=(226980,) dtype=bool
   global/cutout_mask: shape=(542080,) dtype=bool

If these are not present, you can try to add them to your checkpoint by
running ``anemoi-inference patch <your_checkpoint>``.

.. note::

   If your LAM does not have a cutout mask, you can still use it and
   specify the mask as ``null`` in the configuration file.

.. code:: yaml

   input:
     cutout:
       - lam_0:
         mars: {}
         mask: null

An example configuration for the ``cutout`` input is shown below, the
sources can be provided as a list of positional arguments, with each
source specified as a mapping from source name to source configuration:

.. literalinclude:: yaml/inputs_11.yaml
   :language: yaml

The different sources are specified exactly as you would for a single
source, as shown in the previous sections.

An easy way to then extract regions from the predicated state is to use
the ``extract_from_state`` postprocessor, which will subset the state to
the specified source. For example, to extract the ``lam_0`` source from
the state, you can use the following configuration:

.. code:: yaml

   post_processors:
   - extract_from_state: 'lam_0'
