.. _usage-getting-started:

###########################
 Create Your First Dataset
###########################

The following example presents a simple but comprehensive recipe to
create an Anemoi dataset based on the atmospheric reanalysis ERA5. Once
built, the dataset can be used to train an ML atmospheric model, such as
AIFS, at low resolution. Note that a prebuilt version of the
anemoi-dataset generated by this recipe can be downloaded from this link
`era5-o48-2020-2021-6h-v1.zip
<https://data.ecmwf.int/anemoi-datasets/era5-o48-2020-2021-6h-v1.zip>`_.
Do not unzip the file; you can pass it directly to ``open_dataset``.

.. warning::

   Please note that the size of the file
   ``era5-o48-2020-2021-6h-v1.zip`` is approximately 2.5 GB.

.. literalinclude:: yaml/aifs-ea-an-oper-0001-mars-o48-2020-2021-6h-v1.yaml
   :language: yaml

Let's break down the recipe to understand its main components! Some
concepts presented below will be explained in more detail in the USER
GUIDE section.

********************************
 Dataset Naming and Description
********************************

.. code:: yaml

   name: era5-o48-2020-2021-6h-v1

   description: Low-resolution reduced dataset for documentation purposes

   attribution: ECMWF/C3S

   licence: CC-BY-4.0

In the first lines of the recipe, we define the dataset name, a brief
description, the attribution to the data source, and the licence under
which the dataset is released. This information will be stored in the
dataset metadata.

**Dates**

.. code:: yaml

   dates:
       start: '2020-01-01T00:00:00'
       end: '2021-12-31T23:00:00'
       frequency: 6h

Here, we define the time dimension of the Anemoi dataset. In particular,
we specify the start and end dates of the dataset and the frequency of
the time steps. In our specific example, the dataset is based on ERA5,
which has a 1-hour frequency, but we choose to downsample it to 6 hours
– the typical timestep of the AIFS model. Note that this does not imply
that the timestep of the final model has to be 6 hours; it could be a
multiple of it.

*****************************************
 Getting the Data from Different Streams
*****************************************

In the remainder of the recipe (input), we join variables coming from
different streams to form the dataset. In particular, the following
streams are used:

-  **Surface fields** from the Copernicus Climate Data Store (CDS)
   through MARS requests.
-  **Pressure level fields** from the CDS through MARS requests.
-  **Accumulated fields** from the CDS through MARS requests.
-  **Computed constants** generated by anemoi-datasets.

At this point, we also specify the resolution of the dataset, an *o48*
octahedral Gaussian grid in our specific case.

******************************
 How Do We Build the Dataset?
******************************

The dataset is built via the `anemoi-datasets` command-line tool as
follows:

.. code:: bash

   $ anemoi-datasets create era5-o48-2020-2021-6h-v1.yaml era5-o48-2020-2021-6h-v1.zarr

The YAML file ``era5-o48-2020-2021-6h-v1.yaml`` contains the recipe, and
the output is stored in ``era5-o48-2020-2021-6h-v1.zarr``.

.. warning::

   Note that running the recipe requires that the access to the CDS is
   correctly set up. For this, please refer to the :ref:`CDS page
   <cds_source>`.

Once the build is complete, you can inspect the dataset using the
following command:

.. code:: bash

   $ anemoi-datasets inspect era5-o48-2020-2021-6h-v1.zarr

Leading to the following output:

.. code:: bash

   📦 Path          : ./era5-o48-2020-2021-6h-v1.zarr
   🔢 Format version: 0.30.0

   📅 Start      : 2020-01-01 00:00
   📅 End        : 2021-12-31 18:00
   ⏰ Frequency  : 6h
   🚫 Missing    : 0
   🌎 Resolution : O48
   🌎 Field shape: [10944]

   📐 Shape      : 2,924 × 43 × 1 × 10,944 (5.1 GiB)
   💽 Size       : 2.8 GiB (2.8 GiB)
   📁 Files      : 3,040

   Index │ Variable       │          Min │       Max │         Mean │       Stdev
   ──────┼────────────────┼──────────────┼───────────┼──────────────┼────────────
       0 │ 10u            │     -35.0512 │   30.6795 │    -0.550947 │     5.47129
       1 │ 10v            │     -34.5802 │    33.542 │     0.212642 │     4.51989
       2 │ 2d             │      191.632 │   304.995 │      282.699 │     15.8811
       3 │ 2t             │      195.362 │   325.785 │       287.83 │     16.1394
       4 │ cos_julian_day │    -0.999998 │         1 │   -0.0562081 │    0.721861
       5 │ cos_latitude   │    0.0249178 │  0.999867 │     0.785126 │    0.237451
       6 │ cos_local_time │           -1 │         1 │  9.31395e-13 │    0.707107
       7 │ cos_longitude  │           -1 │         1 │  3.48565e-10 │    0.707107
       8 │ cp             │            0 │ 0.0859051 │  0.000384542 │  0.00126894
       9 │ insolation     │            0 │         1 │     0.250838 │    0.323889
       10 │ lsm            │            0 │         1 │     0.287684 │    0.441725
       11 │ msl            │      92639.9 │    106800 │       101148 │      1132.7
       12 │ q_1000         │        1e-08 │ 0.0293589 │   0.00963055 │  0.00588138
       13 │ q_250          │ -4.33519e-05 │  0.001311 │  8.47611e-05 │ 9.17112e-05
       14 │ q_500          │ -5.44315e-05 │ 0.0100051 │   0.00120283 │  0.00132836
       15 │ q_850          │        1e-08 │ 0.0228447 │   0.00629476 │  0.00432839
       16 │ sdor           │            0 │   679.961 │      20.9144 │     61.4512
       17 │ sin_julian_day │    -0.999999 │  0.999999 │     0.180368 │    0.665751
       18 │ sin_latitude   │     -0.99969 │   0.99969 │            0 │    0.572008
       19 │ sin_local_time │           -1 │         1 │  1.49023e-13 │    0.707107
       20 │ sin_longitude  │           -1 │         1 │ -2.17853e-09 │    0.707107
       21 │ skt            │      194.347 │   342.479 │      288.565 │     16.9845
       22 │ slor           │       0.0001 │  0.115619 │   0.00346549 │   0.0100738
       23 │ sp             │      50213.9 │    106807 │      98447.5 │     7059.61
       24 │ t_1000         │      219.141 │   324.407 │      288.891 │     13.7525
       25 │ t_250          │      195.464 │   247.214 │      226.305 │     7.57534
       26 │ t_500          │      218.042 │   285.169 │      259.119 │     11.1935
       27 │ t_850          │      219.383 │   313.381 │      281.671 │     12.6545
       28 │ tcw            │    0.0523909 │   127.525 │      25.5015 │      17.518
       29 │ tp             │            0 │  0.204357 │  0.000744018 │  0.00257298
       30 │ u_1000         │     -31.3894 │   29.2405 │     -0.60533 │     6.04254
       31 │ u_250          │     -65.3296 │   112.971 │       13.497 │     18.5855
       32 │ u_500          │     -49.8631 │   82.0021 │      6.03161 │     11.8884
       33 │ u_850          │      -56.938 │   52.5499 │     0.718384 │     8.11392
       34 │ v_1000         │     -34.9257 │    30.746 │     0.220698 │     4.95779
       35 │ v_250          │     -87.8586 │   100.591 │   -0.0613504 │     12.6573
       36 │ v_500          │      -63.541 │   76.2497 │   -0.0431783 │     8.24538
       37 │ v_850          │     -47.4795 │   52.8012 │     0.108581 │     5.63597
       38 │ z              │     -374.223 │     54350 │      2377.04 │     6366.77
       39 │ z_1000         │     -6045.95 │   4721.73 │      943.065 │     915.956
       40 │ z_250          │      87717.1 │    109625 │       103869 │     4752.65
       41 │ z_500          │      43732.3 │     58934 │      55645.5 │     2822.68
       42 │ z_850          │      6671.23 │   16961.5 │      14286.1 │     1239.99
   ──────┴────────────────┴──────────────┴───────────┴──────────────┴────────────
   🪫 Dataset not initialised
