# Support multiple datasets

## Status

Merged – 19/01/2026

## Context

The need to support multiple datasets in Anemoi arises from use cases such as:

- Spatial downscaling
- Temporal interpolation
- Adding observations
- Model coupling
- Complex configurations (e.g., combining a LAM with model levels & a global dataset with pressure levels)
- Autoencoder

Previously, the configuration was limited to a single dataset, restricting flexibility for these advanced scenarios.

## Decision

- The configuration now supports a dictionary-based approach ("dict-ification") for datasets and related components.
- This work is limited to time-aligned datasets.
- The main entry point is a `datasets:` field, which is a dictionary where each key is a dataset name and the value is its configuration.
- For each dataset, you can specify different `forcing`, `diagnostic`, and `target` variables, enabling flexible use cases such as emulation, downscaling, and more.
- This approach allows users to define per-dataset settings for data processing, normalisation, variable selection and others.
- The single-dataset configuration is deprecated in favor of this new structure, but backward compatibility is temporarily maintained with a deprecation warning (without config validation).
- Not all components are yet configurable per-dataset. For example, `graphs` and `model` remain global for now. To avoid this limitation, graphs can be loaded separately for each dataset. Per-dataset configuration for these will be considered in follow-up PRs.

## Scope of Change

- anemoi-training
- anemoi-models
- anemoi-inference

## Consequences

**Benefits:**
- Enables advanced workflows (downscaling, emulation, etc.) by allowing different datasets and variable roles in a single configuration.
- Lays the foundation for future per-dataset configuration in other components (e.g., graphs, model).

**Trade-offs:**
- Some configuration complexity increases due to the dictionary-based structure.
- Not all components are yet fully per-dataset configurable. Further work is needed.

**Technical Debt:**
- Temporary support for the old single-dataset configuration will need to be removed in a future release.

**Next steps:**
- Implementation of `DynamicGraphProvider`s to support data-driven models with dynamic observations.
- Implementation of per-dataset configuration for graphs and models.
- Increased flexibility in selecting samples from different datasets (e.g. different temporal resolution, number of steps, etc.).

## Alternatives Considered

- Introducing per-dataset configuration for all components in one PR (too large in scope for a single change).
- Refactoring training worklflow for multiple datasets (hard to test and merge)

## References

- [PR #594: feat(models,training): multi dataset integration](https://github.com/ecmwf/anemoi-core/pull/594)
