kedro.config.TemplatedConfigLoader — Kedro 0.17.6 documentation

# kedro.config.TemplatedConfigLoader

class `kedro.config.` `TemplatedConfigLoader`(conf_paths, *, globals_pattern=None, globals_dict=None)

  Extension of the `ConfigLoader` class that allows for template values, wrapped in brackets like: ${…}, to be automatically formatted based on the configs.

  The easiest way to use this class is by registering it into the `KedroContext` using hooks. This can be done by updating the hook implementation register_config_loader in hooks.py, making it return a `TemplatedConfigLoader` object instead of a `ConfigLoader` object.

  Example:

      from kedro.config import TemplatedConfigLoader

      class ProjectHooks:
          @hook_impl
          def register_config_loader(self, conf_paths: Iterable[str]) -> ConfigLoader:
              return TemplatedConfigLoader(
                  conf_paths,
                  globals_pattern="*globals.yml",
                  globals_dict={"param1": "pandas.CSVDataSet"}
              )

  The contents of the dictionary resulting from the globals_pattern get merged with the `globals_dict`. In case of conflicts, the keys in `globals_dict` take precedence. If the formatting key is missing from the dictionary, the default template value is used (the format is “${key|default value}”). If no default is set, a `ValueError` will be raised.

  Global parameters can be namespaced as well. An example could work as follows:

  globals.yml

      bucket: "my_s3_bucket"

      environment: "dev"

      datasets:
          csv: "pandas.CSVDataSet"
          spark: "spark.SparkDataSet"

      folders:
          raw: "01_raw"
          int: "02_intermediate"
          pri: "03_primary"
          fea: "04_feature"

  catalog.yml

      raw_boat_data:
          type: "${datasets.spark}"
          filepath: "s3a://${bucket}/${environment}/${folders.raw}/boats.csv"
          file_format: parquet

      raw_car_data:
          type: "${datasets.csv}"
          filepath: "s3://${bucket}/data/${environment}/${folders.raw}/cars.csv"

  This uses `jmespath` in the background. For more information see: https://github.com/jmespath/jmespath.py and https://jmespath.org/.

  Methods

  `get`(*patterns) | Tries to resolve the template variables in the config dictionary provided by the `ConfigLoader` (super class) `get` method using the dictionary of replacement values obtained in the `__init__` method.

  `__init__`(conf_paths, *, globals_pattern=None, globals_dict=None)

    Instantiate a `TemplatedConfigLoader`.

    Parameters

      - conf_paths (`Union`[`str`, `Iterable`[`str`]]) – Non-empty path or list of paths to configuration directories.

      - globals_pattern (`Optional`[`str`]) – Optional keyword-only argument specifying a glob pattern. Files that match the pattern will be loaded as a formatting dictionary.

      - globals_dict (`Optional`[`Dict`[`str`, `Any`]]) – Optional keyword-only argument specifying a formatting dictionary. This dictionary will get merged with the globals dictionary obtained from the globals_pattern. In case of duplicate keys, the `globals_dict` keys take precedence.

  `get`(*patterns)

    Tries to resolve the template variables in the config dictionary provided by the `ConfigLoader` (super class) `get` method using the dictionary of replacement values obtained in the `__init__` method.

    Parameters

      *patterns – Glob patterns to match. Files, which names match any of the specified patterns, will be processed.

    Return type

      `Dict`[`str`, `Any`]

    Returns

      A Python dictionary with the combined configuration from all configuration files. Note: any keys that start with _ will be ignored. String values wrapped in ${…} will be replaced with the result of the corresponding JMESpath expression evaluated against globals (see __init for more configuration files. Note: any keys that start with _ details).

    Raises

      ValueError – malformed config found.