kedro.contrib.io.catalog_with_default package¶

DataCatalogWithDefault is a DataCatalog that will fall back to a default AbstractDataSet implementation if the requested key in not already registered in the catalog.

Submodules¶

kedro.contrib.io.catalog_with_default.data_catalog_with_default module¶

A DataCatalog with a default DataSet implementation for any data set which is not registered in the catalog.

class kedro.contrib.io.catalog_with_default.data_catalog_with_default.DataCatalogWithDefault(data_sets=None, default=None, remember=False)[source]¶

Bases: kedro.io.data_catalog.DataCatalog

A DataCatalog with a default DataSet implementation for any data set which is not registered in the catalog.

__init__(data_sets=None, default=None, remember=False)[source]¶

A DataCatalog with a default DataSet implementation for any data set which is not registered in the catalog.

Parameters:

data_sets (Optional[Dict[str, AbstractDataSet]]) – A dictionary of data set names and data set instances.
default (Optional[Callable[[str], AbstractDataSet]]) – A callable which accepts a single argument of type string, the key of the data set, and returns an AbstractDataSet. load and save calls on data sets which are not registered to the catalog will be delegated to this AbstractDataSet.
remember (bool) – If True, then store in the catalog any AbstractDataSets provided by the default callable argument. Useful when one want to transition from a DataCatalogWithDefault to a DataCatalog: just call DataCatalogWithDefault.to_yaml, after all required data sets have been saved/loaded, and use the generated YAML file with a new DataCatalog.

Raises:

TypeError – If default is not a callable.

Example:

from kedro.io import CSVLocalDataSet

def default_data_set(name):
    return CSVLocalDataSet(filepath='data/01_raw/' + name)

io = DataCatalog(data_sets={},
                 default=default_data_set)

# load the file in data/raw/cars.csv
df = io.load("cars.csv")

classmethod from_config(catalog, credentials=None, load_versions=None, save_version=None)[source]¶

To create a DataCatalogWithDefault from configuration, please use:

DataCatalogWithDefault.from_data_catalog(
    DataCatalog.from_config(catalog, credentials))

Parameters:	catalog (`Optional`[`Dict`[`str`, `Dict`[`str`, `Any`]]]) – See `DataCatalog.from_config` credentials (`Optional`[`Dict`[`str`, `Dict`[`str`, `Any`]]]) – See `DataCatalog.from_config` load_versions (`Optional`[`Dict`[`str`, `str`]]) – See `DataCatalog.from_config` save_version (`Optional`[`str`]) – See `DataCatalog.from_config`
Raises:	`ValueError` – If you try to instantiate a `DataCatalogWithDefault` directly with this method.

classmethod from_data_catalog(data_catalog, default)[source]¶

Convenience factory method to create a DataCatalogWithDefault from a DataCatalog

A DataCatalog with a default DataSet implementation for any data set which is not registered in the catalog.

Parameters:	data_catalog (`DataCatalog`) – The `DataCatalog` to convert to a `DataCatalogWithDefault`. default (`Callable`[[`str`], `AbstractDataSet`]) – A callable which accepts a single argument of type string, the key of the data set, and returns an `AbstractDataSet`. `load` and `save` calls on data sets which are not registered to the catalog will be delegated to this `AbstractDataSet`.
Return type:	`DataCatalogWithDefault`
Returns:	A new `DataCatalogWithDefault` which contains all the `AbstractDataSets` from the provided data-catalog.

load(name)[source]¶

Loads a registered data set

Parameters:	name (`str`) – A data set to be loaded.
Return type:	`Any`
Returns:	The loaded data as configured.
Raises:	`DataSetNotFoundError` – When a data set with the given name has not yet been registered.

save(name, data)[source]¶

Save data to a registered data set.

Parameters:	name (`str`) – A data set to be saved to. data (`Any`) – A data object to be saved as configured in the registered data set.
Raises:	`DataSetNotFoundError` – When a data set with the given name has not yet been registered.

shallow_copy()[source]¶: Returns a shallow copy of the current object. :rtype: DataCatalogWithDefault :returns: Copy of the current object.