kedro.io.CachedDataSet¶

class kedro.io.CachedDataSet(dataset, version=None, copy_mode=None)[source]¶

Bases: kedro.io.core.AbstractDataSet

CachedDataSet is a dataset wrapper which caches in memory the data saved, so that the user avoids io operations with slow storage media.

You can also specify a CachedDataSet in catalog.yml:

test_ds:
   type: CachedDataSet
   versioned: true
   dataset:
      type: pandas.CSVDataSet
      filepath: example.csv

Please note that if your dataset is versioned, this should be indicated in the wrapper class as shown above.

Methods

`CachedDataSet.__init__`(dataset[, version, …])	Creates a new instance of `CachedDataSet` pointing to the provided Python object.
`CachedDataSet.exists`()	Checks whether a data set’s output already exists by calling the provided _exists() method.
`CachedDataSet.from_config`(name, config[, …])	Create a data set instance using the configuration provided.
`CachedDataSet.load`()	Loads data by delegation to the provided load method.
`CachedDataSet.release`()	Release any cached data.
`CachedDataSet.save`(data)	Saves data by delegation to the provided save method.

__init__(dataset, version=None, copy_mode=None)[source]¶

Creates a new instance of CachedDataSet pointing to the provided Python object.

Parameters:

dataset (Union[AbstractDataSet, Dict[~KT, ~VT]]) – A Kedro DataSet object or a dictionary to cache.
version (Optional[Version]) – If specified, should be an instance of kedro.io.core.Version. If its load attribute is None, the latest version will be loaded. If its save attribute is None, save version will be autogenerated.
copy_mode (Optional[str]) – The copy mode used to copy the data. Possible values are: “deepcopy”, “copy” and “assign”. If not provided, it is inferred based on the data type.

Raises:

ValueError – If the provided dataset is not a valid dict/YAML representation of a dataset or an actual dataset.

exists()¶

Checks whether a data set’s output already exists by calling the provided _exists() method.

Return type:	`bool`
Returns:	Flag indicating whether the output already exists.
Raises:	`DataSetError` – when underlying exists method raises error.

classmethod from_config(name, config, load_version=None, save_version=None)¶

Create a data set instance using the configuration provided.

Parameters:	name (`str`) – Data set name. config (`Dict`[`str`, `Any`]) – Data set config dictionary. load_version (`Optional`[`str`]) – Version string to be used for `load` operation if the data set is versioned. Has no effect on the data set if versioning was not enabled. save_version (`Optional`[`str`]) – Version string to be used for `save` operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.
Return type:	`AbstractDataSet`
Returns:	An instance of an `AbstractDataSet` subclass.
Raises:	`DataSetError` – When the function fails to create the data set from its config.

load()¶

Loads data by delegation to the provided load method.

Return type:	`Any`
Returns:	Data returned by the provided load method.
Raises:	`DataSetError` – When underlying load method raises error.

release()¶

Release any cached data.

Raises:	`DataSetError` – when underlying release method raises error.
Return type:	`None`

save(data)¶

Saves data by delegation to the provided save method.

Parameters:	data (`Any`) – the value to be saved by provided save method.
Raises:	`DataSetError` – when underlying save method raises error.
Return type:	`None`