kedro.io.AbstractDataSet¶

class kedro.io.AbstractDataSet[source]¶

Bases: abc.ABC

AbstractDataSet is the base class for all data set implementations. All data set implementations should extend this abstract class and implement the methods marked as abstract.

Example:

from pathlib import Path, PurePosixPath
import pandas as pd
from kedro.io import AbstractDataSet


class MyOwnDataSet(AbstractDataSet):
    def __init__(self, filepath, param1, param2=True):
        self._filepath = PurePosixPath(filepath)
        self._param1 = param1
        self._param2 = param2

    def _load(self) -> pd.DataFrame:
        return pd.read_csv(self._filepath)

    def _save(self, df: pd.DataFrame) -> None:
        df.to_csv(str(self._filepath))

    def _exists(self) -> bool:
        return Path(self._filepath.as_posix()).exists()

    def _describe(self):
        return dict(param1=self._param1, param2=self._param2)

Example catalog.yml specification:

my_dataset:
    type: <path-to-my-own-dataset>.MyOwnDataSet
    filepath: data/01_raw/my_data.csv
    param1: <param1-value> # param1 is a required argument
    # param2 will be True by default

Methods

`AbstractDataSet.exists`()	Checks whether a data set’s output already exists by calling the provided _exists() method.
`AbstractDataSet.from_config`(name, config[, …])	Create a data set instance using the configuration provided.
`AbstractDataSet.load`()	Loads data by delegation to the provided load method.
`AbstractDataSet.release`()	Release any cached data.
`AbstractDataSet.save`(data)	Saves data by delegation to the provided save method.

__init__¶: Initialize self. See help(type(self)) for accurate signature.

exists()[source]¶

Checks whether a data set’s output already exists by calling the provided _exists() method.

Return type:	`bool`
Returns:	Flag indicating whether the output already exists.
Raises:	`DataSetError` – when underlying exists method raises error.

classmethod from_config(name, config, load_version=None, save_version=None)[source]¶

Create a data set instance using the configuration provided.

Parameters:	name (`str`) – Data set name. config (`Dict`[`str`, `Any`]) – Data set config dictionary. load_version (`Optional`[`str`]) – Version string to be used for `load` operation if the data set is versioned. Has no effect on the data set if versioning was not enabled. save_version (`Optional`[`str`]) – Version string to be used for `save` operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.
Return type:	`AbstractDataSet`
Returns:	An instance of an `AbstractDataSet` subclass.
Raises:	`DataSetError` – When the function fails to create the data set from its config.

load()[source]¶

Loads data by delegation to the provided load method.

Return type:	`Any`
Returns:	Data returned by the provided load method.
Raises:	`DataSetError` – When underlying load method raises error.

release()[source]¶

Release any cached data.

Raises:	`DataSetError` – when underlying release method raises error.
Return type:	`None`

save(data)[source]¶

Saves data by delegation to the provided save method.

Parameters:	data (`Any`) – the value to be saved by provided save method.
Raises:	`DataSetError` – when underlying save method raises error.
Return type:	`None`