kedro.extras.datasets.pickle.PickleDataSet — Kedro 0.17.6 documentation

# kedro.extras.datasets.pickle.PickleDataSet

class `kedro.extras.datasets.pickle.` `PickleDataSet`(filepath, backend='pickle', load_args=None, save_args=None, version=None, credentials=None, fs_args=None)

  `PickleDataSet` loads/saves data from/to a Pickle file using an underlying filesystem (e.g.: local, S3, GCS). The underlying functionality is supported by the specified backend library passed in (defaults to the `pickle` library), so it supports all allowed options for loading and saving pickle files.

  Example adding a catalog entry with YAML API:

      test_model: # simple example without compression
        type: pickle.PickleDataSet
        filepath: data/07_model_output/test_model.pkl
        backend: pickle

      final_model: # example with load and save args
        type: pickle.PickleDataSet
        filepath: s3://your_bucket/final_model.pkl.lz4
        backend: joblib
        credentials: s3_credentials
        save_args:
          compression: lz4
        load_args:
          compression: lz4

  Example using Python API:

      from kedro.extras.datasets.pickle import PickleDataSet
      import pandas as pd

      data = pd.DataFrame({'col1': [1, 2], 'col2': [4, 5],
                           'col3': [5, 6]})

      # data_set = PickleDataSet(filepath="gcs://bucket/test.pkl")
      data_set = PickleDataSet(filepath="test.pkl", backend="pickle")
      data_set.save(data)
      reloaded = data_set.load()
      assert data.equals(reloaded)

      # Add "compress_pickle[lz4]" to requirements.txt
      data_set = PickleDataSet(filepath="test.pickle.lz4",
                               backend="compress_pickle",
                               load_args={"compression":"lz4"},
                               save_args={"compression":"lz4"})
      data_set.save(data)
      reloaded = data_set.load()
      assert data.equals(reloaded)

  Attributes

  `DEFAULT_LOAD_ARGS` |
  `DEFAULT_SAVE_ARGS` |

  Methods

  `exists`() | Checks whether a data set’s output already exists by calling the provided _exists() method.
  `from_config`(name, config[, load_version, …]) | Create a data set instance using the configuration provided.
  `load`() | Loads data by delegation to the provided load method.
  `release`() | Release any cached data.
  `resolve_load_version`() | Compute the version the dataset should be loaded with.
  `resolve_save_version`() | Compute the version the dataset should be saved with.
  `save`(data) | Saves data by delegation to the provided save method.

  `DEFAULT_LOAD_ARGS`: Dict[str, Any] = {}
  `DEFAULT_SAVE_ARGS`: Dict[str, Any] = {}
  `__init__`(filepath, backend='pickle', load_args=None, save_args=None, version=None, credentials=None, fs_args=None)

    Creates a new instance of `PickleDataSet` pointing to a concrete Pickle file on a specific filesystem. `PickleDataSet` supports custom backends to serialize/deserialize objects.

    Example backends that are compatible (non-exhaustive):

      - pickle

      - joblib

      - dill

      - compress_pickle

    Example backends that are incompatible:

      - torch

    Parameters

      - filepath (`str`) – Filepath in POSIX format to a Pickle file prefixed with a protocol like s3://. If prefix is not provided, file protocol (local filesystem) will be used. The prefix should be any protocol supported by `fsspec`. Note: http(s) doesn’t support versioning.

      - backend (`str`) – Backend to use, must be an import path to a module which satisfies the `pickle` interface. That is, contains a load and dump function. Defaults to ‘pickle’.

      - load_args (`Optional`[`Dict`[`str`, `Any`]]) – Pickle options for loading pickle files. You can pass in arguments that the backend load function specified accepts, e.g: pickle.load: https://docs.python.org/3/library/pickle.html#pickle.load joblib.load: https://joblib.readthedocs.io/en/latest/generated/joblib.load.html dill.load: https://dill.readthedocs.io/en/latest/dill.html#dill._dill.load compress_pickle.load: https://lucianopaz.github.io/compress_pickle/html/api/compress_pickle.html#compress_pickle.compress_pickle.load All defaults are preserved.

      - save_args (`Optional`[`Dict`[`str`, `Any`]]) – Pickle options for saving pickle files. You can pass in arguments that the backend dump function specified accepts, e.g: pickle.dump: https://docs.python.org/3/library/pickle.html#pickle.dump joblib.dump: https://joblib.readthedocs.io/en/latest/generated/joblib.dump.html dill.dump: https://dill.readthedocs.io/en/latest/dill.html#dill._dill.dump compress_pickle.dump: https://lucianopaz.github.io/compress_pickle/html/api/compress_pickle.html#compress_pickle.compress_pickle.dump All defaults are preserved.

      - version (`Optional`[`Version`]) – If specified, should be an instance of `kedro.io.core.Version`. If its `load` attribute is None, the latest version will be loaded. If its `save` attribute is None, save version will be autogenerated.

      - credentials (`Optional`[`Dict`[`str`, `Any`]]) – Credentials required to get access to the underlying filesystem. E.g. for `GCSFileSystem` it should look like {“token”: None}.

      - fs_args (`Optional`[`Dict`[`str`, `Any`]]) – Extra arguments to pass into underlying filesystem class constructor (e.g. {“project”: “my-project”} for `GCSFileSystem`), as well as to pass to the filesystem’s open method through nested keys open_args_load and open_args_save. Here you can find all available arguments for open: https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.open All defaults are preserved, except mode, which is set to wb when saving.

    Raises

      - ValueError – If `backend` does not satisfy the pickle interface.

      - ImportError – If the `backend` module could not be imported.

  `exists`()

    Checks whether a data set’s output already exists by calling the provided _exists() method.

    Return type

      `bool`

    Returns

      Flag indicating whether the output already exists.

    Raises

      DataSetError – when underlying exists method raises error.

  classmethod `from_config`(name, config, load_version=None, save_version=None)

    Create a data set instance using the configuration provided.

    Parameters

      - name (`str`) – Data set name.

      - config (`Dict`[`str`, `Any`]) – Data set config dictionary.

      - load_version (`Optional`[`str`]) – Version string to be used for `load` operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.

      - save_version (`Optional`[`str`]) – Version string to be used for `save` operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.

    Return type

      `AbstractDataSet`

    Returns

      An instance of an `AbstractDataSet` subclass.

    Raises

      DataSetError – When the function fails to create the data set from its config.

  `load`()

    Loads data by delegation to the provided load method.

    Return type

      `Any`

    Returns

      Data returned by the provided load method.

    Raises

      DataSetError – When underlying load method raises error.

  `release`()

    Release any cached data.

    Raises

      DataSetError – when underlying release method raises error.

    Return type

      `None`

  `resolve_load_version`()

    Compute the version the dataset should be loaded with.

    Return type

      `Optional`[`str`]

  `resolve_save_version`()

    Compute the version the dataset should be saved with.

    Return type

      `Optional`[`str`]

  `save`(data)

    Saves data by delegation to the provided save method.

    Parameters

      data (`Any`) – the value to be saved by provided save method.

    Raises

      - DataSetError – when underlying save method raises error.

      - FileNotFoundError – when save method got file instead of dir, on Windows.

      - NotADirectoryError – when save method got file instead of dir, on Unix.

    Return type

      `None`