kedro.io.ParquetLocalDataSet¶

class kedro.io.ParquetLocalDataSet(filepath, engine='auto', load_args=None, save_args=None, version=None)[source]¶

AbstractDataSet with functionality for handling local parquet files.

Example:

from kedro.io import ParquetLocalDataSet
import pandas as pd

data = pd.DataFrame({'col1': [1, 2], 'col2': [4, 5],
                     'col3': [5, 6]})
data_set = ParquetLocalDataSet('myFile')
data_set.save(data)
loaded_data = data_set.load()
assert data.equals(loaded_data)

__init__(filepath, engine='auto', load_args=None, save_args=None, version=None)[source]¶

Creates a new instance of ParquetLocalDataSet pointing to a concrete filepath.

Parameters:

filepath (str) – Path to a parquet file or a metadata file of a multipart parquet collection or the directory of a multipart parquet.
engine (str) – The engine to use, one of: auto, fastparquet, pyarrow. If auto, then the default behavior is to try pyarrow, falling back to fastparquet if pyarrow is unavailable.
load_args (Optional[Dict[str, Any]]) – Additional loading options pyarrow: https://arrow.apache.org/docs/python/generated/pyarrow.parquet.read_table.html or fastparquet: https://fastparquet.readthedocs.io/en/latest/api.html#fastparquet.ParquetFile.to_pandas
save_args (Optional[Dict[str, Any]]) – Additional saving options for pyarrow: https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.from_pandas or fastparquet: https://fastparquet.readthedocs.io/en/latest/api.html#fastparquet.write
version (Optional[Version]) – If specified, should be an instance of kedro.io.core.Version. If its load attribute is None, the latest version will be loaded. If its save attribute is None, save version will be autogenerated.

Return type:

None

Methods

`__init__`(filepath[, engine, load_args, …])	Creates a new instance of `ParquetLocalDataSet` pointing to a concrete filepath.
`exists`()	Checks whether a data set’s output already exists by calling the provided _exists() method.
`from_config`(name, config[, load_version, …])	Create a data set instance using the configuration provided.
`load`()	Loads data by delegation to the provided load method.
`save`(data)	Saves data by delegation to the provided save method.