kedro.extras.datasets.pandas.GBQTableDataSet¶

class kedro.extras.datasets.pandas.GBQTableDataSet(dataset, table_name, project=None, credentials=None, load_args=None, save_args=None)[source]¶

Bases: kedro.io.core.AbstractDataSet

GBQTableDataSet loads and saves data from/to Google BigQuery. It uses pandas-gbq to read and write from/to BigQuery table.

Example:

from kedro.extras.datasets.pandas import GBQTableDataSet
import pandas as pd

data = pd.DataFrame({'col1': [1, 2], 'col2': [4, 5],
                     'col3': [5, 6]})

data_set = GBQTableDataSet('dataset',
                           'table_name',
                           project='my-project')
data_set.save(data)
reloaded = data_set.load()

assert data.equals(reloaded)

Attributes

`GBQTableDataSet.DEFAULT_LOAD_ARGS`
`GBQTableDataSet.DEFAULT_SAVE_ARGS`

Methods

`GBQTableDataSet.__init__`(dataset, table_name)	Creates a new instance of `GBQTableDataSet`.
`GBQTableDataSet.exists`()	Checks whether a data set’s output already exists by calling the provided _exists() method.
`GBQTableDataSet.from_config`(name, config[, …])	Create a data set instance using the configuration provided.
`GBQTableDataSet.load`()	Loads data by delegation to the provided load method.
`GBQTableDataSet.release`()	Release any cached data.
`GBQTableDataSet.save`(data)	Saves data by delegation to the provided save method.

DEFAULT_LOAD_ARGS = {}¶

DEFAULT_SAVE_ARGS = {'progress_bar': False}¶

__init__(dataset, table_name, project=None, credentials=None, load_args=None, save_args=None)[source]¶

Creates a new instance of GBQTableDataSet.

Parameters:	dataset (`str`) – Google BigQuery dataset. table_name (`str`) – Google BigQuery table name. project (`Optional`[`str`]) – Google BigQuery Account project ID. Optional when available from the environment. https://cloud.google.com/resource-manager/docs/creating-managing-projects credentials (`Union`[`Dict`[`str`, `Any`], `Credentials`, `None`]) – Credentials for accessing Google APIs. Either `google.auth.credentials.Credentials` object or dictionary with parameters required to instantiate `google.oauth2.credentials.Credentials`. Here you can find all the arguments: https://google-auth.readthedocs.io/en/latest/reference/google.oauth2.credentials.html load_args (`Optional`[`Dict`[`str`, `Any`]]) – Pandas options for loading BigQuery table into DataFrame. Here you can find all available arguments: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_gbq.html All defaults are preserved. save_args (`Optional`[`Dict`[`str`, `Any`]]) – Pandas options for saving DataFrame to BigQuery table. Here you can find all available arguments: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_gbq.html All defaults are preserved, but “progress_bar”, which is set to False.
Raises:	`DataSetError` – When `load_args['location']` and `save_args['location']` are different.
Return type:	`None`

exists()¶

Checks whether a data set’s output already exists by calling the provided _exists() method.

Return type:	`bool`
Returns:	Flag indicating whether the output already exists.
Raises:	`DataSetError` – when underlying exists method raises error.

classmethod from_config(name, config, load_version=None, save_version=None)¶

Create a data set instance using the configuration provided.

Parameters:	name (`str`) – Data set name. config (`Dict`[`str`, `Any`]) – Data set config dictionary. load_version (`Optional`[`str`]) – Version string to be used for `load` operation if the data set is versioned. Has no effect on the data set if versioning was not enabled. save_version (`Optional`[`str`]) – Version string to be used for `save` operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.
Return type:	`AbstractDataSet`
Returns:	An instance of an `AbstractDataSet` subclass.
Raises:	`DataSetError` – When the function fails to create the data set from its config.

load()¶

Loads data by delegation to the provided load method.

Return type:	`Any`
Returns:	Data returned by the provided load method.
Raises:	`DataSetError` – When underlying load method raises error.

release()¶

Release any cached data.

Raises:	`DataSetError` – when underlying release method raises error.
Return type:	`None`

save(data)¶

Saves data by delegation to the provided save method.

Parameters:	data (`Any`) – the value to be saved by provided save method.
Raises:	`DataSetError` – when underlying save method raises error.
Return type:	`None`