# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['deltalake']

package_data = \
{'': ['*']}

install_requires = \
['fsspec>=0.8.5,<0.9.0',
 'pandas>=1.1.5,<2.0.0',
 'pip>=20.3.3,<21.0.0',
 'pyarrow>=2.0.0,<3.0.0']

extras_require = \
{'aws': ['s3fs>=0.5.2,<0.6.0'],
 'azure': ['adlfs>=0.5.9,<0.6.0', 'aiohttp>=3.7.3,<4.0.0'],
 'gcp': ['gcsfs>=0.7.1,<0.8.0']}

setup_kwargs = {
    'name': 'delta-lake-reader',
    'version': '0.2.1',
    'description': 'Lightweight wrapper for reading Delta tables without Spark',
    'long_description': '![Build package](https://github.com/jeppe742/DeltaLakeReader/workflows/Build%20python%20package/badge.svg)\n# Delta Lake Reader\nThe [Delta](https://github.com/delta-io/delta) format, developed by Databricks, is often used to build data lakes.\n\nWhile it tries to solve many issues with data lakes, one of the downsides is that delta tables rely on Spark to read the data. If you only need to read a small table, this can introduce a lot of unnecessary overhead.\n\nThis package tries to fix this, by providing a lightweight python wrapper around the delta file format, **without** any Spark dependencies\n\n\n# Installation\nInstall the package using pip\n\n\n```\npip install delta-lake-reader\n```\nThis will only install the minimal dependencies for working with local file system.\nTo access Delta Tables stored in popular cloud storages, use one of the following commands, to include the cloud specific dependencies\n\n### Azure\n```\npip install delta-lake-reader[azure]\n```\n### Amazon Web Services (AWS)\n```\npip install delta-lake-reader[aws]\n```\n### Google Cloud Platform (GCP)\n```\npip install delta-lake-reader[gcp]\n```\n# Usage\nPackage is build on [PyArrow](https://arrow.apache.org/docs/python/) and [FSSpec](https://filesystem-spec.readthedocs.io/en/latest/index.html).\n\nThis means that you get all the features of PyArrow, like predicate pushdown, partition pruning and easy interoperability with Pandas.\n\nMeanwhile, FSSpec serves as a FileSystem agnostic backend, that lets you read files from many places, including popular cloud providers.\n\nTo read a DeltaTable, first create a `DeltaTable` object.\nThis will read the delta transaction log to find the current files, and get the schema. This will, however, **not** read any data.\nTo read the content of the table, call `to_table()` to get a `pyarrow.Table` object, or `to_pandas()` to get a `pandas.DataFrame`\n## Local file system\n\n```python\nfrom deltalake import DeltaTable\n\n# native file path. Can be relative or absolute\ntable_path = "somepath/mytable"\n\n# Get table as pyarrow table\ndf = DeltaTable(table_path).to_table()\n\n# Get table as pandas dataframe\ndf = DeltaTable(table_path).to_pandas()\n```\n## Azure\nThe Azure integration is based on [adlfs](https://github.com/dask/adlfs) package, developed by the [Dask](https://dask.org/) community.\n\n\n\n`credential` used to authenticate against the storage account, can be either a SAS token, Access Keys or one of the `azure.identity` classes. See [authentication using Azure SDK](https://docs.microsoft.com/en-us/python/api/azure-storage-blob/azure.storage.blob.aio.blobserviceclient?view=azure-python#parameters) for more information.\n\n```python\nfrom deltalake import DeltaTable\nfrom adlfs import AzureBlobFileSystem\n\n#example url  \'abfss://myContainer@myStorageAccount.dfs.core.windows.net/somepath/mytable\'\nfs = AzureBlobFileSystem(\n        account_name="myStorageAccount", \n        credential=\'...\'\n    )\ndf = DeltaTable("mycontainer/somepath/mytable", file_system=fs).to_pandas()\n\n```\n\n## Amazon Web Service (AWS)\nThe AWS integration is based on [s3fs](https://s3fs.readthedocs.io/en/latest/) package, developed by the [Dask](https://dask.org/) community. \n\nTo authenticate you can either specify the access key and secret, or since it\'s build on `boto`, use one of their methods for authentication. See [authentication using AWS SDK ](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) for more information.\n\n```python\nfrom deltalake import DeltaTable\nfrom s3fs import S3FileSystem\n\n#example url  \'s3://myBucket/somepath/mytable\'\nfs = S3FileSystem() #authenticate using environment variables, in this example\ndf = DeltaTable("myBucket/somepath/mytable", file_system=fs).to_pandas()\n\n```\n\n## Google Cloud Platform (GCP)\nThe GCP integration is based on the [gcsfs](https://gcsfs.readthedocs.io/en/latest/), developed by the [Dask](https://dask.org/) community.\n\nFor more information about authentication with GCP see the [gcsfs documentation](https://gcsfs.readthedocs.io/en/latest/#credentials) or the [GCP documentation](https://cloud.google.com/docs/authentication)\n\n```python\nfrom deltalake import DeltaTable\nfrom gcsfs import GCSFileSystem\n\n#example url  \'gs://myBucket/somepath/mytable\'\nfs = GCSFileSystem() #authenticate using environment variables, in this example\ndf = DeltaTable("myBucket/somepath/mytable", file_system=fs).to_pandas()\n\n```\n\n## Time travel\nOne of the features of the Delta format, is the ability to do [timetravel](https://docs.delta.io/0.3.0/delta-batch.html#query-an-older-snapshot-of-a-table-time-travel).\n\nThis can be done using the `as_version` method. Note that this currenly only support specific version, and not timestamp.\n```python\nfrom deltalake import DeltaTable\n\ndf = DeltaTable("somepath/mytable").as_version(5).to_pandas()\n```\nTimetraveling to a version that has been vacuumed, currently results in undefined behavior\n\n\n## Predicate Pushdown, Partition Pruning & Columnar file formats\nSince the resulting `DeltaTable` is based on the `pyarrow.DataSet`, you get many cool features for free. \n\nThe `DeltaTable.to_table` is inherited from `pyarrow.Dataset.to_table`. This means that you can include arguments like `filter`, which will do partition pruning and predicate pushdown. If you have a partitioned dataset, partition pruning can potentially reduce the data needed to be downloaded substantially. The predicate pushdown will not have any effect on the amount of data downloaded, but will reduce the dataset size when loaded into memory.\n\nFurther more, since the underlying parquet file format is columnar, you can select a subset of columns to be read from the files. This can be done by passing a list of column names to `to_table`.\n\nSee documentation of [to_pandas](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.to_pandas), or [to_table](https://arrow.apache.org/docs/python/generated/pyarrow.dataset.FileSystemDataset.html#pyarrow.dataset.FileSystemDataset.to_table) for documentation of all arguments\n\n```python\nimport pyarrow.dataset as ds\n\n#Predicate pushdown. \n#If the table is partitioned on age, it will also to partition pruning\ndf = DeltaTable("...").to_table(filter=ds.field("age")>=18).to_pandas()\n\n#Only load a subset of columns\ndf = DeltaTable("...").to_table(columns=["age","name"]).to_pandas()\n\n```\n\n[Read more about filtering data using PyArrow](https://arrow.apache.org/docs/python/dataset.html#filtering-data)\n\n\n\n\n## Bring Your Own Filesystem\nSince the implementation is using the FSSpec for filesystem abstraction, you can in principle use any FSSpec filesystem. [See more about available FSSpec interfaces](https://filesystem-spec.readthedocs.io/en/latest/api.html#built-in-implementations).\n\n```python\nfs = SomeFSSpecFilesystem()\ndf = DeltaTable(path, file_system=fs).to_pandas()\n```\n\n## Disclaimer\nDatabricks recently announced a stand alone reader for Delta tables in a [blogpost](https://databricks.com/blog/2020/12/22/natively-query-your-delta-lake-with-scala-java-and-python.html)\nThe stand alone reader is JVM based, but a [Rust library](https://github.com/delta-io/delta-rs) with python bindings is also mentioned. This, however, cannot be pip installed which may discourage many python developers. \nAlthough the idea for this library was made independently, some inspirations has been taken from the Rust library.\n# Read more\n[Delta transaction log](https://databricks.com/blog/2019/08/21/diving-into-delta-lake-unpacking-the-transaction-log.html)\n\n[PyArrow Documentation](https://arrow.apache.org/docs/index.html)\n\n[FSSpec Documentation](https://filesystem-spec.readthedocs.io/en/latest/)\n',
    'author': 'Jeppe Johan Waarkjær Olsen',
    'author_email': 'jeppe742@gmail.com',
    'maintainer': None,
    'maintainer_email': None,
    'url': 'https://github.com/jeppe742/DeltaLakeReader',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'extras_require': extras_require,
    'python_requires': '>=3.7,<4.0',
}


setup(**setup_kwargs)
