# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['connectorx', 'connectorx.tests']

package_data = \
{'': ['*']}

install_requires = \
['numpy>=1,<2']

extras_require = \
{'all': ['dask[dataframe]>=2021,<2022',
         'modin>=0.10',
         'pandas>=1,<2',
         'polars>=0.8',
         'pyarrow>=4'],
 'dask': ['dask[dataframe]>=2021,<2022', 'pandas>=1,<2'],
 'modin': ['modin>=0.10', 'pandas>=1,<2'],
 'pandas': ['pandas>=1,<2'],
 'polars': ['polars>=0.8', 'pyarrow>=4'],
 'pyarrow': ['pyarrow>=4']}

setup_kwargs = {
    'name': 'connectorx',
    'version': '0.2.3',
    'description': 'Load data from databases to dataframes, the fastest way.',
    'long_description': '# ConnectorX [![status][ci_badge]][ci_page] [![discussions][discussion_badge]][discussion_page]\n\n[ci_badge]: https://github.com/sfu-db/connector-x/workflows/ci/badge.svg\n[ci_page]: https://github.com/sfu-db/connector-x/actions\n[discussion_badge]: https://img.shields.io/badge/Forum-Github%20Discussions-blue\n[discussion_page]: https://github.com/sfu-db/connector-x/discussions\n\nLoad data from <img src="https://raw.githubusercontent.com/sfu-db/connector-x/main/assets/sources.gif" width="6.5%" style="margin-bottom: -2px"/> to <img src="https://raw.githubusercontent.com/sfu-db/connector-x/main/assets/destinations.gif" width="7%" style="margin-bottom: -2px"/>, the fastest way.\n\nConnectorX enables you to load data from databases into Python in the fastest and most memory efficient way.\n\nWhat you need is one line of code:\n\n```python\nimport connectorx as cx\n\ncx.read_sql("postgresql://username:password@server:port/database", "SELECT * FROM lineitem")\n```\n\nOptionally, you can accelerate the data loading using parallelism by specifying a partition column.\n\n```python\nimport connectorx as cx\n\ncx.read_sql("postgresql://username:password@server:port/database", "SELECT * FROM lineitem", partition_on="l_orderkey", partition_num=10)\n```\n\nThe function will partition the query by **evenly** splitting the specified column to the amount of partitions.\nConnectorX will assign one thread for each partition to load and write data in parallel.\nCurrently, we support partitioning on **numerical** columns (**cannot contain NULL**) for **SPJA** queries. \n\nCheck out more detailed usage and examples [here](#detailed-usage-and-examples). A general introduction of the project can be found in this [blog post](https://towardsdatascience.com/connectorx-the-fastest-way-to-load-data-from-databases-a65d4d4062d5).\n\n# Installation\n\n```bash\npip install connectorx\n```\n\n# Performance\n\nWe compared different solutions in Python that provides the `read_sql` function, by loading a 10x TPC-H lineitem table (8.6GB) from Postgres into a DataFrame, with 4 cores parallelism.\n\n## Time chart, lower is better.\n\n<p align="center"><img alt="time chart" src="https://raw.githubusercontent.com/sfu-db/connector-x/main/assets/pg-time.png"/></p>\n\n## Memory consumption chart, lower is better.\n\n<p align="center"><img alt="memory chart" src="https://raw.githubusercontent.com/sfu-db/connector-x/main/assets/pg-mem.png"/></p>\n\nIn conclusion, ConnectorX uses up to **3x** less memory and **21x** less time (**3x** less memory and **13x** less time compared with Pandas.). More on [here](https://github.com/sfu-db/connector-x/blob/main/Benchmark.md#benchmark-result-on-aws-r54xlarge).\n\n## How does ConnectorX achieve a lightening speed while keeping the memory footprint low?\n\nWe observe that existing solutions more or less do data copy multiple times when downloading the data.\nAdditionally, implementing a data intensive application in Python brings additional cost.\n\nConnectorX is written in Rust and follows "zero-copy" principle.\nThis allows it to make full use of the CPU by becoming cache and branch predictor friendly. Moreover, the architecture of ConnectorX ensures the data will be copied exactly once, directly from the source to the destination.\n\n## How does ConnectorX download the data?\n\nUpon receiving the query, e.g. `SELECT * FROM lineitem`, ConnectorX will first issue a `LIMIT 1` query `SELECT * FROM lineitem LIMIT 1` to get the schema of the result set.\n\nThen, if `partition_on` is specified, ConnectorX will issue `SELECT MIN($partition_on), MAX($partition_on) FROM (SELECT * FROM lineitem)` to know the range of the partition column.\nAfter that, the original query is split into partitions based on the min/max information, e.g. `SELECT * FROM (SELECT * FROM lineitem) WHERE $partition_on > 0 AND $partition_on < 10000`.\nConnectorX will then run a count query to get the partition size (e.g. `SELECT COUNT(*) FROM (SELECT * FROM lineitem) WHERE $partition_on > 0 AND $partition_on < 10000`). If the partition\nis not specified, the count query will be `SELECT COUNT(*) FROM (SELECT * FROM lineitem)`.\n\nFinally, ConnectorX will use the schema info as well as the count info to allocate memory and download data by executing the queries normally.\n\nOnce the downloading begins, there will be one thread for each partition so that the data are downloaded in parallel at the partition level. The thread will issue the query of the corresponding\npartition to the database and then write the returned data to the destination row-wise or column-wise (depends on the database) in a streaming fashion. \n\n#### How to specify the partition number?\n\n`partition_num` will determine how many queries we are going to split from the original one and issue to the database. Underlying, we use [rayon](https://github.com/rayon-rs/rayon) as our parallel executor, which adopts a pool of threads to handle each partitioned query. The number of threads in the pool equals to the number of logical cores on the machine. It is recommended to set the `partition_num` to the number of available logical cores.\n\n#### How to choose the partition column?\n\n`partition_on` specifies on which column we will do the partition as above procedure. In order to achieve the best performance, it is ideal that each partitioned query will return the same number of rows. And since we partition the column evenly, it is recommended that the numerical `partition_on` column is evenly distributed. Whether a column has index or not might also affect the performance depends on the source database. You can give it a try if you have multiple candidates. Also, you can manually partition the query if our partition method cannot match your need. ConnectorX will still return a whole dataframe with all the results of the list of queries you input.\n\n\n# Supported Sources & Destinations\n\nSupported protocols, data types and type mappings can be found [here](Types.md).\nFor more planned data sources, please check out our [discussion](https://github.com/sfu-db/connector-x/discussions/61).\n\n## Sources\n- [x] Postgres\n- [x] Mysql\n- [x] Mariadb (through mysql protocol)\n- [x] Sqlite\n- [x] Redshift (through postgres protocol)\n- [x] Clickhouse (through mysql protocol)\n- [x] SQL Server\n- [x] Azure SQL Database (through mssql protocol)\n- [x] Oracle\n- [ ] Big Query - In Progress\n- [ ] ...\n\n## Destinations\n- [x] Pandas\n- [x] PyArrow\n- [x] Modin (through Pandas)\n- [x] Dask (through Pandas)\n- [x] Polars (through PyArrow)\n  \n# Detailed Usage and Examples\n\nRust docs: [stable](https://docs.rs/connectorx) [nightly](https://sfu-db.github.io/connector-x/connectorx/)\n\n## API\n\n```python\nconnectorx.read_sql(conn: str, query: Union[List[str], str], *, return_type: str = "pandas", protocol: str = "binary", partition_on: Optional[str] = None, partition_range: Optional[Tuple[int, int]] = None, partition_num: Optional[int] = None)\n```\n\nRun the SQL query, download the data from database into a Pandas dataframe.\n\n## Parameters\n- `conn: str`: Connection string URI.\n  - General supported URI scheme: `(postgres|postgressql|mysql|mssql)://username:password@addr:port/dbname`.\n  - For now sqlite only support absolute path, example: `sqlite:///home/user/path/test.db`.\n  - Please check out [here](Types.md) for more connection uri parameters supported for each database (e.g. trusted_connection for Mssql, sslmode for Postgres)\n- `query: Union[str, List[str]]`: SQL query or list of SQL queries for fetching data.\n- `return_type: str = "pandas"`: The return type of this function. It can be `arrow`, `pandas`, `modin`, `dask` or `polars`.\n- `protocol: str = "binary"`: The protocol used to fetch data from source, default is `binary`. Check out [here](Types.md) to see more details.\n- `partition_on: Optional[str]`: The column to partition the result.\n- `partition_range: Optional[Tuple[int, int]]`: The value range of the partition column.\n- `partition_num: Optioinal[int]`: The number of partitions to generate.\n- `index_col: Optioinal[str]`: The index column to set for the result dataframe. Only applicable when `return_type` is `pandas`, `modin` or `dask`. \n\n## Examples\n- Read a DataFrame from a SQL using a single thread\n\n  ```python\n  import connectorx as cx\n\n  postgres_url = "postgresql://username:password@server:port/database"\n  query = "SELECT * FROM lineitem"\n\n  cx.read_sql(postgres_url, query)\n  ```\n\n- Read a DataFrame parallelly using 10 threads by automatically partitioning the provided SQL on the partition column (`partition_range` will be automatically  queried if not given)\n\n  ```python\n  import connectorx as cx\n\n  postgres_url = "postgresql://username:password@server:port/database"\n  query = "SELECT * FROM lineitem"\n\n  cx.read_sql(postgres_url, query, partition_on="l_orderkey", partition_num=10)\n  ```\n\n- Read a DataFrame parallelly using 2 threads by manually providing two partition SQLs (the schemas of all the query results should be same)\n\n  ```python\n  import connectorx as cx\n\n  postgres_url = "postgresql://username:password@server:port/database"\n  queries = ["SELECT * FROM lineitem WHERE l_orderkey <= 30000000", "SELECT * FROM lineitem WHERE l_orderkey > 30000000"]\n\n  cx.read_sql(postgres_url, queries)\n\n  ```\n  \n- Read a DataFrame parallelly using 4 threads from a more complex query\n\n  ```python\n  import connectorx as cx\n\n  postgres_url = "postgresql://username:password@server:port/database"\n  query = f"""\n  SELECT l_orderkey,\n         SUM(l_extendedprice * ( 1 - l_discount )) AS revenue,\n         o_orderdate,\n         o_shippriority\n  FROM   customer,\n         orders,\n         lineitem\n  WHERE  c_mktsegment = \'BUILDING\'\n         AND c_custkey = o_custkey\n         AND l_orderkey = o_orderkey\n         AND o_orderdate < DATE \'1995-03-15\'\n         AND l_shipdate > DATE \'1995-03-15\'\n  GROUP  BY l_orderkey,\n            o_orderdate,\n            o_shippriority \n  """\n\n  cx.read_sql(postgres_url, query, partition_on="l_orderkey", partition_num=4)\n\n  ```\n\n# Next Plan\n\nCheckout our [discussion][discussion_page] to participate in deciding our next plan!\n\n# Historical Benchmark Results\n\nhttps://sfu-db.github.io/connector-x/dev/bench/\n\n# Developer\'s Guide\nPlease see [Developer\'s Guide](https://github.com/sfu-db/connector-x/blob/main/CONTRIBUTING.md) for information about developing ConnectorX.\n\n# Supports\n\nYou are always welcomed to:\n1. Ask questions in stackoverflow. Make sure to have #connectorx attached.\n2. Ask questions & propose new ideas in our [forum][discussion_page].\n3. Ask questions & join the discussion & send direct messages to us in our [discord](https://discord.gg/xwbkFNk) (under `CONNECTOR` category)\n\n# Organizations and Projects using ConnectorX\n\n[<img src="https://raw.githubusercontent.com/pola-rs/polars-static/master/logos/polars-logo-dark.svg" height="100" style="margin-bottom: -2px"/>](https://github.com/pola-rs/polars)\n[<img src="https://raw.githubusercontent.com/sfu-db/dataprep/develop/assets/logo.png" height="100" style="margin-bottom: -2px"/>](https://dataprep.ai/)\n\nTo add your project/organization here, reply our post [here](https://github.com/sfu-db/connector-x/discussions/146)\n',
    'author': 'SFU Database System Lab',
    'author_email': 'dsl.cs.sfu@gmail.com',
    'maintainer': 'Weiyuan Wu',
    'maintainer_email': 'youngw@sfu.ca',
    'url': None,
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'extras_require': extras_require,
    'python_requires': '>=3.7.1,<4.0.0',
}


setup(**setup_kwargs)
