# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['cachesql']

package_data = \
{'': ['*']}

install_requires = \
['pandas>=1.0.0,<2.0.0',
 'pyarrow>=2.0.0,<3.0.0',
 'sqlalchemy>=1.3.22,<2.0.0',
 'sqlparse>=0.4.1,<0.5.0']

setup_kwargs = {
    'name': 'cachesql',
    'version': '0.1.0',
    'description': 'Fast, resilient and reproducible data analysis with cached SQL queries',
    'long_description': '# CacheSQL: Fast, resilient and reproducible data analysis with cached SQL queries\n\n[![Package version](https://img.shields.io/pypi/v/cachesql.svg)](https://pypi.org/project/cachesql) [![Coverage Status](https://coveralls.io/repos/github/felipeam86/cachesql/badge.svg)](https://coveralls.io/github/felipeam86/cachesql) [![Build Status](https://travis-ci.com/felipeam86/cachesql.svg?branch=develop)](https://travis-ci.com/felipeam86/cachesql)\n\nCacheSQL is a simple library for making SQL queries with cache functionality. The main target of this library are\ndata scientists and data analysts that rely on [SQLalchemy](https://pypi.org/project/SQLAlchemy/) to query data from SQL\nand [pandas](https://pypi.org/project/pandas/) to do the heavy lifting in Python.\n\nThe key features are:\n\n- **Speed up development cycles**: Run your queries the first time against the database, get it from cache the next times.\n  You no longer need to wait for your queries anymore.\n- **Resilience**: If you loose access to the databases, you can rely on your cached results to run your code.\n- **Reproducibility**: By freezing the state of your queries results in cache, you guarantee reproducibility of your code.\n- **Simplify collaboration**: By sharing the state of your cache with colleagues, you can guarantee they will get the\n  same results as you.\n- **Your development code is your production code**: When ready to launch your code in production, simply turn off the\n  cache functionality!\n- **Reduce the load on production DBs**: By using the cache, you reduce the number of times you query the DBs.\n- **Fast, efficient and reliable cache**: This comes from the usage of the parquet format through\n  [pyarrow](https://pypi.org/project/pyarrow/).\n- **Simpler code**: No more added complexity on your code to load backups from disk or run the query if no backup exists.\n\n## The basics\nInstall with pip\n\n```bash\npip install cachesql\n```\n\n\nRun your queries once, get them from cache the next time!\n\n```python\n>>> from cachesql import Database\n>>> db = Database(uri="postgresql://user:password@localhost:5432/mydatabase")\n>>> expensive_query = "SELECT * FROM table WHERE {conditions}" #  <--- Imagine this is a very long and expensive query.\n>>> df = db.query(expensive_query)\nINFO:cachesql.sql:Querying \'mydatabase\'\nINFO:cachesql.sql:Finished in 0:23:04.005710s\nINFO:cachesql.sql:Results have been stored in cache\n```\n\nOk, that took 23 minutes, but I need to run my code again from scratch!\n\n```python\n>>> df = db.query(expensive_query)\nINFO:cachesql.sql:Querying \'mydatabase\'\nINFO:cachesql.sql:Loading from cache. #  <--- When you run it again, it will get the data from cache\nINFO:cachesql.sql:The cached query was executed on the 2021-01-03T20:06:21.401556 and lasted 0:23:04.005710s\n```\n\nPhew... that was fast! Although I know that there\'s new data on the DB, I want fresh data! -->\nUse the `force=True`flag:\n\n```python\n>>> df = db.query(expensive_query, force=True) #  <--- force=True will tell cachesql to refresh the cache.\nINFO:cachesql.sql:Querying \'mydatabase\'\nINFO:cachesql.sql:Finished in 0:23:10.023650s\nINFO:cachesql.sql:Results have been stored in cache\n```\n\nPerfect, now that my report is ready to go in production, I wan\'t to run this once a day without\nunnecessarily wasting disk space with cache -->  Use the `cache=False`flag:\n\n```python\n>>> df = db.query(expensive_query, cache=False) #  <--- For production ready code, you can turn off the cache\nINFO:cachesql.sql:Querying \'mydatabase\'\nINFO:cachesql.sql:Finished in 0:22:43.031210s\n```\nYou got your data and nothing is saved to cache!\n\n**NOTE**: By default `cachesql` has logging disabled. This is to allow the user to choose within\ntheir own environment how and when to log messages. If you want to see the log messages as in the\nprevious examples, add this line on top of your code:\n\n```python\nimport logging\nlogging.basicConfig(level=logging.INFO)\n```\n\n\n\n## Rationale\nExploratory data analysis requires doing numerous iterations to test different ideas and hypothesis.\nDuring some of these iterations, it is important to start from scratch and run your code from the\nbeginning to guarantee its integrity or simply refresh your environment (yep, that messy and unordered\njupyter notebook). The problem with this is that often we need to do expensive queries to get the\ninitial data. This poses several problems:\n\n\n1. We don\'t want to have to wait for the data each time. Development cycles should be fast if we\n   want to be efficient at our tasks. We also want to have the freedom to run from scratch our code\n   as much as possible.\n2. We don\'t want to overload our databases with the same query several times. We don\'t always have\n   the luxury to have dedicated DBs for data analysis and sometimes have to query production DBs.\n   I know, it sucks... but that\'s life and we want to keep a helthy relationship with whomever is\n   in charge of the DBs.\n3. Let\'s face it, we don\'t all work in perfect environments, infrastructure sometimes fail and we\n   loose access to the databases.\n\n\nTo remediate this, we all end up putting this type of logic in our codes in some way or another:\n```python\nfrom pathlib import Path\nimport pandas as pd\nfrom sqlalchemy import create_engine\n\nengine = create_engine("postgresql://user:password@localhost:5432/mydatabase")\ndata_backup = Path("data_backup.parquet")\nif data_backup.exists():\n    df = pd.read_parquet(backup_file)\nelse:\n    df = pd.read_sql(sql="SELECT * FROM table WHERE {conditions}", con=engine)\n    df.to_parquet(data_backup)\n```\n\nPerfect! You would think... It does the trick, but it comes with its downsides:\n\n1. If you want fresh data, you need to manually erase the backup or modify the code by hand. Not a\n   big deal, but very much prone to error!\n2. That is an overhead of 6 extra lines of code to add backup logic\n3. Some of our reports are one shot, but some others are meant to be run in production on a regular\n   basis. For those cases, you would need to erase the boilerplate backup code or to add more\n   boilerplate to avoid the backup in production.\n4. Some analyses require more than one query from different databases --> Multiply the previous\n   boilerplate by the number of queries.\n\n\nEnter `cachesql`: \n\n```python\nfrom cachesql import Database\ndb = Database(uri="postgresql://user:password@localhost:5432/mydatabase")\ndf = db.query("SELECT * FROM table WHERE something in (...)")\n```\n\nThe previous 10 lines are perfectly replaced by these 3 lines of code. Although those 3 lines do\nnot exploit `cachesql` at it\'s fullest. Here is a more complete example of the basic usage:\n\n\n```python\nfrom cachesql import Database\n\ndb1 = Database(uri="postgresql://user:password@localhost:5432/mydatabase")\ndb2 = Database(uri="sqlite:///db2.db")\n\ndef get_first_dataset(force=False, cache=True):\n    df = db1.query("SELECT * FROM table WHERE {conditions}", force=force, cache=cache)\n    # Do some operations on df\n    ...\n    return df\n\n\ndef get_second_dataset(force=False, cache=True):\n    df = db2.query("SELECT * FROM table WHERE {conditions}", force=force, cache=cache)\n    # Do some operations on df\n    ...\n    return df\n\n\ndef get_data(force=False, cache=True):\n    df1 = get_first_dataset(force=force, cache=cache)\n    df2 = get_second_dataset(force=force, cache=cache)\n    df = df1.merge(df2, on="common_column")\n    # Do some operations on df\n    ...\n    return df\n\n\nif __name__ == \'__main__\':\n    import argparse\n    parser = argparse.ArgumentParser()\n    parser.add_argument("--force", action=\'store_true\')\n    parser.add_argument("--no-cache", action=\'store_true\')\n    args = parser.parse_args()\n\n    df = get_data(force=args.force, cache=not args.no_cache)\n\n```\n\nSupose the previous code is stored on `report.py`. You can either import `get_data` in other modules\nand control cache with `force` and `cache`parameters, or you can run the report on top of your\njupyter notebook with the magic command\n[%run](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-run) \nto populate your environment with the required data:\n\n\n```jupyter\n%run report.py # For running  the report and rely on the cache\n```\n\n\n```jupyter\n%run report.py --force # Refresh the cache to get new data\n```\n\nOnce you have finalized your development cycles and `report.py` has the complete logic for generating\nyour report, you can run it in production withouth cache as follows:\n\n\n```bash\n$ python report.py --no-cache\n```\n\n\n## Acknowledgements\nThis package was proudly developed at the [Rwanda Revenue Authority](https://www.rra.gov.rw/) which\nkindly agreed to let me open source it. The internal version has been serving a growing team of data\nscientists since 2018. It has been a pilar of our infrastructure to guarantee fast development\ncycles, resilient workflow to infrastructure issues, reproducibility of our analysis and simplified\ncollaboration by sharing cache state among colleagues.\n',
    'author': 'Felipe Aguirre Martinez',
    'author_email': 'felipeam86@gmail.com',
    'maintainer': None,
    'maintainer_email': None,
    'url': 'https://github.com/felipeam86/cachesql',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'python_requires': '>=3.6.1,<4.0.0',
}


setup(**setup_kwargs)
