# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['magictables']

package_data = \
{'': ['*'], 'magictables': ['docs/*']}

install_requires = \
['aiofiles>=24.1.0,<25.0.0',
 'bs4>=0.0.2,<0.0.3',
 'dateparser>=1.2.0,<2.0.0',
 'litellm>=1.41.25,<2.0.0',
 'neo4j>=5.22.0,<6.0.0',
 'pandas>=2.2.2,<3.0.0',
 'polars>=1.2.1,<2.0.0',
 'pyarrow>=17.0.0,<18.0.0',
 'python-dotenv>=1.0.1,<2.0.0',
 'requests>=2.32.3,<3.0.0',
 'scikit-learn>=1.5.1,<2.0.0',
 'spycy-aneeshdurg>=0.0.3,<0.0.4']

setup_kwargs = {
    'name': 'magictables',
    'version': '1.0.6',
    'description': 'An advanced Python library designed to streamline data preparation and exploration for data scientists',
    'long_description': '# MagicTables\n\nMagicTables is an advanced Python library designed to streamline data preparation and exploration for data scientists. It provides a declarative approach to data handling, allowing you to focus on analysis and model development without getting bogged down in data engineering tasks.\n\n## Core Features\n\n- **API to DataFrame Conversion**: Instantly convert API responses into usable DataFrames.\n- **Declarative Data Preparation**: Define your data requirements, not the implementation details.\n- **Dynamic Data Chaining**: Easily combine data from multiple sources with automatic type handling.\n- **Natural Language Queries**: Use plain English to transform and query your data.\n- **Hybrid Storage System**: Flexible storage options with a hybrid approach to caching and querying.\n- **Intelligent Data Retrieval**: Dynamically query and retrieve relevant data based on context.\n- **High-Performance Engine**: Powered by Polars for fast data processing.\n- **Automatic Data Type Handling**: Intelligent conversion and standardization of data types, including dates and complex structures.\n- **Smart Caching**: Efficient caching of data, queries, and transformations to speed up repeated analyses.\n- **Automatic Key Column Identification**: Intelligently identifies suitable key columns for joining datasets.\n- **Schema Detection**: Automatically detects and provides database schema, saving time on manual inspection.\n- **AI-Assisted Operations**: Leverages AI for generating API descriptions and pandas code for complex transformations.\n\n## Installation\n\n```bash\npip install magictables\n```\n\n## Usage Examples\n\n### Fetching Data from an API\n\n```python\nfrom magictables import MagicTable\nimport asyncio\nimport os\nfrom dotenv import load_dotenv\n\nload_dotenv()\nAPI_KEY = os.getenv("TMDB_API_KEY")\n\nasync def fetch_popular_movies():\n    popular_movies = await MagicTable.from_api(\n        f"https://api.themoviedb.org/3/movie/popular?api_key={API_KEY}"\n    )\n    print(popular_movies)\n\nasyncio.run(fetch_popular_movies())\n```\n\nOutput:\n\n```\nFetching popular movies...\nshape: (20, 17)\n┌─────────────┬─────────────┬────────────┬────────────┬───┬──────┬────────────┬───────┬────────────┐\n│ overview    ┆ original_la ┆ original_t ┆ total_page ┆ … ┆ page ┆ id         ┆ adult ┆ vote_count │\n│ ---         ┆ nguage      ┆ itle       ┆ s          ┆   ┆ ---  ┆ ---        ┆ ---   ┆ ---        │\n│ str         ┆ ---         ┆ ---        ┆ ---        ┆   ┆ f64  ┆ f64        ┆ f64   ┆ f64        │\n│             ┆ str         ┆ str        ┆ f64        ┆   ┆      ┆            ┆       ┆            │\n╞═════════════╪═════════════╪════════════╪════════════╪═══╪══════╪════════════╪═══════╪════════════╡\n│ As storm    ┆ 2024-01-23T ┆ Twisters   ┆ 45217.0    ┆ … ┆ 1.0  ┆ 718821.0   ┆ 0.0   ┆ 169.0      │\n│ season inte ┆ 00:00:00    ┆            ┆            ┆   ┆      ┆            ┆       ┆            │\n│ nsifies, t… ┆             ┆            ┆            ┆   ┆      ┆            ┆       ┆            │\n│ A young     ┆ es          ┆ Goyo       ┆ 45217.0    ┆ … ┆ 1.0  ┆ 1.19161e6  ┆ 0.0   ┆ 32.0       │\n│ autistic    ┆             ┆            ┆            ┆   ┆      ┆            ┆       ┆            │\n│ museum      ┆             ┆            ┆            ┆   ┆      ┆            ┆       ┆            │\n│ guide …     ┆             ┆            ┆            ┆   ┆      ┆            ┆       ┆            │\n│ ...         ┆ ...         ┆ ...        ┆ ...        ┆ … ┆ ...  ┆ ...        ┆ ...   ┆ ...        │\n└─────────────┴─────────────┴────────────┴────────────┴───┴──────┴────────────┴───────┴────────────┘\n```\n\n### Chaining API Calls\n\n```python\nasync def fetch_movie_details():\n    popular_movies = await MagicTable.from_api(\n        f"https://api.themoviedb.org/3/movie/popular?api_key={API_KEY}"\n    )\n\n    movie_details = await popular_movies.chain(\n        api_url=f"https://api.themoviedb.org/3/movie/{{id}}?api_key={API_KEY}",\n    )\n    print(movie_details)\n\nasyncio.run(fetch_movie_details())\n```\n\nOutput:\n\n```\nChaining API calls for movie details...\nshape: (20, 30)\n┌────────────┬────────────┬───────────┬───────────┬───┬─────────┬───────────┬──────────┬───────────┐\n│ overview   ┆ original_l ┆ original_ ┆ total_pag ┆ … ┆ runtime ┆ spoken_la ┆ status   ┆ tagline   │\n│ ---        ┆ anguage    ┆ title     ┆ es        ┆   ┆ ---     ┆ nguages   ┆ ---      ┆ ---       │\n│ str        ┆ ---        ┆ ---       ┆ ---       ┆   ┆ i64     ┆ ---       ┆ str      ┆ str       │\n│            ┆ str        ┆ str       ┆ f64       ┆   ┆         ┆ list[stru ┆          ┆           │\n│            ┆            ┆           ┆           ┆   ┆         ┆ ct[3]]    ┆          ┆           │\n╞════════════╪════════════╪═══════════╪═══════════╪═══╪═════════╪═══════════╪══════════╪═══════════╡\n│ As storm   ┆ 2024-01-23 ┆ Twisters  ┆ 45217.0   ┆ … ┆ 122     ┆ [{"Englis ┆ Released ┆ Chase.    │\n│ season int ┆ T00:00:00  ┆           ┆           ┆   ┆         ┆ h","en"," ┆          ┆ Ride.     │\n│ ensifies,  ┆            ┆           ┆           ┆   ┆         ┆ English"} ┆          ┆ Survive.  │\n│ t…         ┆            ┆           ┆           ┆   ┆         ┆ ]         ┆          ┆           |\n│ ...        ┆ ...        ┆ ...       ┆ ...       ┆ … ┆ ...     ┆ ...       ┆ ...      ┆ ...       │\n└────────────┴────────────┴───────────┴───────────┴───┴─────────┴───────────┴──────────┴───────────┘\n```\n\n### Natural Language Transformation\n\n```python\nasync def analyze_high_rated_movies():\n    popular_movies = await MagicTable.from_api(\n        f"https://api.themoviedb.org/3/movie/popular?api_key={API_KEY}"\n    )\n\n    movie_details = await popular_movies.chain(\n        api_url=f"https://api.themoviedb.org/3/movie/{{id}}?api_key={API_KEY}",\n    )\n\n    result = await movie_details.transform(\n        "Find popular movies with a vote average greater than 7.5"\n    )\n    print(result)\n\nasyncio.run(analyze_high_rated_movies())\n```\n\nOutput:\n\n```\nQuerying across chained data...\nquery 65898df8aeebe4b88058a9e8cbd22822\nshape: (3, 3)\n┌─────────────────────────┬──────────────┬─────────────────────┐\n│ title                   ┆ vote_average ┆ release_date        │\n│ ---                     ┆ ---          ┆ ---                 │\n│ str                     ┆ f64          ┆ datetime[ns]        │\n╞═════════════════════════╪══════════════╪═════════════════════╡\n│ Deadpool & Wolverine    ┆ 7.8          ┆ 2024-07-24 00:00:00 │\n│ Furiosa: A Mad Max Saga ┆ 7.648        ┆ 2024-05-22 00:00:00 │\n│ Inside Out 2            ┆ 7.643        ┆ 2024-06-11 00:00:00 │\n└─────────────────────────┴──────────────┴─────────────────────┘\n```\n\n### Comprehensive Example\n\nHere\'s a more comprehensive example that demonstrates multiple features of MagicTables:\n\n```python\nfrom magictables import MagicTable\nimport os\nfrom dotenv import load_dotenv\nimport asyncio\n\nload_dotenv()\nAPI_KEY = os.getenv("TMDB_API_KEY")\n\nasync def main():\n    # Create a MagicTable instance\n    mt = MagicTable()\n\n    # Fetch popular movies\n    popular_movies = await MagicTable.from_api(\n        f"https://api.themoviedb.org/3/movie/popular?api_key={API_KEY}"\n    )\n\n    # Chain API calls for movie details\n    movie_details = await popular_movies.chain(\n        api_url=f"https://api.themoviedb.org/3/movie/{{id}}?api_key={API_KEY}",\n    )\n\n    # Use natural language transformation\n    result = await movie_details.transform(\n        "Find popular movies with a vote average greater than 7.5"\n    )\n    print(result)\n\nif __name__ == "__main__":\n    asyncio.run(main())\n```\n\nThis example demonstrates:\n\n1. API data fetching\n2. API chaining for detailed information\n3. Natural language querying\n4. Automatic data type handling and caching (behind the scenes)\n\nThe output of this comprehensive example would be similar to the output of the Natural Language Transformation example shown above.\n\n### Converting to Pandas DataFrame\n\nMagicTables uses Polars DataFrames internally for high-performance data processing. However, you can easily convert the results to pandas DataFrames when needed. Here\'s how you can do it:\n\n```python\nimport pandas as pd\nfrom magictables import MagicTable\nimport asyncio\nimport os\nfrom dotenv import load_dotenv\n\nload_dotenv()\nAPI_KEY = os.getenv("TMDB_API_KEY")\n\nasync def convert_to_pandas():\n    # Fetch movie data\n    popular_movies = await MagicTable.from_api(\n        f"https://api.themoviedb.org/3/movie/popular?api_key={API_KEY}"\n    )\n\n    # Perform a transformation\n    high_rated_movies = await popular_movies.transform(\n        "Find movies with a vote average greater than 7.5"\n    )\n\n    # Convert to pandas DataFrame\n    pandas_df = high_rated_movies.to_pandas()\n\n    # Now you can use pandas operations\n    print(pandas_df.head())\n    print(pandas_df.describe())\n\nasyncio.run(convert_to_pandas())\n```\n\nOutput:\n\n```\n                 title  vote_average release_date\n0  Deadpool & Wolverine          7.8   2024-07-24\n1    Furiosa: A Mad Max          7.6   2024-05-22\n2         Inside Out 2          7.6   2024-06-11\n\n       vote_average\ncount      3.000000\nmean       7.666667\nstd        0.115470\nmin        7.600000\n25%        7.600000\n50%        7.600000\n75%        7.700000\nmax        7.800000\n```\n\nIn this example, we:\n\n1. Fetch movie data using MagicTables\n2. Perform a transformation to find high-rated movies\n3. Convert the result to a pandas DataFrame using the `to_pandas()` method\n4. Use pandas operations like `head()` and `describe()` on the converted DataFrame\n\nThis allows you to seamlessly integrate MagicTables with existing pandas-based workflows or libraries that expect pandas DataFrames as input.\n\n## Advanced Features\n\n### Automatic Data Type Handling\n\nMagicTables automatically handles various data types, including dates, numbers, and complex data structures like lists and dictionaries. This feature saves time and reduces errors in data preprocessing.\n\n### Smart Caching\n\nThe library implements an intelligent caching system that stores data, queries, and transformations. This significantly speeds up repeated analyses and reduces the load on external data sources.\n\n### Automatic Key Column Identification\n\nWhen joining datasets, MagicTables attempts to automatically identify the most suitable key columns, simplifying the process of merging related data.\n\n### Schema Detection\n\nMagicTables can automatically detect and provide the schema of your data, giving you a quick overview of your dataset\'s structure without manual inspection.\n\n### AI-Assisted Operations\n\nThe library leverages AI to generate API descriptions and even pandas code for complex transformations based on natural language queries, further simplifying the data science workflow.\n\n## Why MagicTables?\n\n1. **Flexible Data Handling**: Utilize a hybrid approach for data storage and retrieval, combining the benefits of in-memory processing and optional persistent storage.\n2. **Faster Iteration**: Reduce time spent on data preparation, allowing more focus on model development and analysis.\n3. **No Data Engineering Bottleneck**: Perform complex data operations without relying on data engineering support.\n4. **Intuitive Interface**: Use natural language for data transformations and queries.\n5. **Performance**: Leverages efficient caching and data processing techniques without requiring a dedicated database.\n6. **Flexibility**: Works with various data sources (APIs, databases, files) seamlessly.\n7. **Reduced Boilerplate**: Eliminate repetitive code for data fetching, cleaning, and transformation.\n8. **Exploratory Freedom**: Quickly test ideas and hypotheses without complex setup.\n\n## Configuration\n\n### Environment Variables\n\nMagicTables requires several environment variables to be set for proper functionality. You can set these in a `.env` file in your project root or in your system environment.\n\nRequired environment variables:\n\n- `NEO4J_URI`: The URI of your Neo4j database (e.g., `bolt://localhost:7687`)\n- `NEO4J_USER`: Your Neo4j username\n- `NEO4J_PASSWORD`: Your Neo4j password\n- `OPENAI_API_KEY`: Your OpenAI API key for natural language processing features\n\nOptional environment variables:\n\n- `OPENAI_BASE_URL`: Custom base URL for OpenAI API (default: https://openrouter.ai/api/v1/chat/completions)\n- `OPENAI_MODEL`: Specific OpenAI model to use (default: gpt-4o-mini)\n- `LLM_PROVIDER`: The LLM provider to use (options: "openai", "openrouter", "ollama"; default: "openai")\n- `OPENROUTER_API_KEY`: Your OpenRouter API key (if using OpenRouter as LLM provider)\n- `OLLAMA_API_KEY`: Your Ollama API key (if using Ollama as LLM provider)\n- `EMBEDDING_PROVIDER`: The provider to use for embeddings (e.g., "openai", "cohere", "azure", etc.)\n- `EMBEDDING_MODEL`: The specific model to use for embeddings\n- `{PROVIDER}_API_KEY`: The API key for the chosen provider (e.g., `OPENAI_API_KEY`)\n- `{PROVIDER}_API_BASE`: The API base URL for the chosen provider (if needed)\n- `AI_PROVIDER`: The provider to use for the `call_ai_model` function\n- `JINA_API_KEY`: Your Jina AI API key for embedding generation and document parsing\n\n### Embedding and LLM Configuration\n\nMagicTables now supports flexible configuration for embedding generation and Language Model interactions. To use different embedding or LLM providers, simply set the appropriate environment variables as listed above.\n\n### Neo4j Configuration\n\nIf you want to store data and leverage full data querying capabilities, you need to set up a Neo4j database. Here are the steps to configure Neo4j:\n\n1. Install Neo4j: Download and install Neo4j from the [official website](https://neo4j.com/download/).\n\n2. Start Neo4j: Start your Neo4j server either through the Neo4j Desktop application or via command line.\n\n3. Create a new database or use an existing one.\n\n```bash\ndocker run \\\n    --name neo4j-magictables \\\n    -p 7474:7474 -p 7687:7687 \\\n    -e NEO4J_AUTH=neo4j/your_password \\\n    -e NEO4J_apoc_export_file_enabled=true \\\n    -e NEO4J_apoc_import_file_enabled=true \\\n    -e NEO4J_apoc_import_file_use__neo4j__config=true \\\n    -e NEO4J_PLUGINS=["graph-data-science"] \\\n    neo4j:latest\n```\n\n4. Set the environment variables:\n\n   - Set `NEO4J_URI` to the URI of your Neo4j instance (e.g., `bolt://localhost:7687`)\n   - Set `NEO4J_USER` to your Neo4j username (default is usually "neo4j")\n   - Set `NEO4J_PASSWORD` to your Neo4j password\n\n5. Ensure your Neo4j database is running before using MagicTables.\n\nExample `.env` file:\n\n```\nNEO4J_URI=bolt://localhost:7687\nNEO4J_USER=neo4j\nNEO4J_PASSWORD=your_password\nOPENAI_API_KEY=your_openai_api_key\nJINA_API_KEY=your_jina_api_key\nLLM_PROVIDER=openai\n```\n\nMake sure to add `.env` to your `.gitignore` file to avoid exposing sensitive information.\n\n## Fallback Behavior\n\n(Work In Progress - we want to make the library not be 100% reliant on a database for neo4j for every installation.)\nWhen Neo4j is not configured or unavailable, MagicTables uses a hybrid driver that falls back to local file storage with JSON-based caching. This allows basic functionality to continue working, but with some limitations:\n\n1. Data persistence is limited to what can be efficiently stored in JSON format.\n2. Complex graph operations that rely on Neo4j\'s capabilities are not available.\n3. Performance may degrade for large datasets compared to using Neo4j.\n4. Cross-session persistence is limited to the data that can be saved in the cache file.\n\nTo get the full benefits of MagicTables, including efficient caching, complex graph operations, and robust cross-session persistence, it\'s recommended to set up and use Neo4j as described in the Configuration section.\n\n## Contributing\n\nContributions are welcome. Please see [CONTRIBUTING.md](CONTRIBUTING.md) for details.\n\n## License\n\nMagicTables is released under the GNU General Public License v3.0 (GPL-3.0). See the [LICENSE](LICENSE) file for details.\n',
    'author': 'Lewis',
    'author_email': 'lewis@magictables.org',
    'maintainer': 'None',
    'maintainer_email': 'None',
    'url': 'None',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'python_requires': '>=3.10',
}


setup(**setup_kwargs)
