# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['base']

package_data = \
{'': ['*']}

install_requires = \
['PyYAML>=6.0,<7.0',
 'click>=8.0.0',
 'colorama>=0.4.4,<0.5.0',
 'numpy>=1.18.5',
 'pandas>=1.4.3,<2.0.0',
 'requests>=1.0.0',
 'ruamel.yaml>=0.17.21,<0.18.0',
 'scikit-learn>=0.23.0']

entry_points = \
{'console_scripts': ['base = base.cli:main']}

setup_kwargs = {
    'name': 'adansons-base',
    'version': '0.1.3',
    'description': 'Adansons Base',
    'long_description': '# Adansons Base Document\n\n  - [Product Concept](#product-concept)\n  - [0. Get Access Key](#0-get-access-key)\n  - [1. Installation](#1-installation)\n  - [2. Configuration](#2-configuration)\n    - [2.1 with CLI](#21-with-cli)\n    - [2.2 Environment Variables](#22-environment-variables)\n  - [3. Tutorial 1: Organize meta data and Create dataset](#3-tutorial-1-organize-meta-data-and-create-dataset)\n    - [Step 0. prepare sample dataset](#step-0-prepare-sample-dataset)\n    - [Step 1. create new project](#step-1-create-new-project)\n    - [Step 2. import data files](#step-2-import-data-files)\n    - [Step 3. import external metadata files](#step-3-import-external-metadata-files)\n    - [Step 4. filter and export dataset with CLI](#step-4-filter-and-export-dataset-with-cli)\n    - [Step 5. filter and export dataset with Python SDK](#step-5-filter-and-export-dataset-with-python-sdk)\n  - [4. API Reference](#4-api-reference)\n    - [4.1 Command Reference](#41-command-reference)\n    - [4.2 Python Reference](#42-python-reference)\n\n\n## Product Concept\n- Adansons Base is a data management tool that organizes metadata of unstructured data and creates and organizes datasets.\n- It makes dataset creation more effective, helps find essential insights from training results, and improves AI performance.\n\nMore detail\n↓↓↓\n\n- Medium\n  - https://medium.com/@KenichiHiguchi/3-things-you-need-to-deal-with-in-data-management-to-create-best-dataset-781177507fc2\n- Product Page\n  - https://adansons.wraptas.site\n\n---\n## 0. Get Access Key\n\nType your email into the form below to join our slack and get the access key.\n\nInvitation Form: https://share.hsforms.com/1KG8Hp2kwSjC6fjVwwlklZA8moen\n\n\n## 1. Installation\n\nAdansons Base contains Command Line Interface (CLI) and Python SDK, and you can install both with `pip` command.\n\n```bash\npip install git+https://github.com/adansons/base\n```\n\n> Note: if you want to use CLI in any directory, you have to install with the python globally installed on your computer.\n\n## 2. Configuration\n\n### 2.1 with CLI\n\nwhen you run any Base CLI command for the first time, Base will ask for your access key provided on our slack.\n\nthen, Base will verify the specified access key was correct.\n\nif you don\'t have an access key, please see [0. Get Access Key](#0-get-access-key).\n\nthis command will show you what projects you have\n\n```bash\nbase list\n```\n\n<details><summary>Output</summary>\n\n```\nWelcome to Adansons Base!!\n\nLet\'s start with your access key provided on our slack.\n\nPlease register your access_key: xxxxxxxxxx\n\nSuccessfully configured as xxxx@yyyy.com\n\nprojects\n========\n```\n</details>\n\n### 2.2 Environment Variables\n\nif you don’t want to configure interactively, you can use environment variables for configuration.\n\n`BASE_USER_ID` is used for the identification of users, this is the email address you submitted via our form.\n\n```bash\nexport BASE_ACCESS_KEY=xxxxxxxxxx\nexport BASE_USER_ID=xxxx@yyyy.com\n```\n\n## 3. Tutorial 1: Organize metadata and Create a dataset\n\nlet’s start the Base tutorial with the mnist dataset.\n\n### Step 0. prepare sample dataset\n\ninstall dependencies for download dataset at first.\n\n```bash\npip install pypng\n```\n\nthen, download a script for mnist from our Base repository\n\n```bash\ncurl -sSL https://raw.githubusercontent.com/adansons/base/main/download_mnist.py > download_mnist.py\n```\n\nrun the download-mnist script. you can specify any folder for downloading as the last argument(default “~/dataset/mnist”). if you run this command on Windows, please replace it with the windows path like “C:\\dataset\\mnist”\n\n```bash\npython3 ./download_mnist.py ~/dataset/mnist\n```\n\n> Note: Base can link the data files if you put them anywhere on the local computer. So if you already downloaded the mnist dataset, you can use it\n\nafter downloading, you can see data files in ~/dataset/mnist.\n\n```\n~\n└── dataset\n     └── mnist\n          ├── train\n          │ \t ├── 0\n          │ \t │   ├── 1.png\n          │ \t │   ├── ...\n          │ \t │   └── 59987.png\n          │ \t ├── ...\n          │ \t └── 9\n          └──\ttest\n                ├── 0\n                └── ...\n```\n\n### Step 1. create a new project\n\ncreate mnist project with [base new <project>](/docs/CLI.md#new) command.\n\n```bash\nbase new mnist\n```\n\n<details><summary>Output</summary>\n\n```\nYour Project UID\n----------------\nabcdefghij0123456789\n\nsave Project UID in the local file (~/.base/projects)\n```\n</details>\n\nBase will issue a Project Unique ID and automatically save it in a local file.\n\n### Step 2. import data files\n\nafter step 0, you have many png image files on the”~/dataset/mnist” directory.\n\nlet’s upload metadata related to their paths into the mnist project with the `base import` command.\n\n```bash\nbase import mnist --directory ~/dataset/mnist --extension png --parse "{dataType}/{label}/{id}.png"\n```\n\n> Note: if you changed the download folder, please replace “~/dataset/mnist” in the above command.\n\n<details><summary>Output</summary>\n\n```\nCheck datafiles...\nfound 70000 files with png extension.\nSuccess!\n```\n</details>\n\n### Step 3. import external metadata files\n\nif you have external metadata files, you can integrate them into the existing project database with the `—-external-file` option.\n\nin this time, we use `wrongImagesInMNISTTestset.csv` published on Github by youkaichao.\n\n[https://github.com/youkaichao/mnist-wrong-test](https://github.com/youkaichao/mnist-wrong-test)\n\nthis is the extra metadata that correct wrong label on the mnist test dataset.\n\nyou can evaluate your model more strictly and correctly by using these extra metadata with Base.\n\ndownload external CSV\n\n```bash\ncurl -SL https://raw.githubusercontent.com/youkaichao/mnist-wrong-test/master/wrongImagesInMNISTTestset.csv > ~/Downloads/wrongImagesInMNISTTestset.csv\n```\n\n```bash\nbase import mnist --external-file --path ~/Downloads/wrongImagesInMNISTTestset.csv -a dataType:test\n```\n\n<details><summary>Output</summary>\n\n```\n1 tables found!\nnow estimating the rule for table joining...\n\n1 table joining rule was estimated!\nBelow table joining rule will be applied...\n\nRule no.1\n\n        key \'index\'     ->      connected to \'id\' key on exist table\n        key \'originalLabel\'     ->      connected to \'label\' key on exist table\n        key \'correction\'        ->      newly added\n\n1 tables will be applied\nTable 1 sample record:\n        {\'index\': 8, \'originalLabel\': 5, \'correction\': \'-1\'}\n\nDo you want to perform table join?\n        Base will join tables with that rule described above.\n\n        \'y\' will be accepted to approve.\n\n        Enter a value: y\nSuccess!\n```\n</details>\n\n### Step 4. filter and export dataset with CLI\n\nnow, we are ready to create a dataset.\n\nlet’s pick up a part of data files, the label is 0, 1, or 2 for training, from project mnist with `base search <project>` command.\n\nyou can use `--conditions <value-only-search>` option for magical search filter and `--query <key-value-pair-search>` option for advanced filter.\n\nNote that the `--query` option can only use the value for searching.\n\n\n\nBe careful that you may get so large output on your console without the `-s, --summary` option.\n\nThe `--query` option\'s grammar is below.\n\n`--query {KeyName} {Operator} {Values}`\n\n- add 1 space between each section\n- **don\'t use space any other**\n\nYou can use these operators below in the query option.\n\n[operators]\n```\n  == : equal\n  != : not equal\n  >= : greater than\n  <= : less than\n  >  : greater\n  <  : less\n  in : inner list of Values\n  not in : not inner list of Values\n```\n\n(check [search docs](/docs/CLI.md#search) for more information).\n\n```bash\nbase search mnist --conditions "train" --query "label in [\'1\',\'2\',\'3\']"\n```\n\n> Note: in the query option, you have to specify each component as a string in the list without space like `“[’1’,’2’,’3’]”`, when you want to operate `in` or `not in` query.\n> \n\n<details><summary>Output</summary>\n\n```\n18831 files\n========\n\'/home/xxxx/dataset/mnist/train/1/42485.png\'\n...\n```\n</details>\n\n> Note: If you specify no conditions or query, Base will return whole data files.\n\nIf you want to use the \'OR search\' with the `--query` command, please use our Python SDK.\n\n### Step 5. filter and export dataset with Python SDK\n\nin python script, you can filter and export datasets easily and simply with `Project class` and `Files class`. (see [SDK docs](/docs/SDK.md#project-class))\n\n(If you don\'t have the packages below, please install them by using `pip`)\n```bash\npip install NumPy pillow torch torchvision\n```\n\n```python\nfrom base import Project, Dataset\nimport numpy as np\nfrom PIL import Image\n\n\n# export dataset as you want to use\nproject = Project("mnist")\nfiles = project.files(conditions="train", query=["label in [\'1\',\'2\',\'3\']"])\n\nprint(files[0])\n# this returns path-like `File` object\n# -> \'/home/xxxx/dataset/mnist/0/12909.png\'\nprint(files[0].label)\n# this returns the value of attribute \'lable\' of first `File` object\n# -> \'0\'\n\n# function to load image from path\n# this is necessary, if you want to use image in your dataset\n# because base Dataset class doesn\'t convert path to image\ndef preprocess_func(path):\n    image = Image.open(path)\n    image = image.resize((28, 28))\n    image = np.array(image)\n    return image\n\ndataset = Dataset(files, target_key="label", transform=preprocess_func)\n\n# you can also use dataset objects like this.\nfor data, label in dataset:\n    # data: an image-data. ndarray\n    # label: the label of an image data, like 0\n    pass\n\nx_train, x_test, y_train, y_test = dataset.train_test_split(split_rate=0.2)\n\n# or use with torch\nimport torch\nimport torchvision.transforms as transforms\nfrom PIL import Image\n\ndef preprocess_func(path):\n    image = transforms.ToTensor()(transforms.Resize((28, 28))(Image.open(path)))\n    return image\n\ndataset = Dataset(files, target_key="label", transform=preprocess_func)\nloader = torch.utils.data.DataLoader(dataset, batch_size=32, shuffle=True)\n```\n\nfinally, let’s try one of the most characteristic use cases on Adansons Base.\n\nin the external file, you imported in step.3, some mnist test data files are annotated as `“-1”` in the correction column. this means that it is difficult to classify that files even for a human.\n\nso, you should exclude that files from your dataset to evaluate your AI models more properly.\n\n```python\n# you can exclude files which have "-1" on "correction" with below code\neval_files = project.files(conditions="test", query=["correction != -1"])\n\nprint(len(eval_files))\n# this returns the number of files matched with requested conditions or query\n# -> 9963\n\neval_dataset = Dataset(eval_files, target_key="label", transform=preprocess_func)\n```\n\n## 4. API Reference\n\n### 4.1 Command Reference\n\n[Command Reference](/docs/CLI.md)\n\n### 4.2 Python Reference\n\n[Python Reference](/docs/SDK.md)\n',
    'author': 'Adansons Developers',
    'author_email': 'engineer@adansons.co.jp',
    'maintainer': None,
    'maintainer_email': None,
    'url': 'https://github.com/adansons/base',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'entry_points': entry_points,
    'python_requires': '>=3.8,<3.11',
}


setup(**setup_kwargs)
