# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['cartwright',
 'cartwright.categories',
 'cartwright.datasets',
 'cartwright.models',
 'cartwright.resources']

package_data = \
{'': ['*']}

install_requires = \
['arrow==1.0.3',
 'faker>=14.0',
 'fuzzywuzzy==0.18.0',
 'joblib==1.0.1',
 'numpy>=1.19',
 'pandas>=1.1',
 'pydantic==1.8.2',
 'python-levenshtein==0.20.7',
 'scipy>=1.5',
 'torch>=1.8',
 'torchvision>=0.9']

entry_points = \
{'console_scripts': ['cartwright = cartwright.categorize:main']}

setup_kwargs = {
    'name': 'cartwright',
    'version': '0.0.1',
    'description': 'A recurrent neural network paired with heuristic methods that automatically infer geospatial, temporal and feature columns',
    'long_description': '# Geotime Classify\n\n![Tests](https://github.com/jataware/geotime_classify/actions/workflows/tests.yml/badge.svg)\n\nThis model is a recurrent neural network that uses LSTM to learn text classification. The goal of this project was for a given spreadsheet where we expect some kind of geospatial and temporal columns, can we automatically infer things like:\n\n-   Country\n-   Admin levels (0 through 3)\n-   Timestamp (from arbitrary formats)\n-   Latitude\n-   Longitude\n-   Which column likely contains the "feature value"\n-   Which column likely contains a modifier on the  `feature`\n\n The model and transformation code can be used locally by installing the pip package or downloaded the github repo and following the directions found in /training_model/README.md.\n\n## Pip install geotime classify\nFirst you need to install numpy, scipy, pandas, joblib, pip, torch and torchvision.\n\n    pip install -r requirements.txt\n\n  or \n\n    conda install -c conda-forge numpy\n    conda install -c conda-forge scipy\n    conda install -c conda-forge scikit-learn\n    conda install -c conda-forge matplotlib\n    conda install -c conda-forge pandas\n    conda install -c conda-forge joblib\n    conda install -c conda-forge pip\n    conda install pytorch torchvision cpuonly -c pytorch\n    \n    \n    \nNow you can pip install the geotime_classify repo. To pip install this repo use:\n\n    pip install geotime-classify\n \n \nOnce it is installed you can instantiate the geotime_classify with the number of random samples (n) you want to take from each column of your csv. To take 100 samples from each column run. In most cases more samples of each column will result is more accurate classifications, however it will increase the time of processing. \n\n    from geotime_classify import geotime_classify as gc\n    GeoTimeClass = gc.GeoTimeClassify(1000)\n\nNow we have our GeoTimeClassify class instantiated we can use the ***columns_classified*** function. \n\n   \n ### geotime_classify.columns_classified(path)\n  The next function is ***columns_classified***. This function returns an array that classifies each column in our csv by using a combination of the predictions from the LSTM model along with validation code for each classification. \n  \n\n    c_classified=GeoTimeClass.columns_classified(\'pathtocsv\')\n    print(c_classified)\n\n  Possible information returned for each column are:\n  1. **\'column\'**:Column name in csv\n  2. **\'classification\'**: an array with a few possible values:\n      - *\'category\'*: The final classified name for the column** most important return\n      - *\'subcategory\'*: This will be returned if there is addition sub categories for the classification. E.g. [{\'Category\': \'Geo\', \'type\': \'Latitude (number)\'}]\n      - *\'format\'*: If the column is classified as a date it will give the best guess at the format for that date column. \n      - *\'time_resolution\'*: for temporal columns, the automatically detected density of timestamps in this column.\n      - *\'parser\'*: This lets you know what parser was used for determining if it was a valid date or not. The two options are \'Arrow\' and \'Util\' which represent the arrow and dateutil libraries respectively.\n      - *\'dayFirst\'*: A boolean value. If the column is classified as a date the validation code will try to test if day or month come first in the format, which is necessary for accurate date standardization.\n      - *\'match_type\'*: How the classification was made. LSTM for the model, fuzzy for fuzzywuzzy match on column headers.\n\n5. **\'fuzzyColumn\'**: This is returned if the column name is similar enough to any word in a list of interest. Shown below.\n    [  \n    "Date",  \n    "Datetime",  \n    "Timestamp",  \n    "Epoch",  \n    "Time",  \n    "Year",  \n    "Month",  \n    "Lat",  \n    "Latitude",  \n    "lng",  \n    "Longitude",  \n    "Geo",  \n    "Coordinates",  \n    "Location",  \n    "location",  \n    "West",  \n    "South",  \n    "East",  \n    "North",  \n    "Country",  \n    "CountryName",  \n    "CC",  \n    "CountryCode",  \n    "State",  \n    "City",  \n    "Town",  \n    "Region",  \n    "Province",  \n    "Territory",  \n    "Address",  \n    "ISO2",  \n    "ISO3",  \n    "ISO_code",  \n    "Results",  \n]\n   \nFor the classification data there are a set number of possible classifications after the validation code. dayFirst can be \'True\' or \'False\'.\nThe \'format\' of a date classification are created using this reference sheet: https://strftime.org/ . \nPossible classifciation options: \n1. "category": "None"\n2. "category": "geo", "subcategory":"continent"\n3. "category": "geo", "subcategory":"country_name"\n4. "category": "geo", "subcategory":"state_name"\n5. "category": "geo", "subcategory":"city_name"\n6. "category": "geo", "subcategory":"ISO3"\n7. "category": "geo", "subcategory":"ISO2"\n12. "category": "geo", "subcategory": "longitude"\n13. "category": "geo", "subcategory": "latitude"\n19. "category": "unknown date" , "subcategory": None\n14. "category": "time", "subcategory": "date", "format": format, "Parser": "Util",  "DayFirst": dayFirst\n14. "category": "time", "subcategory": "date", "format": format, "Parser": "arrow",  "DayFirst": dayFirst\n51. "category": "Boolean"\n\n\n## Under the hood\nThe workflow consists of four main sections. \n1. Geotime Classify Model\n2. Heuristic Functions\n3. Column Header Fuzzy Match\n4. Automatic Temporal Resolution Detection\n\n\nWorkflow overview\n\n![Alt text](training_model/images/GeoTime_ClassifyWorkflow.png?raw=true "Workflows")\n\n\n## Geotime Classify Model\nThis model is a type recurrent neural network that uses LSTM to learn text classification. The model is trained on Fake data provided by [Faker](https://faker.readthedocs.io/en/master/). The goal was for a given spreadsheet where we expect some kind of geospatial and temporal columns, can we automatically infer things like:\n\n-   Country\n-   Admin levels (0 through 3)\n-   Timestamp (from arbitrary formats)\n-   Latitude\n-   Longitude\n-   Which column likely contains the "feature value"\n-   Which column likely contains a modifier on the  `feature`\n\nTo do this, we collected example data from Faker along with additional locally generated data. The model was built using pytorch. We used padded embedding, and LSTM cell, a linear layer and finally a LogSoftmax layer. This model was trained with a dropout of .2 to reduce overfitting and improving model performance. \n\n\t    self.embedding = nn.Embedding(vocab_size, embedding_dim)\n        self.lstm = nn.LSTM(embedding_dim, hidden_dim, num_layers=1)\n        self.hidden2out = nn.Linear(hidden_dim, output_size)\n        self.softmax = nn.LogSoftmax(dim=1)\n        self.dropout_layer = nn.Dropout(p=0.2)\nAfter a few iterations the model was performing well enough with accuracy hovering around 91 percent with 57 categories.\nConfusion Matrix:\n\n  ![Alt text](training_model/images/confusionMatrix.png?raw=true "Confusion Matrix")\n\nNow the model was able to ingest a string and categorize it into one the 57 categories. \n\n## The Heuristic functions\nThe heuristic functions ingest the prediction classifications from the model along with the original data for  validation tests. If the data passes the test associated with the classification the final classification is made and returned. If it failed it will return \'None\' or \'Unknown Date\' if the model classified the column as a date. If addition information is needed for future transformation of the data these functions try to capture that. For example if a column is classified as a Date the function will try validate the format and return it along with the classification.\n\n## Column Header Fuzzy Match\nThis is the most simple part of the workflow. For each column header we try to match that string to a word of interest. If there is a high match ratio the code returns the word of interest. For more info you can see Fuzzywuzzy docs [here](https://pypi.org/project/fuzzywuzzy/).\n\n\n## Automatic Temporal Resolution Detection\nIf a dataset contains data at evenly spaced intervals, the resolution is automatically detected according to the following process:\n1. convert all unique dates/times to unix timestamps\n2. sort the timestamps and compute the delta time between each\n3. find the median of the deltas\n4. characterize the uniformity of the deltas\n    - if all deltas are identical (to within some small ϵ), mark as `PERFECT`\n    - if the maximum deviation from the median is less than 1% the magnitude of the median, mark as (approximate) `UNIFORM`\n    - otherwise mark as `NOT_UNIFORM`\n5. if the deltas are perfectly or approximately uniform, find the closest matching time unit from a preset list:\n    - ~~`millisecond` (1e-3 * second)~~\n    - `second` (1)\n    - `minute` (60 * second)\n    - `hour` (60 * minute)\n    - `day` (24 * hour)\n    - `week` (7 * day)\n    - `year` (365 * day)\n    - `month` (year / 12)\n    - `decade` (10 * year + 2 * day)\n    - `century` (100 * year + 24 * day)\n    - `millennium` (1000 * year + 242 * day)\n\n    in the future, temporal units will be drawn from a more comprehensive units ontology\n6. convert the median delta to a proportion of the matched unit, and set as the temporal unit for the dataset\n\nCurrently milliseconds may experience issues due to floating point precision errors, and thus may not be detected by this process.\n\ntime resolutions are represented by a `TimeResolution` object with values: `uniformity` enum, `unit` enum, `density` value (in the unit given), and mean `error` value. If the detection process fails, the object will be `None`\n\n\n\n## Retraining geotime_classify with github repo\nTo get started read the README in the training_model directory. \n\n## Building the `pip` package\n\n```\nbump2version --current-version=XYZ patch setup.py\npython3 setup.py sdist bdist_wheel\npython3 -m twine upload dist/*\n```\n',
    'author': 'Kyle Marsh',
    'author_email': 'kyle@jataware.com',
    'maintainer': 'None',
    'maintainer_email': 'None',
    'url': 'None',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'entry_points': entry_points,
    'python_requires': '>=3.7,<4.0',
}


setup(**setup_kwargs)
