Metadata-Version: 2.1
Name: pyoptimus
Version: 21.8.0b0
Summary: Optimus is the missing framework for cleaning and pre-processing data in a distributed fashion.
Home-page: https://github.com/hi-primus/optimus/
Author: Argenis Leon
Author-email: argenisleon@gmail.com
License: APACHE
Keywords: datacleaner,data-wrangling,data-cleansing,data-profiling
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Description-Content-Type: text/markdown
Requires-Dist: numba (>=0.49)
Requires-Dist: jsonschema
Requires-Dist: simplejson (>=3.16.0)
Requires-Dist: cryptography (>=2.7)
Requires-Dist: imgkit (>=1.0.1)
Requires-Dist: packaging (>=19.1)
Requires-Dist: requests (>=2.20.0)
Requires-Dist: tqdm (>=4.28.1)
Requires-Dist: fastnumbers (>=2.2.1)
Requires-Dist: multipledispatch (>=0.6.0)
Requires-Dist: numpy (>=1.20.2)
Requires-Dist: deprecated (>=1.2.5)
Requires-Dist: setuptools (>=41.6.0)
Requires-Dist: Jinja2 (>=2.10.1)
Requires-Dist: pygments (>=2.2.0)
Requires-Dist: ipython (>=7.5.0)
Requires-Dist: humanize (>=0.5.1)
Requires-Dist: psutil (>=5.6.3)
Requires-Dist: ordered-set (>=3.1.1)
Requires-Dist: deepdiff (>=4.0.6)
Requires-Dist: statsmodels (>=0.10.1)
Requires-Dist: glom (>=19.10.0)
Requires-Dist: fastavro (>=0.22.7)
Requires-Dist: fast-histogram (>=0.8)
Requires-Dist: string-grouper (==0.1.0)
Requires-Dist: nltk
Requires-Dist: pendulum (>=2.0.5)
Requires-Dist: rply (==0.7.7)
Requires-Dist: pybigquery (==0.4.15)
Requires-Dist: pandavro (>=1.6.0)
Requires-Dist: openpyxl (>=3.0.5)
Requires-Dist: tabulate (>=0.8.7)
Requires-Dist: matplotlib (>=3.0.3)
Requires-Dist: seaborn (>=0.9.0)
Requires-Dist: wordninja (==2.0.0)
Requires-Dist: jellyfish (==0.8.2)
Requires-Dist: Metaphone (==0.6)
Requires-Dist: num2words (==0.5.10)
Requires-Dist: python-Levenshtein (==0.12.2)
Requires-Dist: xlrd (>=1.2.0)
Requires-Dist: s3fs (>=0.5.2)
Requires-Dist: python-magic (>=0.4.15) ; platform_system != "Windows"
Requires-Dist: python-magic-bin (==0.4.14) ; platform_system == "Windows"
Provides-Extra: ai
Requires-Dist: tensorflow (>=2.0.0b1) ; extra == 'ai'
Requires-Dist: keras (>=2.4.3) ; extra == 'ai'
Requires-Dist: nltk (>=3.4.5) ; extra == 'ai'
Provides-Extra: all
Requires-Dist: pyspark (>=2.4.1) ; extra == 'all'
Requires-Dist: findspark (>=1.3.0) ; extra == 'all'
Requires-Dist: dask[complete] (==2021.5.0) ; extra == 'all'
Requires-Dist: distributed (==2021.5.0) ; extra == 'all'
Requires-Dist: dask-ml (>=1.6.0) ; extra == 'all'
Requires-Dist: pyarrow (==1.0.1) ; extra == 'all'
Requires-Dist: coiled (>=0.0.30) ; extra == 'all'
Requires-Dist: vaex (==4.1) ; extra == 'all'
Requires-Dist: gputil ; extra == 'all'
Requires-Dist: dask[complete] (==2021.4.0) ; extra == 'all'
Requires-Dist: distributed (==2021.4.0) ; extra == 'all'
Requires-Dist: tensorflow (>=2.0.0b1) ; extra == 'all'
Requires-Dist: keras (>=2.4.3) ; extra == 'all'
Requires-Dist: nltk (>=3.4.5) ; extra == 'all'
Requires-Dist: sqlalchemy (==1.3.18) ; extra == 'all'
Provides-Extra: cudf
Requires-Dist: gputil ; extra == 'cudf'
Requires-Dist: dask[complete] (==2021.4.0) ; extra == 'cudf'
Requires-Dist: distributed (==2021.4.0) ; extra == 'cudf'
Provides-Extra: dask
Requires-Dist: dask[complete] (==2021.5.0) ; extra == 'dask'
Requires-Dist: distributed (==2021.5.0) ; extra == 'dask'
Requires-Dist: dask-ml (>=1.6.0) ; extra == 'dask'
Requires-Dist: pyarrow (==1.0.1) ; extra == 'dask'
Requires-Dist: coiled (>=0.0.30) ; extra == 'dask'
Provides-Extra: db
Requires-Dist: sqlalchemy (==1.3.18) ; extra == 'db'
Provides-Extra: docs
Requires-Dist: sphinx ; extra == 'docs'
Requires-Dist: pytest ; extra == 'docs'
Requires-Dist: mock ; extra == 'docs'
Requires-Dist: nose ; extra == 'docs'
Provides-Extra: lint
Requires-Dist: pep8 ; extra == 'lint'
Requires-Dist: pyflakes ; extra == 'lint'
Provides-Extra: spark
Requires-Dist: pyspark (>=2.4.1) ; extra == 'spark'
Requires-Dist: findspark (>=1.3.0) ; extra == 'spark'
Provides-Extra: test
Requires-Dist: pytest ; extra == 'test'
Requires-Dist: mock ; extra == 'test'
Requires-Dist: nose ; extra == 'test'
Provides-Extra: vaex
Requires-Dist: vaex (==4.1) ; extra == 'vaex'

[![Logo Optimus](https://raw.githubusercontent.com/hi-primus/optimus/develop-21.8/images/logoOptimus.png)](https://hi-optimus.com)

[![PyPI version](https://badge.fury.io/py/pyoptimus.svg)](https://badge.fury.io/py/pyoptimus) [![Updates](https://pyup.io/repos/github/hi-primus/optimus/shield.svg)](https://pyup.io/repos/github/hi-primus/optimus/)  [![GitHub release](https://img.shields.io/github/release/hi-primus/optimus.svg)](https://github.com/hi-primus/optimus/)
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/02b3ba0fe2b64d6297c6b8320f8b15a7)](https://www.codacy.com/app/argenisleon/Optimus?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=hi-primus/optimus&amp;utm_campaign=Badge_Grade)
[![Coverage Status](https://coveralls.io/repos/github/hi-primus/optimus/badge.svg?branch=develop-21.8)](https://coveralls.io/github/hi-primus/optimus?branch=develop-21.8) [![Mentioned in Awesome Data Science](https://awesome.re/mentioned-badge.svg)](https://github.com/bulutyazilim/awesome-datascience) ![Discord](https://img.shields.io/discord/579030865468719104.svg)
[![CalVer](https://img.shields.io/badge/calver-YY.MM.MICRO-22bfda.svg)](http://calver.org)

[![Downloads](https://pepy.tech/badge/pyoptimus)](https://pepy.tech/project/pyoptimus)
[![Downloads](https://pepy.tech/badge/pyoptimus/month)](https://pepy.tech/project/pyoptimus/month)
[![Downloads](https://pepy.tech/badge/pyoptimus/week)](https://pepy.tech/project/pyoptimus/week)

To launch a live notebook server to test optimus using binder or Colab, click on one of the following badges:

[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/hi-primus/optimus/develop-21.8)
[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/hi-primus/optimus/blob/develop-21.8/examples/10_min_from_spark_to_pandas_with_optimus.ipynb)

Optimus is the missing framework to profile, clean, process and do ML in a distributed fashion using Apache Spark(PySpark).

## Installation (pip): 

In your terminal just type  ```pip install pyoptimus```

### Requirements
* Python>=3.6

## Examples

You can go to the 10 minutes to Optimus [notebook](https://github.com/hi-primus/optimus/blob/develop-21.8/examples/10_min_from_pandas_to_spark_with_optimus.ipynb) where you can find the basic to start working.

Also you can go to the [examples](examples/) folder to found specific notebooks about data cleaning, data munging, profiling, data enrichment and how to create ML and DL models.

Besides check the [Cheat Sheet](https://htmlpreview.github.io/?https://github.com/hi-primus/optimus/blob/develop-21.8/docs/cheatsheet/optimus_cheat_sheet.html)

## Feedback

Feedback is what drive Optimus future, so please take a couple of minutes to help shape the Optimus' Roadmap:  http://bit.ly/optimus_survey 

Also if you want to a suggestion or feature request use https://github.com/hi-primus/optimus/issues

## Start Optimus

Start Optimus using ```"pandas"```, ```"dask"```, ```"cudf"``` or ```"dask_cudf"```.

```python
from optimus import Optimus
op = Optimus("pandas")
```

## Loading data

Now Optimus can load data in csv, json, parquet, avro, excel from a local file or URL.

```python
#csv
df = op.load.csv("../examples/data/foo.csv")

#json
df = op.load.json("../examples/data/foo.json")

# using a url
df = op.load.json("https://raw.githubusercontent.com/hi-primus/optimus/develop-21.8/examples/data/foo.json")

# parquet
df = op.load.parquet("../examples/data/foo.parquet")

# ...or anything else
df = op.load.file("../examples/data/titanic3.xls")
```

Also, you can load data from oracle, redshift, mysql and postgres.

## Saving Data

```python
#csv
df.save.csv("data/foo.csv")

# json
df.save.json("data/foo.json")

# parquet
df.save.parquet("data/foo.parquet")
```

You can also save data to oracle, redshift, mysql and postgres.

## Create dataframes

Also, you can create a dataframe from scratch
```python
df = op.create.dataframe({
    'A': ['a', 'b', 'c', 'd'],
    'B': [1, 3, 5, 7],
    'C': [2, 4, 6, None],
    'D': ['1980/04/10', '1980/04/10', '1980/04/10', '1980/04/10']
})
```

Using `display` you have a beautiful way to show your data with extra information like column number, column data type and marked white spaces.

```python
display(df)
```
![](readme/images/table.png)

## Cleaning and Processing

Optimus was created to make data cleaning a breeze. The API was designed to be super easy to newcomers and very familiar for people that comes from Pandas.
Optimus expands the standard DataFrame functionality adding `.rows` and `.cols` accessors.

For example you can load data from a url, transform and apply some predefined cleaning functions:

```python
new_df = df\
    .rows.sort("rank", "desc")\
    .cols.lower(["names", "function"])\
    .cols.date_format("date arrival", "yyyy/MM/dd", "dd-MM-YYYY")\
    .cols.years_between("date arrival", "dd-MM-YYYY", output_cols="from arrival")\
    .cols.remove_accents("names")\
    .cols.remove_special_chars("names")\
    .rows.drop(df["rank"]>8)\
    .cols.rename("*", str.lower)\
    .cols.trim("*")\
    .cols.unnest("japanese name", output_cols="other names")\
    .cols.unnest("last position seen", separator=",", output_cols="pos")\
    .cols.drop(["last position seen", "japanese name", "date arrival", "cybertronian", "nulltype"])
```

## Troubleshooting

If you have issues, see our [Troubleshooting Guide](troubleshooting.md)

## Contributing to Optimus

Contributions go far beyond pull requests and commits. We are very happy to receive any kind of contributions  
including: 

* [Documentation](https://github.com/hi-primus/optimus/tree/develop-21.8/docs/source) updates, enhancements, designs, or   bugfixes. 
* Spelling or grammar fixes. 
* README.md corrections or redesigns. 
* Adding unit, or functional [tests](https://github.com/hi-primus/optimus/tree/develop-21.8/tests)  
* Triaging GitHub issues -- especially determining whether an issue still persists or is reproducible.
* [Blogging, speaking about, or creating tutorials](https://hioptimus.com/category/blog/)   about Optimus and its many features. 
* Helping others on our official chats

## Backers and Sponsors

Become a [backer](https://opencollective.com/optimus#backer) or a [sponsor](https://opencollective.com/optimus#backer) and get your image on our README on Github with a link to your site. 

[![OpenCollective](https://opencollective.com/optimus/backers/badge.svg)](#backers) [![OpenCollective](https://opencollective.com/optimus/sponsors/badge.svg)](#sponsors)


