Metadata-Version: 2.1
Name: spiderutil
Version: 0.1.6
Summary: Utilities for spider.
Home-page: https://github.com/Thesharing/spider-utility
Author: Thesharing
Author-email: 
License: MIT
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Topic :: Utilities
Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, <4
Description-Content-Type: text/markdown
Requires-Dist: redis (>=3.3.0)
Requires-Dist: pymongo (>=3.8.0)
Requires-Dist: requests (>=2.20.0)

# Spider Utility

![Version](https://img.shields.io/pypi/v/spiderutil)
![Download](https://img.shields.io/pypi/dm/spiderutil)
![License](https://img.shields.io/pypi/l/spiderutil)
![Status](https://img.shields.io/pypi/status/spiderutil)

Utilities for spider, including connectors to databases, path generators, loggers and exceptions.

## Docs

### Connector

#### MongoDB

#### Redis

#### LocalFolder

#### LocalFile

### Network

#### Session

#### User Agent

### Path Generator

#### Store Simply

#### Store By UserName

#### Store By User Name Per Folder

### Data Structure

#### Dict

#### TextDict

### Log

#### Logger

### Exceptions

### Typing

#### Media Type

## Recommend packages

* BeautidfulSoup: beautifulsoup4

* Parsel: parsel

* Brotli: brotli

## How to contribute

### Distribute

Refer to the official doc: https://packaging.python.org/guides/distributing-packages-using-setuptools/

```bash
python setup.py sdist
python setup.py bdist_wheel --universal
twine upload dist/*
```

## Todo

* Fulfill the docs

* Multi-thread / multi-processing / Asycio

* Decorator

* Monitor


