Metadata-Version: 2.1
Name: spiderutil
Version: 0.1.7
Summary: Utilities for spider.
Home-page: https://github.com/Thesharing/spider-utility
Author: Thesharing
Author-email: 
License: MIT
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Topic :: Utilities
Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, <4
Description-Content-Type: text/markdown
Requires-Dist: redis (>=3.3.0)
Requires-Dist: pymongo (>=3.8.0)
Requires-Dist: requests (>=2.20.0)

# Spider Utility

![Version](https://img.shields.io/pypi/v/spiderutil)
![Download](https://img.shields.io/pypi/dm/spiderutil)
![License](https://img.shields.io/pypi/l/spiderutil)
![Status](https://img.shields.io/pypi/status/spiderutil)

Utilities for spider.

## Features

* Database connector
* Local file path generator
* Network session
* Data structure
* Log
* Exception

## Installation

```bash
pip install spiderutil
```

## Documentation

Still working...

## Examples

Still working...

## How to contribute

Open a [new issue]() when you have questions, advices, or you encounter a bug.

## Recommend packages

### Parsers

* [beautifulsoup4](https://www.crummy.com/software/BeautifulSoup/bs4/doc/): parse the webpage like a tree.

```bash
pip install beautifulsoup4
```

* [scrapy/parsel](https://github.com/scrapy/parsel): Parse webpage with XPath or CSS selectors.

```bash
pip install parsel
```

### Encoding and Decoding

* [google/brotli](https://github.com/google/brotli/tree/master/python): decode brotli encoding.

```bash
pip install brotli
```

### Database 

* [SQLAlchemy](https://github.com/sqlalchemy/sqlalchemy/): Process the SQL statement like object

```bash
pip install SQLAlchemy
```

* [PyMySQL](https://github.com/PyMySQL/PyMySQL): Connect to MySQL

```bash
pip install pymysql
```

* [mkleehammer/pyodbc](https://github.com/mkleehammer/pyodbc): Connect to Microsoft SQL Server

```bash
pip install pyodbc
```

In addition to: [level12/sqlalchemy_pyodbc_mssql](https://github.com/level12/sqlalchemy_pyodbc_mssql).


