Metadata-Version: 2.1
Name: scrapy-plus
Version: 1.0.4
Summary: scrapy 常用爬网必备工具包
Home-page: http://www.github.com/dotnetage/scrapy_plus
Author: Ray
Author-email: csharp2002@hotmail.com
License: BSD
Keywords: scrapy,crawl,redis,tor
Platform: any
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.6
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Utilities
Description-Content-Type: text/markdown
Requires-Dist: adblockparser (==0.7)
Requires-Dist: aliyun-python-sdk-core-v3 (==2.13.0)
Requires-Dist: aliyun-python-sdk-kms (==2.5.0)
Requires-Dist: asn1crypto (==0.24.0)
Requires-Dist: astroid (==2.1.0)
Requires-Dist: attrs (==18.2.0)
Requires-Dist: Automat (==0.7.0)
Requires-Dist: autopep8 (==1.4.3)
Requires-Dist: certifi (==2018.11.29)
Requires-Dist: cffi (==1.12.1)
Requires-Dist: chardet (==3.0.4)
Requires-Dist: constantly (==15.1.0)
Requires-Dist: crcmod (==1.7)
Requires-Dist: cryptography (==2.5)
Requires-Dist: cssselect (==1.0.3)
Requires-Dist: dateparser (==0.7.1)
Requires-Dist: funcparserlib (==0.3.6)
Requires-Dist: hyperlink (==18.0.0)
Requires-Dist: idna (==2.8)
Requires-Dist: incremental (==17.5.0)
Requires-Dist: isort (==4.3.4)
Requires-Dist: jmespath (==0.9.3)
Requires-Dist: lazy-object-proxy (==1.3.1)
Requires-Dist: lxml (==4.3.1)
Requires-Dist: mccabe (==0.6.1)
Requires-Dist: oss2 (==2.6.1)
Requires-Dist: parsel (==1.5.1)
Requires-Dist: Pillow (==5.4.1)
Requires-Dist: psutil (==5.5.1)
Requires-Dist: psycopg2 (==2.7.7)
Requires-Dist: psycopg2-binary (==2.7.7)
Requires-Dist: pyasn1 (==0.4.5)
Requires-Dist: pyasn1-modules (==0.2.4)
Requires-Dist: pycodestyle (==2.5.0)
Requires-Dist: pycparser (==2.19)
Requires-Dist: pycryptodome (==3.7.3)
Requires-Dist: PyDispatcher (==2.0.5)
Requires-Dist: PyHamcrest (==1.9.0)
Requires-Dist: pylint (==2.2.2)
Requires-Dist: pymongo (==3.7.2)
Requires-Dist: pyOpenSSL (==19.0.0)
Requires-Dist: pyquery (==1.4.0)
Requires-Dist: python-dateutil (==2.8.0)
Requires-Dist: pytz (==2018.9)
Requires-Dist: qt5reactor (==0.5)
Requires-Dist: queuelib (==1.5.0)
Requires-Dist: redis (==3.2.1)
Requires-Dist: regex (==2019.2.21)
Requires-Dist: requests (==2.21.0)
Requires-Dist: Scrapy (==1.6.0)
Requires-Dist: scrapy-splash (==0.7.2)
Requires-Dist: scrapyd (==1.2.0)
Requires-Dist: scrapyd-client (==1.1.0)
Requires-Dist: selenium (==3.141.0)
Requires-Dist: service-identity (==18.1.0)
Requires-Dist: six (==1.12.0)
Requires-Dist: splash (==3.3.1)
Requires-Dist: SQLAlchemy (==1.2.18)
Requires-Dist: stem (==1.7.1)
Requires-Dist: Twisted (==18.9.0)
Requires-Dist: typed-ast (==1.3.1)
Requires-Dist: tzlocal (==1.5.1)
Requires-Dist: urllib3 (==1.24.1)
Requires-Dist: w3lib (==1.20.0)
Requires-Dist: wrapt (==1.11.1)
Requires-Dist: xvfbwrapper (==0.2.9)
Requires-Dist: zope.interface (==4.6.0)

# Scrapy+

Scrapy扩展工具包。为[《从0学爬虫专栏》](https://www.imooc.com/read/34) 提供，详细的使用方法请到专栏内参考。

```
$ pip install scrapy_plus
```

Scrapy+提供以下的内容

- 过滤器
  - Redis 去重过滤器
  - Redis 布隆去重过滤器
- 中间件
  - 自登录中间件
  - 花瓣网专用中间件
  - Chrome通用中间件
  - Splash渲染中间件
  - Tor中间件
  - 随机UA中间件
  - 随机代理中间件
- 管道
  - MongoDB数据存储管道
  - 可支持阿里云的OSS图片管道
- SQL存储端
- 输入/输出处理器
- 蜘蛛
  - `BookSpider`
  - `NeteaseSpider`
  - `TaobaoSpider`

