Metadata-Version: 2.1
Name: gzspidertools
Version: 0.0.7
Summary: 魔改使用工具库
Home-page: https://github.com/georgeJzzz/gzspidertools
Keywords: crawler,scraping,aiohttp,asyncio,scrapy
Author: George
Author-email: persistencehoo@gmail.com
Maintainer: George
Maintainer-email: persistencehoo@gmail.com
Requires-Python: >=3.8.1,<4.0.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Provides-Extra: all
Provides-Extra: database
Requires-Dist: DBUtils (>=3.0.2,<4.0.0)
Requires-Dist: Pillow (>=9.2.0,<10.0.0) ; extra == "all"
Requires-Dist: PyMySQL (>=1.0.2,<2.0.0)
Requires-Dist: SQLAlchemy (>=2.0.23,<3.0.0)
Requires-Dist: Scrapy (==2.11.1)
Requires-Dist: aiohttp (>=3.9.3,<3.10.0)
Requires-Dist: aiomysql (>=0.1.1,<0.2.0)
Requires-Dist: elasticsearch-dsl (>=8.11.0,<9.0.0) ; extra == "all" or extra == "database"
Requires-Dist: itemadapter (>=0.7.0,<0.8.0)
Requires-Dist: kafka-python (==2.0.2)
Requires-Dist: loguru (>=0.7.0,<0.8.0)
Requires-Dist: mmh3 (>=3.0.0,<4.0.0) ; extra == "all"
Requires-Dist: motor (==2.5.1) ; python_version < "3.11"
Requires-Dist: motor (==3.3.0) ; python_version >= "3.11"
Requires-Dist: numpy (>=1.24.3,<1.25.0) ; (python_version < "3.9") and (extra == "all")
Requires-Dist: numpy (>=1.26.0,<1.27.0) ; (python_version >= "3.9" and python_version < "3.13") and (extra == "all")
Requires-Dist: opencv-python (>=4.8.0.74,<4.9.0.0) ; extra == "all"
Requires-Dist: oracledb (>=1.4.2,<2.0.0) ; extra == "all" or extra == "database"
Requires-Dist: oss2 (>=2.18.4,<3.0.0) ; extra == "all"
Requires-Dist: pika (>=1.3.2,<1.4.0)
Requires-Dist: psycopg (>=3.1.13,<4.0.0) ; extra == "all" or extra == "database"
Requires-Dist: psycopg-binary (>=3.1.13,<4.0.0) ; extra == "all" or extra == "database"
Requires-Dist: psycopg-pool (>=3.2.0,<4.0.0) ; extra == "all" or extra == "database"
Requires-Dist: pycryptodome (>=3.15.0,<4.0.0) ; extra == "all"
Requires-Dist: pymongo (>=3.12.3,<4.0.0) ; python_version < "3.11"
Requires-Dist: pymongo (>=4.5.0,<5.0.0) ; python_version >= "3.11"
Requires-Dist: python-hcl2 (>=4.3.0,<5.0.0) ; extra == "all"
Requires-Dist: pyyaml (>=6.0,<6.1) ; extra == "all"
Requires-Dist: requests (>=2.28.1,<3.0.0)
Requires-Dist: retrying (>=1.3.3,<2.0.0)
Project-URL: Repository, https://github.com/georgeJzzz/gzspidertools
Description-Content-Type: text/markdown

# 来自
> https://github.com/shengchenyang/AyugeSpiderTools/blob/master/docs//docs/intro/install.md
> 
> 增加个人使用的模板

## 安装

> `python 3.8+` 可以直接输入以下命令：

```shell
pip install gzspidertools
```

> 可选安装1，安装数据库相关的所有依赖：

```shell
pip install gzspidertools[database]
```

> 可选安装2，通过以下命令安装所有依赖：

```shell
pip install gzspidertools[all]
```

*注：详细的安装介绍请查看[安装指南](https://ayugespidertools.readthedocs.io/en/latest/intro/install.html)。*

## 用法
```shell
# 查看库版本
gzcmd version

# 创建项目
gzcmd startproject <project_name>

# 进入项目根目录
cd <project_name>

# 替换(覆盖)为真实的配置 .conf 文件：
# 这里是为了演示方便，正常情况是直接在 VIT 中的 .conf 文件填上你需要的配置即可
cp /root/mytemp/.conf DemoSpider/VIT/.conf

# 生成爬虫脚本
gzcmd genspider <spider_name> <example.com>

# 生成 scrapy-redis 爬虫脚本
gzcmd genspider <spider_name> <example.com>

# 运行脚本
scrapy crawl <spider_name>
# 注：也可以使用 gzcmd crawl <spider_name>
```


