SCRAPY(1)

NAME
scrapy - the Scrapy command-line tool

SYNOPSIS
scrapy [command] [OPTIONS] ...

DESCRIPTION

Scrapy is controlled through the scrapy command-line tool. The script provides several commands, for different purposes. Each command supports its own particular syntax. In other words, each command supports a different set of arguments and options.

OPTIONS

  fetch [OPTION] URL

Fetch a URL using the Scrapy downloader

--headers
    Print response HTTP headers instead of body

  runspider [OPTION] spiderfile
Run a spider

--output=FILE
    Store scraped items to FILE in XML format

  settings [OPTION]
Query Scrapy settings

--get=SETTING
    Print raw setting value

--getbool=SETTING
    Print setting value, intepreted as a boolean

--getint=SETTING
    Print setting value, intepreted as an integer

--getfloat=SETTING
    Print setting value, intepreted as an float

--getlist=SETTING
    Print setting value, intepreted as an float

--init
    Print initial setting value (before loading extensions and spiders)

  shell URL | file
Launch the interactive scraping console

  startproject projectname
Create new project with an initial project template

  --help, -h
Print command help and options

  --logfile=FILE
Log file. if omitted stderr will be used

  --loglevel=LEVEL, -L LEVEL
Log level (default: None)

  --nolog
Disable logging completely

  --spider=SPIDER
Always use this spider when arguments are urls

  --profile=FILE
Write python cProfile stats to FILE

  --lsprof=FILE
Write lsprof profiling stats to FILE

  --pidfile=FILE
Write process ID to FILE

  --set=NAME=VALUE, -s NAME=VALUE
Set/override setting (may be repeated)

AUTHOR
Scrapy was written by the Scrapy Developers <scrapy-developers@googlegroups.com>.

This manual page was written by Ignace Mouzannar <mouzannar@gmail.com>, for the Debian project (but may be used by others).