Welcome to Grab’s documentation! — Grab 0.6 documentation

# Welcome to Grab’s documentation!

## Useful Links

- Source code: https://github.com/lorien/grab

- Documentation: https://grab.readthedocs.io/en/latest/

- Russian Web Scraping Chat Group: https://t.me/grablab_ru

- English Web Scraping Chat Group: https://t.me/grablab

## What is Grab?

Grab is a python framework for building web scrapers. With Grab you can build web scrapers of various complexity, from simple 5-line scripts to complex asynchronous website crawlers processing millions of web pages. Grab provides an API for performing network requests and for handling the received content e.g. interacting with DOM tree of the HTML document.

There are two main parts in the Grab library:

> 1) The single request/response API that allows you to build network request, perform it and work with the received content. The API is built on top of urllib3 and lxml libraries.

> 2) The Spider API to build asynchronous web crawlers. You write classes that define handlers for each type of network request. Each handler is able to spawn new network requests. Network requests are processed concurrently with a pool of asynchronous web sockets.

# Table of Contents

## Grab User Manual

- Grab Installation
  - Common problems
  - Installation on Linux
  - Installation on Windows
  - Installation on FreeBSD
  - Installation on MacOS
  - Dependencies
- Testing Grab Framework
  - Controlling what parts of Grab to test
  - Testing with Tox
  - Database configuration
  - Travis Testing
  - Test Coverage
- Grab Quickstart
  - Make a request
  - Creating POST requests
  - Response Content
  - Response Status Code
- Request Methods
  - GET Request
  - POST Request
  - PUT Request
  - Other Methods
- Setting up the Grab Request
  - Grab Config Object
  - Grab Instance Cloning
- Grab Settings
  - Network options
  - HTTP Options
  - Proxy Options
  - Response Processing Options
  - Debugging
- Debugging
  - Using the logging module
  - Logging messages about network request
  - Saving the content of requests and responses
- Work with HTTP Headers
  - Custom headers
  - User-Agent header
  - Referer header
  - HTTP Authentication
- Processing the Response Body
  - Partial Response Body Download
  - Response Compression Method
- File Uploading
  - Form Processing
  - Custom File Name
  - Custom Content Type
- Redirect Handling
  - HTTP 301/302 Redirect
  - Meta Refresh Redirect
  - Original and Destination URLs
- Form Processing
- Network Errors Handling
  - Network Errors
  - Network Timeout
- HTML Document Charset
  - Why does charset matter?
  - Charset Detection Algorithm
  - Setting the charset manually
- Cookie Support
  - Custom Cookies
  - Loading/dumping cookies
  - Permanent file to load/store cookies
- Proxy Server Support
  - Basic Usage
  - Proxy List Support
  - Proxy List Source
  - Automatic Proxy Rotation
  - Getting Proxy From Proxy List
- Searching the response body
  - String search
  - Regexp Search
- Work With Network Response
  - Response Object
- Network Transport
  - Urllib3 transport
  - Use your own transport

## Grab::Spider User Manual

Grab::Spider is a framework to build well-structured asynchronous web-site crawlers.

- What is Grab::Spider?
- Task Object
  - Constructor of Task Class
  - Task Object as Data Storage
  - Cloning Task Object
  - Setting Up Initial Tasks
  - Explicit Ways to Add New Task
  - Default Grab Instance
  - Updating Any Grab Instance
- Task Queue
  - Task Priorities
  - Tasks Queue Backends
- Spider Cache
  - Spider Cache Backends
  - Cache Compression
- Spider Error Handling
  - Rules of Network Request Handling
  - Network Errors
  - Handling of Failed Tasks
  - Manual Processing of Failed Tasks
  - Error Statistics
- Explanation

## API Reference

Using the API Reference you can get an overview of what modules, classes, and methods exist, what they do, what they return, and what parameters they accept.

- Module grab.base
- Module grab.error
- Module grab.cookie
- Module grab.spider
- Module grab.document
- Module grab.spider.task

# Indices and tables

- Index

- Module Index

- Search Page