# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

qCrawl is a fast async web crawling & scraping framework for Python that extracts structured data from web pages. It's similar to Scrapy but built on asyncio with a focus on high throughput and extensibility through middleware.

**Key Technologies:**
- **asyncio** for concurrency
- **aiohttp** for HTTP client
- **lxml** for HTML parsing
- **msgspec** for fast request serialization in scheduler queues
- **orjson** for high-performance JSON operations

## Development Commands

**Linting and Type Checking:**
```bash
ruff check .           # Run linter
mypy .                 # Run type checker
```

**Testing:**
```bash
pytest                 # Run all tests
pytest tests/path/to/test_file.py::test_function  # Run single test
pytest --cov=qcrawl --cov-report=term-missing     # Run with coverage
```

**Running Examples:**
```bash
python example_cli.py              # CLI-based spider example
python example_programmable.py     # Programmatic spider example
qcrawl run example_cli.py          # Using qcrawl CLI
```

## Architecture

### Core Components (Request Flow)

The architecture follows a pipeline pattern with middleware:

```
Spider.start_requests()
    ↓
[Spider Middlewares] (start_requests phase)
    ↓
Scheduler (priority queue + fingerprinting for deduplication)
    ↓
[Downloader Middlewares] (process_request)
    ↓
Downloader (aiohttp HTTP fetch)
    ↓
[Downloader Middlewares] (process_response/process_exception)
    ↓
[Spider Middlewares] (input phase)
    ↓
Spider.parse(response) → yields Items/Requests
    ↓
[Spider Middlewares] (output phase)
    ↓
Pipelines (for Items) OR back to Scheduler (for Requests)
```

### Key Classes

**Crawler** (`qcrawl/core/crawler.py`):
- High-level API accepting a Spider and Settings
- Manages component lifecycle (create Downloader, Scheduler, Engine)
- Handles middleware registration and resolution
- Registers signal handlers for stats collection
- Settings precedence: base Settings → spider.custom_settings → CLI args

**CrawlEngine** (`qcrawl/core/engine.py`):
- Core orchestrator for scheduler/downloader/spider
- Executes downloader middleware chains
- Manages worker tasks for concurrent request processing
- Signal emission for observability

**Settings** (`qcrawl/settings.py`):
- Frozen dataclass with UPPERCASE field names
- Immutable with `with_overrides()` method for creating new instances
- Priority system via `Priority` enum (DEFAULT < CONFIG_FILE < ENV < SPIDER < CLI < EXPLICIT)
- All settings keys are UPPERCASE; `get_setting()` performs case-insensitive lookup

**Spider** (`qcrawl/core/spider.py`):
- Base class for spiders
- `start_requests()` yields initial Requests
- `parse(response)` yields Items or more Requests
- Can define `custom_settings` dict to override Settings

**Item** (`qcrawl/core/item.py`):
- Flexible data container with `dict[str, object]` type
- Supports structured data (strings, numbers, lists, nested dicts)
- Has `.data` and `.metadata` properties

### Middleware System

**Two Types:**

1. **DownloaderMiddleware** - wraps HTTP download:
   - `process_request(request, spider)` - before download
   - `process_response(request, response, spider)` - after download
   - `process_exception(request, exception, spider)` - on error
   - All return `MiddlewareResult` (CONTINUE, KEEP, RETRY, DROP)

2. **SpiderMiddleware** - wraps spider processing:
   - `process_start_requests(start_requests, spider)` - filter initial requests
   - `process_spider_input(response, spider)` - before spider.parse()
   - `process_spider_output(response, result, spider)` - filter Items/Requests
   - `process_spider_exception(response, exception, spider)` - handle parse errors

**Middleware Resolution** (`_resolve_downloader_middleware`, `_resolve_spider_middleware`):
1. If class has `from_crawler(crawler)` classmethod → call it
2. If instance → use directly
3. If class → instantiate with `()`
4. If callable → try factory with `(settings)`, `(spider)`, or `()`

**Middleware Registration:**
- Defined in `Settings.DOWNLOADER_MIDDLEWARES` / `SPIDER_MIDDLEWARES` as `{"dotted.path": priority}`
- Lower priority number = executed first
- Middlewares are validated to have async hooks

### Type System Patterns

**Settings Typing:**
- `RuntimeSettings` is TYPE_CHECKING import of `Settings` class
- Use `Settings | dict[str, object] | None` for snapshot parameters
- `_build_final_settings()` always returns `RuntimeSettings` (Settings instance)

**Item Typing:**
- Items use `dict[str, object]` not `dict[str, str]`
- Allows scraped data to contain lists, numbers, nested structures
- Validation happens in pipelines, not at Item level

**Request Meta:**
- `request.meta: dict[str, object]`
- Common meta keys: `"proxy"` (str), `"retry_count"` (int), `"retry_delay"` (float)
- Always check types when accessing meta values

## Configuration Priority

Settings are merged with this precedence (lowest to highest):
1. **DEFAULT** - Library defaults from Settings dataclass
2. **CONFIG_FILE** - From TOML/JSON config file
3. **ENV** - Environment variables (QCRAWL_*)
4. **SPIDER** - Spider class/instance `custom_settings`
5. **CLI** - Command-line --setting arguments
6. **EXPLICIT** - Programmatic runtime overrides

Spider `custom_settings` can have mixed-case keys; they're normalized to UPPERCASE and merged using `Settings.with_overrides(filtered)`.

## Signals System

Uses a global signal dispatcher (`qcrawl.signals.signals_registry`):
- `spider_opened`, `spider_closed`
- `request_scheduled`, `request_reached_downloader`
- `response_received`, `bytes_received`
- `item_scraped`, `request_dropped`

Stats handlers connect to these for metrics collection.

## Important Implementation Details

**msgspec Usage:**
- Used for request serialization in scheduler queues (not for Items)
- Enables fast disk/Redis queue backends

**Queue System:**
- Priority queue with FIFO tiebreak
- Fingerprinting for deduplication using `RequestFingerprinter`
- Supports memory, disk, and Redis backends

**Concurrency Model:**
- Worker tasks spawn from `CrawlEngine.crawl()`
- Respects `CONCURRENCY` (global) and `CONCURRENCY_PER_DOMAIN` settings
- Managed by `ConcurrencyMiddleware` and `DownloadDelayMiddleware`

## Type Checking Notes

**Mypy Configuration:**
- Strict mode enabled with selective error code disabling
- `disable_error_code` includes: no-untyped-def, no-untyped-call, var-annotated, misc, attr-defined
- Always use TYPE_CHECKING imports for circular dependencies

**Common Patterns:**
- Use `# type: ignore[assignment]` for dynamic spider/engine wiring
- Assert non-None before accessing optional references (e.g., `assert self.engine is not None`)
- Settings always has `with_overrides()` method - no need for hasattr checks