# Contributing to extractor

Thank you for your interest in contributing. This guide covers everything you need to get a working development environment and submit a pull request.

---

## Prerequisites

- Python 3.10 or later
- git

---

## Setup

```bash
git clone https://github.com/your-org/extractor.git
cd extractor
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
```

This installs the package in editable mode with all development dependencies (pytest, ruff, mypy, pre-commit, etc.).

---

## Running tests

### Unit tests (fast, no external dependencies)

```bash
pytest tests/unit/ -q
```

### Integration tests (requires sample files)

```bash
pytest tests/integration/ -q
```

Integration tests that exercise PDF or audio parsers will **skip silently** if the `sample_data/` directory is absent — this is expected behaviour. `sample_data/` contains large binary files and is not committed to the repository. Ask a maintainer for access if you need to run the full integration suite locally.

### Regenerating regression snapshots

If you intentionally change parser output, regenerate the golden files:

```bash
pytest tests/integration/ --force-regen
```

Commit the updated snapshots alongside your code change.

---

## Linting and formatting

```bash
ruff check src/ tests/        # lint
ruff format src/ tests/       # format
```

Both must pass with zero errors before a PR can be merged. The CI job runs `ruff check` only (no `--fix`), so fix issues locally before pushing.

---

## Type checking

```bash
mypy src/
```

The project runs mypy in strict mode. All new code must pass without `# type: ignore` suppressions unless there is a documented reason.

---

## Pre-commit hooks

The repository ships a `.pre-commit-config.yaml` that runs ruff and a set of standard file hygiene checks on every commit:

```bash
pip install pre-commit
pre-commit install
```

After installation, hooks run automatically on `git commit`. To run them manually against all files:

```bash
pre-commit run --all-files
```

---

## Adding a new parser

See [`docs/writing-a-parser.md`](docs/writing-a-parser.md) for the step-by-step guide covering:

- Registering a MIME type in `classifier.py`
- Implementing the `BaseParser` interface
- Writing unit and integration tests
- Adding the parser to the format table in `README.md`

---

## Schema version bump protocol

When a change adds, removes, or renames fields in the output schema:

1. Update `SCHEMA_VERSION` in `src/extractor/schema.py`.
2. Update the `schema_version` field in the SPEC header in `docs/SPEC.md`.
3. Add a **Schema** entry under the relevant version heading in `CHANGELOG.md`.

Patch-level schema changes (additive, fully backward-compatible) bump the patch number only. Any removal or rename of an existing field is a major version bump.

---

## PR expectations

Before opening a pull request, verify:

- [ ] All unit tests pass: `pytest tests/unit/ -q`
- [ ] `ruff check src/ tests/` exits with no errors
- [ ] `ruff format --check src/ tests/` exits with no errors
- [ ] `mypy src/` exits with no errors
- [ ] New behaviour is covered by tests
- [ ] `CHANGELOG.md` has an entry under `[Unreleased]`

---

## Architecture overview

The `docs/` directory contains:

- `SPEC.md` — protocol specification (element types, envelope format, schema version contract)
- `writing-a-parser.md` — guide for adding new parsers
- `writing-a-chunker.md` — guide for customising the chunking strategy (if present)

Key source files:

| File | Responsibility |
|------|---------------|
| `src/extractor/pipeline.py` | Normalization, heading stack, element stream |
| `src/extractor/chunker.py` | Section-aware chunker, atomic element protection |
| `src/extractor/schema.py` | Pydantic v2 models, `ELEMENT_TYPES`, `SCHEMA_VERSION` |
| `src/extractor/writer.py` | JSONL + manifest output |
| `src/extractor/cli.py` | Typer CLI (`run` / `view` / `validate` / `info` / `schema`) |
| `src/extractor/classifier.py` | MIME routing to parsers |
| `src/extractor/quarantine.py` | Security checks (XXE, SSRF, zip bomb) |
