# CLAUDE.md

## Project overview
Winnow is an open-source data curation toolkit for ML training datasets. It provides a CLI, Python SDK, and (planned) web UI for filtering, deduplicating, scoring, and exploring data.

## Architecture
```
src/winnow/
  __init__.py          -- package root, version re-export
  _version.py          -- auto-generated by setuptools-scm
  cli.py               -- Typer-based CLI (7 commands: version, curate, stats, embed, search, outliers, explore)

  core/
    filters.py         -- Base Filter ABC + 12 built-in heuristic filters
    pipeline.py         -- Pipeline orchestration (YAML config, per-filter stats, streaming execution)
    registry.py         -- Filter registry for YAML-based pipeline construction + custom filter registration

  io/
    readers.py          -- auto_read() + readers for JSONL, Parquet, CSV, HuggingFace datasets
    writers.py          -- write_jsonl(), write_parquet()

  quality/
    dedup.py            -- ExactDedup (SHA-256), MinHashDedup (datasketch LSH)

  embeddings/
    compute.py          -- EmbeddingModel wrapper (sentence-transformers), compute_embeddings()
    search.py           -- EmbeddingIndex (FAISS), AnomalyScorer (k-NN outlier detection)
    dedup.py            -- Semantic dedup: find_semantic_duplicates(), deduplicate_by_embeddings()

  explore/
    __init__.py         -- Placeholder for web UI (FastAPI + frontend, not yet implemented)

examples/
  basic_pipeline.yaml      -- Minimal YAML config example
  advanced_pipeline.yaml   -- Comprehensive YAML config with all filters
  quickstart.py            -- Runnable script: load, filter, dedup, embed, search, outliers

tests/                     -- pytest test suite
```

## All available filters (in registry)
1. `LengthFilter` -- min/max character length
2. `WordCountFilter` -- min/max word count
3. `LineCountFilter` -- min/max line count
4. `WhitespaceFilter` -- max whitespace ratio
5. `RepetitionFilter` -- max n-gram repetition ratio
6. `SpecialCharFilter` -- max special character ratio
7. `AlphaFilter` -- min alphabetic character ratio
8. `URLFilter` -- max URL character ratio
9. `StopwordFilter` -- min stopword ratio (catches keyword spam)
10. `LanguageFilter` -- language detection (lingua or fastText backend) [requires `[nlp]`]
11. `FieldExistsFilter` -- require a non-empty field
12. `RegexFilter` -- include/exclude by regex pattern
13. `ExactDedup` -- SHA-256 exact dedup [requires `[dedup]` only for MinHash]
14. `MinHashDedup` -- MinHash-LSH fuzzy dedup [requires `[dedup]`]

## CLI commands
```
winnow version       -- print version
winnow curate        -- run curation pipeline (flags or --config YAML)
winnow stats         -- show dataset statistics (doc count, lengths)
winnow embed         -- compute embeddings (sentence-transformers), save as .npy
winnow search        -- semantic search over a dataset (FAISS)
winnow outliers      -- find anomalous documents via k-NN distance
winnow explore       -- launch web UI (not yet implemented)
```

## Development
```bash
pip install -e ".[dev]"
pre-commit install
pytest
ruff check src/ tests/
ruff format src/ tests/
```

## Key conventions
- Python 3.10+, type hints on all public APIs
- src/ layout, single package `winnow`
- Ruff for linting and formatting (line-length 119)
- All filters inherit from `winnow.core.filters.Filter` (ABC with `filter()` and `name()`)
- Optional dependencies via extras: `[io]`, `[nlp]`, `[embeddings]`, `[dedup]`, `[all]`
- Lazy imports for optional dependencies with clear ImportError messages
- Filters are registered in `winnow.core.registry.FILTER_REGISTRY` for YAML config support
- Custom filters can be registered via `register_filter(name, cls)`

## Testing
- `pytest` with markers: `@pytest.mark.slow`, `@pytest.mark.gpu`
- Run fast tests: `pytest -m "not slow"`
- Tests live in `tests/` mirroring `src/winnow/` structure

## Dependencies
- Core: pyarrow, pydantic, rich, typer, fsspec, tqdm, pyyaml
- `[io]`: datasets, pandas
- `[nlp]`: lingua-language-detector, tokenizers
- `[embeddings]`: sentence-transformers, faiss-cpu
- `[dedup]`: datasketch
