# PR-FAQ (Draft): Embedding Retrieval and Chunking

## Press release

Today we are adding true embedding-based retrieval to Biblicus, including a pluggable chunking surface and a
textbook-simple local index.

Biblicus already supports deterministic lexical retrieval and hybrid retrieval wiring. This feature adds retrieval
backends that build a reusable embedding index under the corpus, so retrieval is fast, repeatable, and evaluatable.
Chunking is treated as part of the indexing contract (not an afterthought): embeddings are computed over chunks, and
retrieval returns evidence with item-level provenance plus chunk boundaries.

## FAQ

### What problem does this solve?

- Provide a real embedding retrieval backend with an explicit build/query lifecycle.
- Make chunking a first-class, fully configurable pipeline stage for embedding retrieval.
- Establish a stable surface for swapping embedding providers without rewriting backends.

### What will users be able to do?

- Build a retrieval run that materializes an embedding index under a corpus.
- Query that run and receive evidence-first outputs with stable identifiers and provenance.
- Choose a chunking strategy (and its configuration) per recipe, and compare results across recipes.

### Why do we need chunking now?

Whole-document embeddings are misleading for most real corpora. Chunking is the practical unit of retrieval and must
be part of the indexing contract so that:

- embeddings are computed over well-defined text spans,
- evidence can cite exact spans,
- hybrid retrieval and evaluation can compare like-for-like.

### How is chunking configured?

Chunking is a pluggable interface selected by identifier in the retrieval recipe:

- `chunker_id`
- `chunker_config` (Pydantic validated; `extra="forbid"`)

No fallbacks: if a recipe selects a chunker but its implementation or required configuration is missing, Biblicus
fails fast with a user-facing error that explains exactly what to install/configure.

### What chunkers are provided in the initial implementation?

We provide multiple built-in chunkers from day 1, each with its own configuration model:

- Fixed character window chunking (size + overlap).
- Paragraph chunking (blank-line delimited, with configurable joining/splitting behavior).
- Fixed token window chunking, via a tokenizer interface (see below).

### How is token-based chunking handled without locking in one tokenizer library?

Token-based chunking uses a separate pluggable interface (for example `Tokenizer` or `TokenCounter`), selected by id.

This keeps the chunker configurable while avoiding hidden dependencies. If a token-based chunker is selected without a
configured tokenizer implementation, Biblicus fails with explicit guidance.

### What embedding index backends are included?

Two concrete backends are introduced to avoid external vector stores while still being “real” retrievers:

1) In-memory exact index
- Intended for small corpora and demos.
- Enforces a hard safety cap (for example maximum vectors or bytes).

2) File-backed exact index (NumPy-backed)
- Writes an embedding matrix and id mapping as run artifacts under the corpus.
- Queries by memory-mapping and scanning in batches so memory usage is bounded.

Both backends use exact cosine similarity. This is intentionally “textbook” behavior that is easy to validate and
compare. Approximate nearest neighbor indexes are explicitly out of scope for this slice.

### How do embeddings get generated?

Embeddings are generated by a pluggable `EmbeddingProvider` interface (an abstract base class), selected by Pydantic
configuration in the recipe. Concrete implementations can wrap OpenAI, Bedrock, or other providers via Biblicus’s AI
provider wiring.

### How does this fit the “derived artifacts live under the corpus” rule?

Embedding retrieval is implemented as a run + artifacts:

- `build_run(...)` materializes the embedding index and records artifact paths in a run manifest.
- `query(...)` loads artifacts from the run and returns evidence with stable chunk provenance.

### How do we test this in CI?

- Behavior specs cover all success and failure behaviors and enforce 100% coverage.
- CI fetches and caches WikiText-2 raw (`wikitext-2-raw-v1`) to provide “real-ish” text at scale without committing the
  dataset into the repository.

### Dependencies

- `numpy` is required for the file-backed index backend (pip-only, no system services).
- `datasets` is used only for CI fixture fetching and is installed in CI.

## Non-goals (this slice)

- External vector stores (Qdrant, pgvector, Pinecone, etc.).
- ANN indexing (FAISS/HNSW tuning).
- Backwards-compatible aliasing for renamed backend ids.

