# Biblicus Architecture (Detailed)

This document is the long-form architecture and policy reference intended for agents and deep
implementation review. For a human-oriented overview, see `ARCHITECTURE.md`.

Biblicus is a toolkit for managing large, unstructured corpora and turning them into usable,
repeatable pipelines. It covers ingestion, extraction, transformation, analysis, retrieval, and
evaluation. Retrieval-augmented generation is one important use case, but the system is designed
for any workflow that needs to reason over aggregates of messy, multi-format data.

## How we design

Design starts from strict behavior-driven development:
- The authoritative description of behavior lives in `features/*.feature`.
- All changes should follow specification-first behavior-driven development: failing scenario,
  implementation, passing scenario, then refactor.
- Behavior-driven development scenarios are not an afterthought: they are how we keep the domain
  vocabulary consistent and the platform comparable across backends and configurations.
- **Specification completeness** is mandatory: if behavior exists, it must be specified.
  Ambiguous or untestable behavior should be removed or turned into an explicit error.

## Executable textbook principle

Documentation is treated as a runnable textbook. Each chapter includes commands and scripts that
can be executed locally, and every concept described in the docs has a working example in
`docs/DEMOS.md` or a bundled lab script.

## Domain-specific cognitive framework

The domain-specific cognitive framework is the set of **stable nouns**, **verbs**, and
**invariants** that make Biblicus pleasant to use over time. We prefer a small set of universal
concepts with strict semantics over a large set of ad-hoc flags.

We also treat **Pydantic models** as the canonical way to codify and validate these constructs at
boundaries.

## The Python developer mental model

If this system is pleasant to use, a Python developer should be able to describe intent with the
core nouns:

- I have a **corpus** at this path or uniform resource identifier.
- I ingest an **item** with optional **metadata**.
- I rebuild the derived **index** after edits.
- I run a **configuration** against the same corpus.
- I query and receive **evidence**.

Anything that does not map cleanly to these nouns is either a derived helper or a backend-specific
implementation detail that should not leak.

## Relationship to agent frameworks

Biblicus is designed to integrate with agent frameworks through explicit tools and clear
application programming interfaces. Tactus is one target environment with strong isolation
requirements.

- **Tools and toolsets**, including the Model Context Protocol, are the primary capability
  boundary.
- **Sandboxing and brokered or secretless execution** are primary deployment modes.
- **Durability and evaluations** are central: invariants via specifications, quality via
  evaluations.

## Core concepts

### Concepts

- **Corpus**: a named, mutable collection rooted at a path or uniform resource identifier. In
  version zero it is typically a local folder containing raw files plus a `.biblicus/` directory
  for minimal metadata.
- **Item**: the unit of ingestion in a corpus: raw bytes of any modality, including text, images,
  Portable Document Format documents, audio, and video, plus optional metadata and provenance.
- **Knowledge base backend**: an implementation that can ingest and retrieve from a corpus, such
  as scan, full text search, vector retrieval, or hybrid retrieval, exposed to procedures through
  retrieval primitives.
- **Retrieval configuration**: a named configuration bundle for a backend, such as chunking rules,
  embedding model and version, hybrid weights, reranker choice, and filters. This is what we
  benchmark and compare.
- **Configuration manifest**: a reproducibility record describing the backend and configuration parameters,
  plus any referenced snapshot artifacts and build snapshots.
- **Snapshot artifacts**: optional, persisted representations derived from raw content for a given
  configuration and backend, such as chunks, embeddings, or indexes. Some backends intentionally have
  none and operate on demand.
- **Evidence**: structured retrieval output from backend queries. Evidence includes spans, scores,
  and provenance used by downstream retrieval augmented generation procedures.
- **Pipeline stage / editorial layer**: a structured step that transforms, filters, extracts, or
  curates content, such as raw, curated, and published, or extract text from Portable Document
  Format documents.

## Design principles

- **Primitives + derived constructs**: keep the protocol surface small and composable; ship
  higher-level helpers and example procedures on top.
- **Composability definition**: composable means each stage has a small input and output contract,
  so you can connect stages in different orders without rewriting them.
- **Minimal opinion raw store**: raw ingestion should work for a folder of files with optional
  lightweight tagging.
- **Reproducibility by default**: comparisons require manifests (even when there are no persisted
  snapshot artifacts).
- **Mutability is real**: corpora are edited, pruned, and reorganized; re-indexing must be a core
  workflow.
- **Separation of concerns**: retrieval returns evidence; retrieval-augmented generation patterns
  live in Tactus procedures (not inside the knowledge base backend).
- **Deployment flexibility**: same interface across local/offline, brokered external services, and
  hybrid environments.
- **Evidence is the primary output**: every retrieval returns structured evidence; everything else
  is a derived helper.

## Locked decisions (version zero)

These are explicit, opinionated policies encoded into the project:

- **Evidence schema strictness**: moderate-to-strong schema. Evidence must include stable
  identifiers, provenance, and retrieval scores; richer fields (spans, stage, configuration and run
  identifiers) are expected.
- **Retrieval stages**: multi-stage is explicit (retrieve, rerank, then filter). Pipelines are
  expressed through evidence metadata rather than hard-coded backends.
- **Corpus versioning**: snapshot or reindex runs are versioned; full directed acyclic graph
  lineage is deferred.
- **Evaluation datasets**: mixed human-labeled and synthetic questions; human-labeled for truth,
  synthetic for scale.
- **Baseline retriever**: hybrid is the strategic target, but the first reference backend is
  deterministic lexical.
- **Context budgeting**: evidence selection is governed by budgets (token, unit, and per-source
  limits), not a fixed count.

## Evidence schema (version zero)

Evidence is the canonical output of retrieval. Required fields:

- `item_id`, `source_uri`, `media_type`
- `score` and `rank`
- `text` (or `content_ref` when non-text)
- `stage` (for example, `scan`, `full-text-search`, `rerank`)
- `configuration_id` / `snapshot_id` (for reproducibility)
- Optional: `span_start`, `span_end`, `hash`

## Evidence lifecycle

Evidence flows through explicit stages and remains inspectable at every step:

1. **Retrieval**: backends return evidence with `stage` labels and scores.
2. **Processing**: optional reranking or filtering updates scores while preserving provenance.
3. **Context shaping**: context packs select and format evidence into model-ready text.
4. **Evaluation**: evaluation datasets compare evidence rankings to expectations.

At each stage, the output remains a structured object, so you can inspect, store, and compare
runs without re-running the entire pipeline.

## Architectural policies version zero

### Integration boundary

- Biblicus can integrate with Tactus as a **Model Context Protocol toolset**, for example with
  tool names such as `knowledge_base_ingest`, `knowledge_base_query`, and
  `knowledge_base_stats`.
- We do **not** add a knowledge base or retrieval augmented generation language primitive in
  version zero. Revisit only if we need semantics that tools cannot express cleanly, such as
  enforceable policy boundaries, runtime managed durability, caching hooks, or guaranteed
  instrumentation.

### Interface packaging

- The knowledge base interface is a **small protocol and reference implementation**, including
  tool schemas and a reference Model Context Protocol server. We do not build a full managed
  service in version zero.

### Corpus identity and layout

- Corpora are identified by a **uniform resource identifier**; simple strings and paths normalize
  to canonical `file://...`.
- The raw corpus is the source of truth and must support:
  - a plain folder of arbitrary files
  - optional Markdown + Yet Another Markup Language front matter for lightweight tagging
  - sidecar metadata for any file type (for example, `file.pdf.biblicus.yml`)
- Raw items are written with **usable file extensions** whenever possible (based on `media_type`)
  so the corpus remains easy to browse and recover with ordinary operating system tools.

### Mutability and editorial workflow

- Corpora are **mutable**. Re-indexing and refresh are primary operations.
- Filtering, pruning, and curation are primary needs; we may model this as a **multi-layer
  editorial pipeline** such as raw, curated, then published.

### Pipeline stages

- Text extraction (Portable Document Format, office documents, or image optical character
  recognition) is a **pipeline stage**, not part of raw ingestion.

### Backend hosting modes

Biblicus ships an in-process backend interface today and keeps the same contract for future
hosting modes:

- **In-process plugin**: current local minimum viable product and deterministic testing.
- **Out-of-process local daemon**: target mode for heavier systems that need warm indexes.
- **Remote service**: target mode for managed deployments and multi-tenant separation.

Backend hosting mode is a primary benchmark dimension (cold start, warm start, latency,
throughput, cost, operational complexity).

### Security / sandbox topology (all supported)

Biblicus must support all three deployment topologies, selected as appropriate per environment
and backend:

- **In-sandbox**: the knowledge base runs inside the Tactus sandbox container (local, offline,
  simplest wiring).
- **Brokered or external**: the knowledge base runs outside the sandbox and is accessed via tools
  (aligns with secretless or brokered execution).
- **Hybrid**: mix modes across environments (for example, local development in-sandbox; production
  external).

The interface stays the same; topology is configuration.

### Query semantics

- `knowledge_base_query` returns **evidence objects** as the low-level, composable building block.
- Biblicus may ship higher-level convenience helpers built on top of evidence (for example, a
  prompt-ready context pack formatter), but those helpers remain derived and swappable.

### Reproducibility

- Biblicus always records a **configuration manifest** for reproducibility.
- When a backend produces persisted snapshot artifacts, Biblicus treats them as **versioned build
  snapshots** identified by `snapshot_id` (rather than overwriting in place by default).
- Manifests exist even for just-in-time backends (snapshot artifacts may be empty).
- Full directed acyclic graph lineage is not included in version zero; revisit only if needed.
- Optional: define **shared snapshot artifact formats** (canonical chunk and embedding stores) so
  multiple backends can reuse intermediates when it makes sense; keep it opt-in.

### Evaluation

- Evaluate **both** knowledge base level behavior and end-to-end procedure behavior using
  **shared datasets**:
  - **Knowledge base level**: retrieval metrics and system properties (for example, recall and
    mean reciprocal rank, latency, index size, and cost).
  - **Procedure-level (Tactus)**: end-to-end success, policy compliance, and quality metrics
    across real inputs.

### Catalog stance

- The corpus catalog is **file-based** (committable, portable, backend-agnostic) so any
  backend/tool can consume it without requiring a database engine.
- Canonical version zero format is a single JavaScript Object Notation file at
  `.biblicus/catalog.json`, written atomically (temporary file and rename) on updates.
- The catalog includes `latest_snapshot_id` and snapshot manifests are stored at
  `.biblicus/snapshots/<snapshot_id>.json`.
- If this becomes a bottleneck at very large scales, we **change the specification** (bump
  `schema_version`) rather than introduce multiple “supported” catalog storage modes.

## Next textbook chapters

The executable textbook continues to expand in these directions:

1. Tool schemas for knowledge base workflows (Model Context Protocol) with runnable examples.
2. Reference backend examples for out-of-process and remote hosting modes.
3. A reference Tactus procedure that demonstrates retrieval-augmented generation with Biblicus
   evidence.
4. A shared evaluation dataset that demonstrates end-to-end procedure metrics.

## Open questions

- **Editorial pipeline model**: do layers live as directory views, metadata flags, or both?
- **Chunking strategy**: semantic vs fixed-size, and how to compare fairly across corpora.
- **Re-ranking tradeoffs**: quality versus cost and latency, and when to use cross-encoders.
- **Context synthesis**: raw snippets vs summary-based packs, and how to evaluate hallucination
  risk.
