# dot-search Architecture Diagrams

## 1. Architecture Overview

```mermaid
graph TD
    User((User))

    subgraph Engine["SearchEngine(db_url)"]
        IX["index(config | index_id)"]
        S["search(query, index_id)"]
    end

    subgraph "Store Layer"
        MakeStore["make_store(db_url)"]
        PG["PgStore<br/><small>pgvector + ParadeDB<br/>SQLAlchemy async</small>"]
        SQLite["SQLiteStore<br/><small>sqlite-vec (no BM25)<br/>aiosqlite</small>"]
        DSConfigs["ds_configs<br/><small>persisted TableConfig JSON<br/>keyed by index_id</small>"]
    end

    User --> Engine
    Engine --> MakeStore
    MakeStore -->|"postgresql://"| PG
    MakeStore -->|"sqlite://"| SQLite
    PG --- DSConfigs
```

---

## 2. Default Usage

```python
from dot_search import SearchEngine, TableConfig, EmbeddingConfig

engine = SearchEngine(db_url="postgresql+asyncpg://user:pass@localhost/mydb")

# Index — config saved to ds_configs
await engine.index(config=TableConfig(
    table="documents",
    embeddings=[EmbeddingConfig(source_column="body", target_column="ds_body")],
))

# Search
results = await engine.search("neural networks", "documents", limit=5)
```

---

## 3. Power Usage

```python
from dot_search import SearchEngine, TableConfig, EmbeddingConfig, BM25Config, ExactConfig

engine = SearchEngine(db_url="postgresql+asyncpg://user:pass@localhost/mydb")

# --- Index 1: full articles (body + title + BM25) ---
await engine.index(config=TableConfig(
    table="articles",
    pk_column="id",                 # defaults to "id" if omitted
    batch_size=100,                 # defaults to 500 if omitted
    embeddings=[
        EmbeddingConfig(source_column="body", target_column="ds_body"),
        EmbeddingConfig(source_column="title", target_column="ds_title"),
    ],
    bm25=[
        BM25Config(source_column="title"),
        BM25Config(source_column="body"),
    ],
    exact=[ExactConfig(source_column="name")],
))

# --- Index 2: title-only, separate index_id on the same table ---
# Use case: a lightweight title-only index for autocomplete,
# separate from the full hybrid index above.
await engine.index(config=TableConfig(
    table="articles",
    index_id="article_titles",
    embeddings=[
        EmbeddingConfig(source_column="title", target_column="ds_article_titles_title"),
    ],
))

# --- Re-index when new rows arrive ---
await engine.index(index_id="articles")        # reloads saved config, re-embeds all rows
await engine.index(index_id="article_titles")

# --- Search: hybrid (vector + BM25), SQL filter, per-query weight overrides ---
results = await engine.search(
    "fermentation and gut health",
    "articles",
    limit=10,
    where="published_year >= 2022 AND topic = 'health'",
    embedding_weights={
        "ds_body": 1.0,
        "ds_title": 0.3,
    },
    bm25_weights={"title": 0.5, "body": 2.0},
)

# --- Search: BM25-only ---
results = await engine.search("fermentation", "articles", strategy="bm25", limit=5)

# --- Search: exact-only ---
results = await engine.search("Dupont", "articles", strategy="exact", limit=5)

# --- Search: title-only index ---
results = await engine.search("gut health", "article_titles", limit=5)
```

---

## 4. index() Flow

`index(*, index_id=None, config=None)` — exactly one must be provided.

```mermaid
sequenceDiagram
    participant User
    participant Engine
    participant Embedder
    participant Store

    User->>Engine: index(config | index_id)
    Engine->>Store: ensure_config_table()
    Note over Engine: resolve config<br/>(see 4a / 4b)

    Note over Engine: embedder = _get_embedder()<br/>(env vars or injected _embedder)

    loop each EmbeddingConfig in config
        Engine->>Store: read_column(table, pk_col, source_col)
        Store-->>Engine: [(id, text), ...]
        Engine->>Store: add_embedding_column(table, target_col, embedder.dimension)

        loop batches of config.batch_size
            Engine->>Embedder: fn(texts)
            Embedder-->>Engine: vectors
            Note over Engine: first batch only: assert len(vec) == embedder.dimension
            Engine->>Store: write_embeddings(table, target_col, pk_col, [(id, vec), ...])
        end
    end

    opt config.bm25 is non-empty
        Engine->>Store: add_bm25_index(table, [bm25_config.source_column, ...])
    end

    Engine-->>User: ok
```

### 4a. index(config=TableConfig(...))

New index or full reconfiguration — caller provides the complete config.

```mermaid
sequenceDiagram
    participant User
    participant Engine
    participant Store

    User->>Engine: index(config=TableConfig(...))
    Engine->>Store: save_config(index_id, json)
    Note over Engine: config ready → proceed to embed
```

### 4b. index(index_id="articles")

Re-index an existing index — reloads a previously saved config.

```mermaid
sequenceDiagram
    participant User
    participant Engine
    participant Store

    User->>Engine: index(index_id="articles")
    Engine->>Store: load_config("articles")
    Store-->>Engine: config JSON
    Note over Engine: deserializes TableConfig<br/>config ready → proceed to embed
```

---

## 5. Search Flow

`search(query, index_id, limit=10, where=None, strategy="hybrid", embedding_weights=None, bm25_weights=None)`

```mermaid
sequenceDiagram
    participant User
    participant Engine
    participant Embedder
    participant Store
    participant RRF

    User->>Engine: search("headphones", "products", limit=5)

    Engine->>Store: ensure_config_table()
    Engine->>Store: load_config("products")
    Store-->>Engine: config JSON
    Note over Engine: deserializes TableConfig

    Note over Engine: embedder = _get_embedder()

    alt strategy="bm25" or "exact"
        Note over Engine: skip embedding, go straight to BM25/exact
    else strategy="vector" or "hybrid"
        loop each EmbeddingConfig in config
            Engine->>Embedder: fn([query])
            Embedder-->>Engine: query_vector
            Note over Engine: resolve weight from embedding_weights or 1.0<br/>build SubSearch(target_col, vector, weight)
        end
    end

    Note over Engine: build BM25Search per BM25Config<br/>resolve weight from bm25_weights or 1.0

    Engine->>Store: search(table, pk_col, subsearches, bm25_searches, where, limit)
    Note over Store: runs each subsearch independently:<br/>vector queries via cosine distance<br/>BM25 queries via full-text index
    Store-->>Engine: [ranked_list_1, ranked_list_2, ...]

    Engine->>RRF: fuse(ranked_lists, weights)
    Note over RRF: score(id) = Σ weight / (k + rank)
    RRF-->>Engine: merged + sorted results

    Engine-->>User: list[SearchResult][:limit]
```

---

## 6. Reciprocal Rank Fusion

Formula: `score(id) = Σ weight / (k + rank)` where k=60 (constant) and rank is 1-based position in each list.

Each ranked list comes from an independent subsearch. There is one list per `EmbeddingConfig` in `config.embeddings`, plus one per `BM25Config` in `config.bm25`.

```mermaid
graph TD
    subgraph Origin["Origin (from TableConfig)"]
        ET1["EmbeddingConfig<br/>source_column=name<br/>target_column=ds_name"]
        ET2["EmbeddingConfig<br/>source_column=description<br/>target_column=ds_description"]
        BM25C["BM25Config<br/>source_column=description"]
    end

    subgraph Input["Ranked lists (returned by Store)"]
        V1["Vector: ds_name<br/>#1 Headphones<br/>#2 Speaker<br/>#3 Keyboard"]
        V2["Vector: ds_description<br/>#1 Speaker<br/>#2 Headphones<br/>#3 Mouse"]
        BM25["BM25: description<br/>#1 Keyboard<br/>#2 Speaker<br/>#3 Headphones"]
    end

    RRF["RRF Scoring<br/>score(id) = Σ weight/(k+rank)"]

    subgraph Output["Final Ranking"]
        R1["#1 Speaker: 0.0472"]
        R2["#2 Headphones: 0.0470"]
        R3["#3 Keyboard: 0.0323"]
        R4["#4 Mouse: 0.0159"]
    end

    ET1 -->|"produces + weight=1.0"| V1
    ET2 -->|"produces + weight=1.0"| V2
    BM25C -->|"produces + weight=1.0"| BM25

    V1 --> RRF
    V2 --> RRF
    BM25 --> RRF
    RRF --> R1
    RRF --> R2
    RRF --> R3
    RRF --> R4
```

---

## 7. Data Models

**User-facing**

`TableConfig`, `EmbeddingConfig`, `BM25Config` — see [section 3](#3-power-usage).

**`SearchResult`** — returned by `search()`

| field | description |
|---|---|
| `id` | primary key value (type depends on the PK column — int, UUID, …) |
| `score` | final RRF score after fusing all ranked lists |
| `row` | all non-`ds_*` columns as a dict — ready to display, no extra query needed |

---

**Internal**

`SubSearch` and `BM25Search` are built by the engine and passed to the store. Not part of the public API.

**`SubSearch`** — one per `EmbeddingConfig`, passed to `store.search()`

| field | default | description |
|---|---|---|
| `embedding_column` | | the `ds_*` column to run ANN search on |
| `query_embedding` | | the embedded query vector |
| `weight` | `1.0` | resolved from `embedding_weights` or defaults to `1.0` |

**`BM25Search`** — one per `BM25Config`, passed to `store.search()`

| field | default | description |
|---|---|---|
| `column` | | the column to run BM25 search on |
| `query` | | the raw query string |
| `weight` | `1.0` | resolved from `bm25_weights` or defaults to `1.0` |

**`ds_configs`** — managed by dot-search in the target database

| column | type | description |
|---|---|---|
| `table_name` | `TEXT PRIMARY KEY` | the `index_id` this config is stored under |
| `config_json` | `TEXT NOT NULL` | JSON-serialized `TableConfig` |
