# `emb.rs` – Embedding Service Overview

## Purpose
`emb.rs` provides **semantic embedding generation, storage, and advanced search** capabilities for Todozi.
It powers features such as:

- Full‑text semantic search across tasks, ideas, memories, etc.
- Clustering, outlier detection, and drift tracking of embeddings.
- Hybrid keyword‑semantic searches and multi‑query aggregation.
- Model management (loading additional models, comparing embeddings).

All functionality is built on the **Candle** ML framework and the **sentence‑transformers/all‑MiniLM‑L6‑v2** model (384‑dimensional vectors).

---

## Core Types

| Type | Category | Description |
|------|----------|-------------|
| `TodoziEmbeddingConfig` | Config | Holds model name, dimensions, similarity thresholds, cache TTL, clustering flags, etc. |
| `TodoziEmbeddingCache` | Cache entry | Stores a single embedding vector together with its content type, ID, raw text, tags, creation time and TTL. |
| `TodoziContentType` | Enum | Enumerates the kinds of objects that can be embedded (`Task`, `Tag`, `Memory`, `Idea`, `Chunk`, `Feel`, `Train`, `Error`, `Summary`, `Reminder`, `Tdz`). |
| `SimilarityResult` | Struct | Result of a similarity lookup: content ID, type, score, original text, tags and optional metadata. |
| `ClusteringResult` | Struct | Result of a clustering operation: cluster ID, member items, center vector, size and average similarity. |
| `SearchFilters` | Struct | Optional filters for `filtered_semantic_search` (tags, priority, status, assignee, date ranges, progress). |
| `HierarchicalCluster` | Struct | Recursive cluster node used by hierarchical clustering. |
| `LabeledCluster` | Struct | Cluster with an auto‑generated label and confidence score. |
| `DriftReport` / `DriftSnapshot` | Structs | Track embedding drift over time for a piece of content. |
| `SimilarityGraph`, `GraphNode`, `GraphEdge` | Structs | Simple knowledge‑graph representation of similarity relationships. |
| `ModelComparisonResult`, `ModelEmbeddingResult` | Structs | Results when comparing embeddings generated by multiple models. |
| `ValidationReport`, `ValidationIssue` | Structs | Quality‑check report for the embedding store (NaN/Inf/zero‑vector detection, distribution anomalies). |
| `AggregationType` | Enum | Aggregation strategy for multi‑query searches (`Average`, `Max`, `Min`, `Weighted`). |
| `TodoziEmbeddingService` (implemented further down) | Service | Main façade exposing async methods for generation, caching, searching, clustering, backup, etc. |

---

## Key Public Methods (excerpt)

| Method | Signature | Description |
|--------|-----------|-------------|
| `TodoziEmbeddingService::new(config) -> Result<Self>` | async fn(Config) -> Result<Self> | Initialise the service, load the model, and warm‑up caches. |
| `generate_embedding(&self, text: &str) -> Result<Vec<f32>>` | async fn(&self, &str) -> Result<Vec<f32>> | Produce a single 384‑dim vector for `text`. |
| `generate_embeddings_batch(&self, texts: &[String]) -> Result<Vec<Vec<f32>>>` | async fn(&self, &[String]) -> Result<Vec<Vec<f32>>> | Batch generation (10× faster). |
| `semantic_search(&self, query: &str, types: &[TodoziContentType], limit: usize) -> Result<Vec<SimilarityResult>>` | async fn(&self, &str, &[TodoziContentType], usize) -> Result<Vec<SimilarityResult>> | Classic cosine‑similarity search. |
| `hybrid_search(&self, query: &str, keywords: Vec<String>, weight: f32, limit: usize) -> Result<Vec<SimilarityResult>>` | async fn(&self, &str, Vec<String>, f32, usize) -> Result<Vec<SimilarityResult>> | Mixes semantic similarity (weight) with keyword matching. |
| `multi_query_search(&self, queries: Vec<String>, agg: AggregationType, limit: usize) -> Result<Vec<SimilarityResult>>` | async fn(&self, Vec<String>, AggregationType, usize) -> Result<Vec<SimilarityResult>> | Executes several queries and aggregates scores. |
| `filtered_semantic_search(&self, query: &str, filters: SearchFilters, limit: usize) -> Result<Vec<SimilarityResult>>` | async fn(&self, &str, SearchFilters, usize) -> Result<Vec<SimilarityResult>> | Adds tag/priority/status/date filters to semantic search. |
| `hierarchical_clustering(&self, types: Vec<TodoziContentType>, depth: usize) -> Result<Vec<HierarchicalCluster>>` | async fn(&self, Vec<TodoziContentType>, usize) -> Result<Vec<HierarchicalCluster>> | Produces multi‑level clusters. |
| `auto_label_clusters(&self, clusters: Vec<HierarchicalCluster>) -> Result<Vec<LabeledCluster>>` | async fn(&self, Vec<HierarchicalCluster>) -> Result<Vec<LabeledCluster>> | Calls an LLM (via `TodoziEmbeddingService`) to generate human‑readable labels. |
| `track_embedding_drift(&self, id: &str, new_text: &str) -> Result<DriftReport>` | async fn(&self, &str, &str) -> Result<DriftReport> | Compare new embedding against stored baseline. |
| `build_similarity_graph(&self, threshold: f32) -> Result<SimilarityGraph>` | async fn(&self, f32) -> Result<SimilarityGraph> | Constructs a graph where edges exist if similarity ≥ threshold. |
| `backup_embeddings(&self, path: &Path) -> Result<()>` | async fn(&self, &Path) -> Result<()> | Export the entire cache to a timestamped file. |
| `restore_embeddings(&self, path: &Path) -> Result<()>` | async fn(&self, &Path) -> Result<()> | Import a previously saved cache. |
| `validate_embeddings(&self) -> Result<ValidationReport>` | async fn(&self) -> Result<ValidationReport> | Checks for NaN/Inf/zero vectors and reports anomalies. |

---

## Typical Usage Flow (Mermaid)

```mermaid
flowchart TD
    A[Create TodoziEmbeddingService] --> B[Load Model & Warm‑up Cache]
    B --> C[Generate Embedding for New Content]
    C --> D[Store in TodoziEmbeddingCache]
    D --> E{Search Request?}
    E -->|Semantic| F[semantic_search]
    E -->|Hybrid| G[hybrid_search]
    E -->|Filtered| H[filtered_semantic_search]
    F --> I[Return Sorted SimilarityResult list]
    G --> I
    H --> I
    I --> J[Caller formats results (CLI / UI / API)]
```

---

## Integration Points

| Module | Interaction |
|--------|-------------|
| `storage.rs` | Persists `TodoziEmbeddingCache` entries to `~/.todozi/embed/` (HLX format). |
| `search.rs` | High‑level façade that forwards keyword/semantic queries to the embedding service. |
| `cli.rs` | Commands like `todozi emb embed <text>` and `todozi emb search <query>` call the async service methods. |
| `server.rs` | Exposes HTTP‑like endpoints (`/embeddings/search`, `/embeddings/graph`, etc.) that delegate to the service. |
| `tui.rs` (optional) | UI panels for visualising similarity graphs, cluster overviews, and drift reports. |
| `agent.rs` | Agents can request `hybrid_search` or `recommend_similar` to assist in planning or code generation. |

---

## Example Snippet

```rust
use todozi::emb::{TodoziEmbeddingService, TodoziEmbeddingConfig, TodoziContentType};

#[tokio::main]
async fn main() -> todozi::error::Result<()> {
    // Initialise the service with default configuration
    let mut service = TodoziEmbeddingService::new(TodoziEmbeddingConfig::default()).await?;

    // Batch embed a few task descriptions
    let texts = vec![
        "Implement OAuth login flow".to_string(),
        "Write unit tests for the payment module".to_string(),
        "Refactor the configuration loader".to_string(),
    ];
    let embeddings = service.generate_embeddings_batch(&texts).await?;

    // Store them (the service does this automatically via its internal cache)
    for (i, emb) in embeddings.iter().enumerate() {
        service
            .store_embedding(
                &texts[i],
                emb.clone(),
                TodoziContentType::Task,
                format!("task_{}", i),
                vec!["dev".to_string()],
            )
            .await?;
    }

    // Perform a hybrid search
    let results = service
        .hybrid_search(
            "authentication",
            vec!["security".to_string()],
            0.7, // 70 % semantic weight
            5,
        )
        .await?;

    for r in results {
        println!(
            "Found {} (score {:.2}) – {}",
            r.content_id, r.similarity_score, r.text_content
        );
    }

    Ok(())
}
```

---

## Maintenance Notes

- **Model Updates** – To load a newer transformer, call `TodoziEmbeddingService::load_additional_model(name, alias)`.
- **Cache Size** – The service uses an LRU cache (configurable via `cache_ttl_seconds`). Adjust the TTL or replace the cache implementation if memory becomes a concern.
- **Backup Strategy** – Schedule regular `backup_embeddings` calls (e.g., via a cron‑like task) to avoid data loss.
- **Validation** – Run `validate_embeddings` after bulk imports to catch corrupted vectors early.

---

## See Also

- `src/models.rs` – Core data structures (`Task`, `Priority`, `Status`, …).
- `src/search.rs` – Higher‑level search façade used by the CLI and server.
- `src/storage.rs` – Low‑level file I/O and HLX handling for the cache.
- `examples/embedding_enhancements_demo.rs` – Demonstrates all 27 enhanced embedding methods.

---

*Documentation generated for the Todozi embedding subsystem (`emb.rs`).*
