# BeaverDB Design Document

- **Version**: 1.2
- **Status**: Active
- **Last Updated**: October 4, 2025

## 1. Introduction & Vision

`beaver-db` is a local-first, embedded, multi-modal database for Python. Its primary motivation is to provide a simple, single-file solution for modern applications that need to handle complex data types like vectors, documents, and graphs without the overhead of a database server.

The vision for `beaver-db` is to be the go-to "good enough" database for prototypes, desktop utilities, and small-scale applications. It empowers developers to start quickly with a simple API but provides a seamless path to scale from a local, embedded model to a networked client-server architecture without changing their application code.

## 2. Guiding Principles

These principles guide all development and design decisions for beaver-db.

* **Local-First & Embedded:** The default mode of operation is a single SQLite file. This is the non-negotiable source of truth for local use. While a client-server model is supported via the REST API, the core engine remains fundamentally embedded.
* **Standard SQLite Compatibility:** The .db file generated by beaver-db must always be a valid SQLite file that can be opened and queried by any standard SQLite tool. This ensures data portability and interoperability.
* **Minimal & Optional Dependencies:** The core library has zero external dependencies. Features like vector search, the REST server, and the API client are available as optional extras, allowing users to install only what they need.
* **Simplicity and Pythonic API:** The library must present a simple, intuitive, and "Pythonic" interface. We will always prefer simple method calls with clear parameters over custom Domain Specific Languages (DSLs).
* **Developer Experience & API Parity**: The primary user experience goal is to provide a clean and minimal public API. This includes ensuring that the remote `BeaverClient` is a drop-in replacement for the local `BeaverDB`, maintaining perfect API parity.
* **Synchronous Core with Async Potential:** The core library is built on a synchronous foundation for thread safety and simplicity. An async-compatible API is provided via on-demand wrappers that run blocking calls in a background thread pool.
* **Convention over Configuration:** Features should work well out-of-the-box with sensible defaults.

## 3. Architecture & Core Components

`beaver-db` is architected as a set of targeted wrappers around a standard SQLite database. It supports two primary modes of operation: **Embedded** and **Client-Server**.

### 3.1. Core Engine (Embedded Mode)

* **`BeaverDB` Class**: Manages a thread-safe connection to a local SQLite database file.
* **Concurrency**: Enables `PRAGMA journal_mode=WAL;` (Write-Ahead Logging) by default to provide good concurrency between one writer and multiple readers.
* **Schema Management**: All tables are prefixed with `beaver_` to avoid conflicts with user-defined tables.

### 3.2. Client-Server Mode

* **REST API Server**: An optional feature that exposes the full functionality of a `BeaverDB` instance over a RESTful API, including WebSocket endpoints for real-time features.
* **`BeaverClient`**: A drop-in replacement for the `BeaverDB` class that interacts with the REST API. Instead of a database connection, it manages an HTTP client session, allowing applications to switch from local to remote operation without code changes.

### 3.3. Data Models & Features

#### Key-Value Dictionaries (DictManager)

* **Implementation**: A single table (`beaver_dicts`) stores key-value pairs partitioned by a `dict_name`.
* **Design**: The `db.dict("namespace")` method returns a `DictManager` (or `RemoteDictManager`) that provides a complete Pythonic dictionary-like interface.

#### Lists (ListManager)

* **Implementation:** A table (`beaver_lists`) storing `list_name`, `item_order` (REAL), and `item_value` (TEXT). The use of a floating-point `item_order` allows for O(1) insertions.
* **Design:** The `ListManager` provides a rich, Pythonic API (`__len__`, `__getitem__`, `push`, `pop`, etc.).

#### Pub/Sub System

* **Implementation:** A log table (`beaver_pubsub_log`) stores messages with a timestamp and channel name.
* **Design:** In embedded mode, a background thread polls the table for new messages. In client-server mode, the client subscribes to a WebSocket endpoint for real-time updates.

#### Collections (CollectionManager)

This is the most complex component, supporting documents, vectors, text, and graphs.

* **Document Storage:** Documents are stored in the `beaver_collections` table.
* **Vector Search (ANN):**
    * **Implementation**: Uses a hybrid index system with a large, on-disk `faiss` base index and a small, in-memory delta index for fast writes. All changes are crash-safe, logged in SQLite tables before being applied.
    * **Compaction**: An automatic background process compacts the delta index into the base index to maintain search performance over time.
* **Full-Text Search (FTS):**
    * **Implementation:** Uses a virtual table (`beaver_fts_index`) powered by SQLite's FTS5 extension. String values from indexed documents are automatically added to the FTS index.
* **Graph Engine:**
    * **Implementation:** Relationships are stored as directed edges in the `beaver_edges` table.
    * **Design:** The API is purely functional and Pythonic, using recursive Common Table Expressions (CTEs) for efficient multi-hop graph traversals (`walk()`).

### 3.4. Developer Experience Features

* **Async API**: A `.as_async()` method is available on all synchronous manager objects. It returns a parallel `Async` version of the object where all methods are `async def`. This is achieved by running the blocking database calls in a background thread pool via `asyncio.to_thread`.
* **Type-Safe Models**: All data-handling methods (`db.dict`, `db.list`, etc.) accept an optional `model` argument. When a Pydantic model or the built-in `beaver.Model` is provided, the library handles automatic data validation, serialization, and deserialization, enabling static analysis and autocompletion.

## 4. Roadmap & Future Development

With the design of the REST API client and async wrappers, the library is approaching feature-completeness for its core domain. The primary focus for the near future is on stability and usability.

1.  **Comprehensive Unit Testing:** Increase test coverage for all features, including the new `BeaverClient` and async wrappers.
2.  **Elaborate Examples:** Create more examples that demonstrate how to combine features, particularly in a client-server context.
3.  **Performance Benchmarking:** Develop a standardized suite of performance tests for both embedded and client-server modes to document practical scalability limits.

### Explicitly Out of Scope

To maintain focus and simplicity, the following features will **not** be implemented:

* Replication or distributed operation beyond the single-server model.
* Multi-file database formats.
* Any feature that makes the database file incompatible with standard SQLite tools.
* Custom query languages or DSLs.