# BeaverDB Design Document

Version: 1.1
Status: In Progress
Last Updated: September 18, 2025

## 1. Introduction & Vision

`beaver-db` is a local-first, embedded, multi-modal database for Python. Its primary motivation is to provide a simple, single-file solution for modern applications that need to handle complex data types like vectors, documents, and graphs without the overhead of a database server.

The vision for `beaver-db` is to be the go-to "good enough" database for prototypes, desktop utilities, and small-scale applications. It empowers developers to start quickly with a simple API but provides sufficient power and performance so that they do not need to immediately upgrade to a more complex database system as their application's features grow.

## 2. Guiding Principles

These principleembeddings guide all development and design decisions for beaver-db.

* **Local-First & Embedded:** The database will always be a single SQLite file. This is the non-negotiable source of truth. Temporary files for journaling or syncing are acceptable, but the core architecture is serverless. Any feature that requires a client-server model is out of scope.
* **Standard SQLite Compatibility:** The .db file generated by beaver-db must always be a valid SQLite file that can be opened and queried by any standard SQLite tool. This ensures data portability and interoperability.
* **Minimal & Cross-Platform Dependencies:** New dependencies beyond numpy and scipy will only be considered if they provide a monumental performance improvement and are fully cross-platform, with a guarantee of easy installation for all users. The core library should strive to use Python's standard library wherever possible.
* **Simplicity and Pythonic API:** The library must present a simple, intuitive, and "Pythonic" interface. We will always prefer simple function calls with clear parameters over custom Domain Specific Languages (DSLs) or expression parsing. The user should not have to learn a new query language.
* **Developer Experience & Minimal API Surface:** The primary user experience goal is to provide a clean and minimal public API.
  * **User-facing Classes:** The user should only ever need to instantiate BeaverDB and Document directly.
  * **Fluent Interface:** All other functionalities (like list or collection operations) are exposed through methods on the BeaverDB object (e.g., `db.list("my_list")`, `db.collection("my_docs")`). These methods return internal wrapper objects (`ListWrapper`, `CollectionWrapper`) that provide a rich, fluent interface, preventing the need for the user to learn and manage a large number of classes.
  * **API Design:** Public functions will have short, descriptive names and well-documented parameters, adhering to Python's conventions.
* **Synchronousembedding Core with Async Potential:** The core library is built on a synchronous foundation, reflecting the synchronous nature of the underlying `sqlite3` driver. This ensures thread safety and simplicity. An async-compatible API may be introduced in the future, but it will likely be a wrapper around the synchronous core (e.g., using a thread pool).
* **Convention over Configuration:** Features should work well out-of-the-box with sensible defaults. While configuration options can be provided, the user should not be required to tweak many parameters to get good performance and functionality.

## 3. Architecture & Core Components

`beaver-db` is architected as a set of targeted wrappers around a standard SQLite database, with an in-memory component for performance-critical tasks like vector search.

### 3.1. Core Engine (BeaverDB)

* **Connection Management:** The BeaverDB class manages a single connection to the SQLite database file.
* **Concurrency:** It enables `PRAGMA journal_mode=WAL;` (Write-Ahead Logging) by default. This provides a good level of concurrency between one writer and multiple readers, which is a common pattern for the target applications.
* **Schema Management:** All tables created and managed by the library are prefixed with `beaver_` (or `_beaver_` for internal helpers). This avoids conflicts with user-defined tables within the same database file. The schema is evolved by adding new `beaver_*` tables as new features are introduced.

### 3.2. Data Models & Features

### Key-Value Dictionaries (DictWrapper)

* **Implementation**: A single table (`beaver_dicts`) stores key-value pairs partitioned by a `dict_name` (TEXT). The primary key is a composite of `(dict_name, key)`.
* **Design**: The `db.dict("namespace")` method returns a `DictWrapper` that provides a complete and standard Pythonic dictionary-like interface. This includes subscripting (`__getitem__`, `__setitem__`), explicit `get()`/`set()` methods, and iterators (`keys()`, `values()`, `items()`). This feature is ideal for managing structured configurations or namespaced key-value data while adhering to the principle of a Simple and Pythonic API.

#### Lists (ListWrapper)

* **Implementation:** A table (`beaver_lists`) storing list_name, item_order (REAL), and item_value (TEXT).
* **Design:** The item_order is a floating-point number. This allows for O(1) insertions in the middle of the list by calculating the midpoint between two existing order values. This avoids re-indexing all subsequent items. The ListWrapper provides a rich, Pythonic API (`_len_`, `_getitem_`, `push`, `pop`, etc.).

#### Pub/Sub System

* **Implementation:** A log table (`beaver_pubsub_log`) that stores messages with a timestamp, channel name, and payload.
* **Design:** The current SubWrapper is a synchronous iterator that polls the table for new messages since the last seen timestamp. This design is simple and robust. A more performant, event-driven mechanism could be explored in the future, but only if it does not violate the principles of simplicity and minimal dependencies.

#### Collections (CollectionWrapper)

This is the most complex component, supporting documents, vectors, text, and graphs.

* **Document Storage:** Documents are stored in the beaver_collections table. Each document has a collection, item_id, optional item_vector (BLOB), and metadata (TEXT, as JSON). The Document class is a flexible container with no enforced schema.
* **Vector Search (ANN):**
  * **Indexing:** Vector embeddings are stored as raw bytes (BLOBs) in the beaver_collections table.
  * **Search Algorithm:** For performance, an in-memory k-d tree (`scipy.spatial.cKDTree`) is used for Approximate Nearest Neighbor (ANN) search.
  * **Stale Index Handling:** The database maintains a version number for each collection in beaver_collection_versions. This version is incremented on every index or drop operation. The CollectionWrapper caches the version of its in-memory index and automatically triggers a `refresh()` if it detects its version is older than the database version.
  * **Design Trade-offs:** This in-memory approach is extremely fast for datasets that fit in RAM. The library will not have hard-coded scalability limits, but performance testing will establish practical boundaries which will be published in the documentation.
* **Full-Text Search (FTS):**
  * **Implementation:** Uses a virtual table (`beaver_fts_index`) powered by SQLite's FTS5 extension.
  * **Indexing:** When a Document is indexed, its metadata is flattened into key-value pairs (e.g., author.name becomes author_name). All string values are automatically inserted into the FTS index.
  * **Querying:** The `match()` method provides a simple interface for searching. It supports FTS5 syntax (like "OR", "AND") passed directly in the query string. The goal is to provide powerful search out-of-the-box without requiring complex configuration.
* **Graph Engine:**
  * **Implementation:** Relationships are stored as directed edges in the beaver_edges table, linking a source_item_id to a target_item_id with a label.
  * **Design:** The API is purely functional and Pythonic. It does not use a custom graph query language.
    * `connect()`: Creates a directed edge.
    * `neighbors()`: Retrieves immediate (1-hop) neighbors.
    * `walk()`: Uses a recursive Common Table Expression (CTE) in SQL to perform efficient multi-hop graph traversals (BFS). This is powerful yet keeps the implementation clean and contained within SQL.

### 3.3. Reliability and Data Handling

* **Atomicity and Transactions:** Correctness is paramount. Any public method that performs multiple database modifications (e.g., collection.index, which writes to the documents, FTS, and version tables) **must** be wrapped in a single, atomic database transaction. If any step of the operation fails, the entire transaction will be rolled back, ensuring the database is never left in an inconsistent state.
* **Error Handling:** The library will use standard Python exceptions whenever possible (e.g., TypeError, ValueError, sqlite3.Error). Custom exceptions will be avoided to ensure the API feels native and predictable to Python developers.
* **Data Serialization:**
  * The default serialization format is JSON.
  * The library will provide built-in, standardized converters for common Python types not native to JSON, such as `datetime` (to ISO 8601 strings) and `bytes` (to a standard encoding like base64).
  * For custom user-defined objects, it is the user's responsibility to serialize them into a JSON-compatible format (e.g., a string or a dictionary) before storing them.

## 4. Roadmap & Future Development

### 4.1. Immediate Priorities

The primary focus for the near future is on stability and usability.

1. **Comprehensive Unit Testing:** Increase test coverage to ensure all features are robust and reliable.
2. **Elaborate Examples:** Create more examples that demonstrate how to combine features (e.g., a hybrid search that uses FTS, graph traversal, and vector search together).
3. **Performance Benchmarking:** Develop a standardized suite of performance tests to track regressions and identify optimization opportunities. The results will be used to document the practical scalability limits of the library for various use cases.

### 4.2. Long-Term Vision

The library is approaching feature-completeness for its core domain. The long-term vision is to make it a stable, trusted, and well-documented tool for its niche.

* **Stability:** The API will soon be stabilized, with a commitment to backward compatibility as defined in the versioning policy.
* **New Modalities:** In the distant future, other data modalities could be considered if they fit the embedded, single-file philosophy.

### 4.3. Explicitly Out of Scope

To maintain focus and simplicity, the following features will **not** be implemented:

* Client-server architecture.
* Replication or distributed operation.
* Multi-file database formats.
* Any feature that makes the database file incompatible with standard SQLite tools.
* Custom query languages or DSLs.

### 4.4. Versioning and Backward Compatibility

The project will adhere to **Semantic Versioning (Major.Minor.Patch)**.

* **Patch Releases (x.y.Z):** For bug fixes and non-breaking changes. No API or schema changes are permitted.
* **Minor Releases (x.Y.z):** For adding new, backward-compatible features. Schema additions (e.g., new `beaver_*` tables) are allowed, but schema modifications that break older code are not. Code written for version x.z will be able to open and read a database file from version x.y where z > y.
* **Major Releases (X.y.z):** For introducing breaking changes to the API or the database schema. There is no guarantee of backward compatibility between major versions, and a migration path will not be provided automatically.