# anySQL v0.1.0 — Design Document

**Date:** 2026-03-09
**Status:** Approved
**Approach:** Phased build with validation gates (Approach C)

---

## Project Overview

anySQL is an open-source SQL analytics engine for AI systems. It lets engineers query LLM responses, agent traces, and RAG pipelines with standard SQL — powered by DuckDB in-memory, persisted to SQLite, with zero configuration.

**Tagline:** *"From vibes to queries."*
**Stack:** Python 3.10+, DuckDB, PyArrow, SQLite (stdlib), Pandas
**License:** Apache 2.0

---

## Problem Statement

AI engineers debug with `print()` statements, JSON log files, and pre-built dashboards that show what the tool designer thought you'd want to see. What's missing is raw SQL over normalized AI telemetry data — specifically the cross-layer JOIN that lets you ask whether your RAG pipeline is failing at retrieval or generation.

---

## Architecture & Data Flow

```
User Code
    │
    ├── @anysql.context(feature="x")     ← Python contextvars, sync+async safe
    ├── OpenAI/Claude wrapped client      ← transparent proxy, one-line swap
    ├── AgentTracer (LangChain callback)  ← manual or callback-based
    └── RAGTracer.after_retrieval()       ← auto-detects LangChain/LlamaIndex/dict
              │
              ▼ insert()
    AnySQL engine
    ├── in-memory buffer (dict lists per table)
    ├── SQLite persistence (JSON blobs, cross-session)
    └── DuckDB (Arrow views, SQL at query time)
              │
              ▼ query()
    6 PyArrow tables as DuckDB views:
    llm_responses, eval_results, pipeline_runs,
    agent_tool_calls, agent_trace, rag_chunks
```

**Key design decisions:**
- Schema enforcement at Arrow layer — SQLite stores raw JSON, validation happens on `_refresh_views()` at query time
- Dot-namespace tables (`llm.responses`) map to flat SQL view names (`llm_responses`) to avoid DuckDB schema conflicts
- Contextvars for thread-safe and async-safe tagging — no manual pass-through required

---

## Module Breakdown

| File | Responsibility | Key interfaces |
|------|---------------|----------------|
| `schema.py` | 6 PyArrow schemas + `SCHEMAS` registry | `SCHEMAS`, `TABLE_NAMES` |
| `storage.py` | SQLite persistence (JSON blobs) | `Storage.save()`, `.load()`, `.row_count()` |
| `engine.py` | DuckDB engine + all 5 UC convenience methods | `AnySQL.insert()`, `.query()`, `.model_comparison()` etc. |
| `context.py` | `@context` decorator + `context_scope()` CM + `_flush_run()` | `context()`, `context_scope()`, `get_context()` |
| `adapters/openai.py` | Transparent proxy for OpenAI client | `OpenAIAdapter.wrap(client)` |
| `adapters/claude.py` | Transparent proxy for Anthropic client | `ClaudeAdapter.wrap(client)` |
| `adapters/generic.py` | Generic JSON/dict adapter | `GenericAdapter` |
| `tracers/agent.py` | LangChain callback + manual tracer | `AgentTracer.trace_tool_call()`, `.trace_step()` |
| `tracers/rag.py` | RAG retrieval tracer + eval recording | `RAGTracer.before_retrieval()`, `.after_retrieval()`, `.record_eval()` |
| `__init__.py` | Public API surface | `init()`, `openai()`, `claude()`, `agent_tracer()`, `rag_tracer()` |
| `cli.py` | CLI entry point | `anysql query "..."`, `anysql stats` |

**Build order (dependency-safe):**
`schema.py` → `storage.py` → `engine.py` → `context.py` → adapters → tracers → `__init__.py` → `cli.py`

---

## The 6 Canonical Tables

| Table | Use Cases | Join Keys |
|-------|----------|-----------|
| `llm.responses` | UC1, UC2 | `response_id` |
| `eval.results` | UC1, UC2, UC5 | `response_id`, `run_id`, `query_id` |
| `pipeline.runs` | UC3 | `run_id`, `session_id` |
| `agent.tool_calls` | UC4 | `session_id` |
| `agent.trace` | UC4 | `session_id` |
| `rag.chunks` | UC5 | `query_id` (cross-layer join key to `eval.results`) |

---

## The 5 Use Cases

| UC | Name | Tables | Key Methods |
|----|------|--------|-------------|
| UC1 | Multi-Model Comparison | `llm_responses` JOIN `eval_results` | `model_comparison()`, `model_by_task()` |
| UC2 | Prompt Regression Detection | `eval_results` (windowed LAG) | `prompt_regressions()`, `eval_debt()`, `silent_degradation()` |
| UC3 | Cost Attribution | `pipeline_runs` (via `@context`) | `cost_by_feature()`, `cost_anomalies()` |
| UC4 | Agent Debugging | `agent_tool_calls`, `agent_trace` | `tool_failure_rates()`, `loop_detector()`, `session_diff()`, `human_intervention_points()` |
| UC5 | RAG Forensics | `rag_chunks` JOIN `eval_results` | `rag_failure_modes()`, `chunk_quality_ranking()`, `similarity_calibration()` |

The cross-layer join (UC5) is the killer feature — `query_id` threads RAG retrieval to eval results, enabling retrieval vs generation failure classification.

---

## Test Strategy

| File | Coverage |
|------|----------|
| `test_schema.py` | All 6 schemas instantiate; field types and nullability match spec; `SCHEMAS` registry complete |
| `test_storage.py` | `save()`/`load()` round-trip; `row_count()`; `delete()` with/without WHERE; in-memory mode skips persistence; table name sanitization |
| `test_engine.py` | `insert()` + `query()` for all 6 tables; all 5 UC methods return DataFrames; `count()`; `clear()`; Arrow type coercion; persistence load on init |
| `test_adapters.py` | OpenAI/Claude mock client; cost calculation; token parsing; context propagation; pricing prefix match; unknown model returns `None` |
| `test_tracers.py` | AgentTracer manual + LangChain callbacks; RAGTracer: LangChain tuple, LlamaIndex NodeWithScore, plain dict, custom normalize_fn; cross-join setup |

All tests use `anysql.init(":memory:")` — no disk I/O, fully isolated.

---

## Demo Files

| File | Dataset | Articles | Models |
|------|---------|----------|--------|
| `realtime_openai_demo.py` | BBC News (2004–05) | 12 | `gpt-4o`, `gpt-4o-mini` |
| `realtime_claude_demo.py` | AG News | 15 | `claude-sonnet-4-6`, `claude-haiku-4-5` |
| `realtime_combined_demo.py` | Reuters R8 | 20 | All 4 models head-to-head |

All demos auto-detect missing API keys and fall back to mock mode with realistic synthetic data. Articles are embedded in the script — no download required.

---

## Build Phases

### Phase 1 — Core Layer
Files: `schema.py`, `storage.py`, `engine.py`, `__init__.py` (stub)
Gate: `python -c "import anysql; db = anysql.init(); print(db)"` passes

### Phase 2 — Integration Layer
Files: `context.py`, `adapters/openai.py`, `adapters/claude.py`, `adapters/generic.py`, `tracers/agent.py`, `tracers/rag.py`, `__init__.py` (full), `cli.py`
Gate: Mock inserts via all adapters + tracers, all 5 UC queries return non-empty DataFrames

### Phase 3 — Validation & Distribution
Files: `tests/` (5 test files, comprehensive), `examples/` (3 demo scripts), `pyproject.toml`, `README.md`
Gate: `pytest tests/` passes; `python examples/realtime_combined_demo.py` runs in mock mode

---

## Repository Structure

```
anysql/
├── anysql/
│   ├── __init__.py
│   ├── engine.py
│   ├── schema.py
│   ├── storage.py
│   ├── context.py
│   ├── cli.py
│   ├── adapters/
│   │   ├── __init__.py
│   │   ├── openai.py
│   │   ├── claude.py
│   │   └── generic.py
│   └── tracers/
│       ├── __init__.py
│       ├── agent.py
│       └── rag.py
├── tests/
│   ├── __init__.py
│   ├── test_engine.py
│   ├── test_schema.py
│   ├── test_storage.py
│   ├── test_adapters.py
│   └── test_tracers.py
├── examples/
│   ├── realtime_openai_demo.py
│   ├── realtime_claude_demo.py
│   └── realtime_combined_demo.py
├── docs/
│   ├── ANYSQL_CLAUDE_CODE.md
│   └── QUERIES.md
├── pyproject.toml
└── README.md
```

---

*anySQL v0.1.0 — Apache 2.0*
