# Architecture: Concurrent DuckDB Sessions

## Overview

The Ontario Data MCP server uses DuckDB as a local cache for datasets downloaded from Ontario's Open Data Catalogue. DuckDB acquires an exclusive file lock on `connect()` and holds it until `close()`. To support multiple MCP server processes sharing the same cache file (e.g. Claude Desktop + Claude Code), the `CacheManager` opens a short-lived connection per operation instead of holding one for the server's lifetime.

## System diagram

```
┌─────────────────────────────────────┐  ┌─────────────────────────────────────┐
│        MCP Session A                │  │        MCP Session B                │
│     (e.g. Claude Desktop)           │  │     (e.g. Claude Code)              │
│                                     │  │                                     │
│  ┌───────────┐  ┌────────────────┐  │  │  ┌───────────┐  ┌────────────────┐  │
│  │ FastMCP   │  │  CKANClient    │  │  │  │ FastMCP   │  │  CKANClient    │  │
│  │ Server    │  │  (HTTP)        │  │  │  │ Server    │  │  (HTTP)        │  │
│  └─────┬─────┘  └────────────────┘  │  │  └─────┬─────┘  └────────────────┘  │
│        │                            │  │        │                            │
│  ┌─────┴──────────────────────────┐ │  │  ┌─────┴──────────────────────────┐ │
│  │         CacheManager           │ │  │  │         CacheManager           │ │
│  │                                │ │  │  │                                │ │
│  │  db_path ─────────────────┐    │ │  │  │  db_path ─────────────────┐    │ │
│  │  _extensions: [httpfs,    │    │ │  │  │  _extensions: [httpfs,    │    │ │
│  │    json, spatial]         │    │ │  │  │    json, spatial]         │    │ │
│  │                           │    │ │  │  │                           │    │ │
│  │  _connect() ──► short-    │    │ │  │  │  _connect() ──► short-    │    │ │
│  │    lived conn per op      │    │ │  │  │    lived conn per op      │    │ │
│  │  _with_retry() ──► write  │    │ │  │  │  _with_retry() ──► write  │    │ │
│  │    ops retry on lock      │    │ │  │  │    ops retry on lock      │    │ │
│  └───────────┬────────────────┘   │ │  │  └───────────┬────────────────┘   │ │
│              │                    │ │  │              │                    │ │
└──────────────┼────────────────────┘ │  └──────────────┼────────────────────┘ │
               │                      │                 │                      │
               └──────────┬───────────┘                 │                      │
                          │                             │                      │
                          ▼                             ▼                      │
               ┌──────────────────────────────────────────┐                    │
               │                                          │                    │
               │   ~/.cache/ontario-data/ontario_data.duckdb                   │
               │                                          │                    │
               │   ┌──────────────────┐  ┌─────────────┐  │                    │
               │   │ _cache_metadata  │  │ _dataset_   │  │                    │
               │   │ (resource index) │  │ metadata    │  │                    │
               │   └──────────────────┘  └─────────────┘  │                    │
               │                                          │                    │
               │   ┌──────────────────────────────────┐   │                    │
               │   │  cached data tables              │   │                    │
               │   │  "ds_drinking_water_a1b2c3d4"    │   │                    │
               │   │  "geo_provincial_parks_e5f6g7h8" │   │                    │
               │   │  ...                             │   │                    │
               │   └──────────────────────────────────┘   │                    │
               │                                          │                    │
               │   Extensions: httpfs, json, spatial      │                    │
               └──────────────────────────────────────────┘                    │
```

## Connection lifecycle

```
  Tool call arrives (e.g. query_cached)
        │
        ▼
  CacheManager.query(sql)
        │
        ▼
  _connect()
    ├── duckdb.connect(db_path)    ◄── acquires file lock
    ├── LOAD httpfs
    ├── LOAD json
    ├── LOAD spatial
    └── yield conn
        │
        ▼
  conn.execute(sql)                ◄── work happens here
        │
        ▼
  conn.close()                     ◄── releases file lock
```

For write operations (`store_resource`, `remove_resource`, etc.), the same flow is wrapped in `_with_retry()`:

```
  _with_retry(fn)
        │
        ├──► attempt 1: _connect() → fn(conn) → close
        │       │
        │       └── duckdb.IOException? ──► sleep 100ms
        │
        ├──► attempt 2: _connect() → fn(conn) → close
        │       │
        │       └── duckdb.IOException? ──► sleep 200ms
        │
        └──► attempt 3: _connect() → fn(conn) → close
                │
                └── duckdb.IOException? ──► raise (give up)
```

## CacheManager method classification

| Method | Type | Connection strategy |
|--------|------|-------------------|
| `initialize` | write | `_connect_raw` (no LOAD — extensions not yet known) |
| `store_resource` | write | `_with_retry` (multi-statement, single conn for atomicity) |
| `remove_resource` | write | `_with_retry` (multi-statement) |
| `remove_all` | write | `_with_retry` (multi-statement) |
| `update_expires_at` | write | `_with_retry` |
| `store_dataset_metadata` | write | `_with_retry` |
| `is_cached` | read | `_connect` |
| `get_table_name` | read | `_connect` |
| `list_cached` | read | `_connect` |
| `get_stats` | read | `_connect` |
| `query` | read | `_connect` |
| `query_df` | read | `_connect` |
| `get_dataset_metadata` | read | `_connect` |
| `execute_sql` | read | `_connect` |
| `execute_sql_dict` | read | `_connect` |
| `get_resource_meta` | read | `_connect` |

## Tool → CacheManager dependency map

All tool modules access DuckDB exclusively through `CacheManager` methods. No tool module holds a direct connection reference.

```
  tools/discovery.py ──────► CKANClient only (no cache)
  tools/metadata.py ───────► CKANClient + cache.store_dataset_metadata,
                               cache.get_dataset_metadata
  tools/retrieval.py ──────► cache.is_cached, get_table_name, get_resource_meta,
                               store_resource, store_dataset_metadata,
                               update_expires_at, remove_resource, remove_all,
                               list_cached, get_stats
  tools/querying.py ───────► cache.query, query_df, get_table_name
  tools/quality.py ────────► cache.execute_sql, execute_sql_dict
  tools/geospatial.py ────► cache.store_resource, is_cached, get_table_name,
                               has_spatial_extension, execute_sql_dict
  staleness.py ────────────► cache.get_resource_meta
```

## Extension loading strategy

DuckDB extensions have two phases:

- **`INSTALL`** — downloads the extension binary. Persistent to the database file. Only needs to happen once.
- **`LOAD`** — activates the extension in a connection. Per-connection. Must happen on every new connection.

During `initialize()`, both `INSTALL` and `LOAD` run. The names of successfully loaded extensions are saved to `self._extensions`. The `_connect()` context manager then runs `LOAD` for each extension on every new connection.

`_connect_raw()` (used only by `initialize()`) skips `LOAD` because extensions aren't known yet.

## Concurrency model

```
  Session A                          Session B
  ─────────                          ─────────
  query_cached(sql)
    connect()  ◄─── lock acquired
    execute(sql)
    close()    ◄─── lock released
                                     download_resource(r1)
                                       connect()  ◄─── lock acquired
                                       INSERT INTO ...
                                       close()    ◄─── lock released
  check_data_quality(r1)
    connect()  ◄─── lock acquired
    SELECT count(*)...
    close()    ◄─── lock released
```

Contention only happens if two operations overlap in time. Since each operation holds the lock for milliseconds (DuckDB is embedded, not networked), collisions are rare. When they do occur on write paths, `_with_retry` handles them with linear backoff (100ms, 200ms, 300ms).

## Key design decisions

1. **Connect-per-operation over persistent connection.** DuckDB's ~1-5ms connect overhead is negligible compared to CKAN API network calls (100ms-2s). The simplicity of not managing connection lifetime outweighs the performance cost.

2. **Single connection per public method.** Multi-statement write methods like `store_resource` (DROP + DELETE + CREATE + INSERT) use one connection for the entire operation, ensuring atomicity.

3. **No persistent `self.conn`.** The `CacheManager` stores only `db_path` and extension metadata. There is nothing to `close()` on shutdown.

4. **Retry only on writes.** Read operations fail fast on lock contention. Write operations retry 3 times with linear backoff. This is sufficient because write-write collisions between human-driven MCP sessions are exceedingly rare.

5. **Full encapsulation.** No code outside `cache.py` touches DuckDB connections directly. Tool modules use `execute_sql()`, `execute_sql_dict()`, `get_resource_meta()`, and other typed methods.
