# Plan: Support Concurrent MCP Sessions

## Problem

DuckDB uses an exclusive file lock — only one process can connect to a given `.duckdb` file at a time. All MCP server instances default to `~/.cache/ontario-data/ontario_data.duckdb`, so launching a second session (e.g. Claude Desktop + Claude Code, or two Claude Code projects) fails with:

```
IO Error: Could not set lock on file "...ontario_data.duckdb":
Conflicting lock is held in ... (PID XXXX)
```

This is silent and confusing — the MCP server just fails to connect with no user-facing explanation.

## Constraints

- Both sessions may need to write (download_resource, cache_manage, refresh_cache, load_geodata, get_dataset_info all write)
- READ_ONLY mode is not viable since any session might cache data
- Ideally cached data is shared across sessions (downloading the same resource twice wastes time and bandwidth)

## Options

### A. Per-session database file

Give each server process its own DuckDB file (e.g. `ontario_data_{uuid}.duckdb` or `ontario_data_{pid}.duckdb`).

- **Pro:** Zero lock contention, trivially simple
- **Con:** No shared cache — each session re-downloads everything. Orphan files accumulate.

### B. Graceful fallback with clear error

Catch the lock error at startup. If the primary DB is locked, create a temporary per-session DB and warn the user that cache won't be shared.

- **Pro:** Primary session gets full cache, secondary sessions still work
- **Con:** Still no shared cache for secondary sessions. Two sessions downloading the same resource don't benefit from each other.

### C. Retry with timeout

Wrap `duckdb.connect()` in a retry loop with backoff.

- **Pro:** Preserves shared cache if lock contention is transient
- **Con:** DuckDB holds the lock for the lifetime of the connection, not per-query. Retry only helps if the first process disconnects entirely. Won't solve concurrent sessions.

### D. Connect/disconnect per operation

Instead of holding a persistent `duckdb.connect()` for the server's lifetime, open and close the connection around each cache operation.

- **Pro:** Shared cache, minimal lock window. Two sessions can coexist as long as they don't write at the exact same moment.
- **Con:** Higher latency per operation (connect overhead). Rare write-write collisions still possible — need retry logic around writes. Reads are fast but still need a connection.

### E. WAL mode + connect-per-write

DuckDB supports concurrent readers with one writer in WAL mode. Keep a persistent read-only connection for queries, and open a short-lived write connection only when caching data.

- **Pro:** Shared cache. Reads never block. Write collisions are rare and can be retried.
- **Con:** More complex connection management. Need to handle the case where a write connection can't be acquired (retry + backoff).

## Recommendation

**Option D (connect/disconnect per operation)** is the best balance of simplicity and correctness. The connect overhead for DuckDB is ~1-5ms (it's an embedded database, not a network round-trip), so the latency cost is negligible compared to network calls to the CKAN API.

## Implementation

### Phase 1: Encapsulate all direct `cache.conn` access

**Prerequisite.** Today, 4 files bypass `CacheManager` and reach into `cache.conn` directly. This must be fixed before changing the connection model — otherwise Phase 2 breaks them immediately.

Call sites to migrate:

| File | Lines | Usage | New CacheManager method |
|------|-------|-------|------------------------|
| `tools/quality.py` | 27, 30, 38, 45, 53, 65 | `check_data_quality` — row count, DESCRIBE, null counts, distinct counts, numeric stats, duplicate detection | `execute_sql(sql, params?) -> list[tuple]` |
| `tools/quality.py` | 149, 155 | `profile_data` — SUMMARIZE, COUNT | same `execute_sql` |
| `tools/retrieval.py` | 77 | `download_resource` — read cache metadata | `get_resource_meta(resource_id) -> tuple \| None` |
| `tools/geospatial.py` | 192 | `spatial_query` — raw spatial SQL | same `execute_sql` |
| `staleness.py` | 40 | `get_staleness_info` — read downloaded_at/expires_at | `get_resource_meta(resource_id)` (reuse) |

Add two methods to `CacheManager`:

```python
def execute_sql(self, sql: str, params=None) -> list[tuple]:
    """Execute read-only SQL and return raw tuples."""
    result = self.conn.execute(sql, params or [])
    return result.fetchall()

def get_resource_meta(self, resource_id: str) -> dict | None:
    """Get full cache metadata for a resource."""
    row = self.conn.execute(
        "SELECT * FROM _cache_metadata WHERE resource_id = ?",
        [resource_id],
    ).fetchone()
    if row is None:
        return None
    cols = ["resource_id", "dataset_id", "table_name", "downloaded_at",
            "row_count", "size_bytes", "source_url", "expires_at"]
    return dict(zip(cols, row))
```

Also add `describe_table(table_name)` and `row_count(table_name)` if it keeps call sites cleaner — judge during implementation.

After this phase: **zero references to `cache.conn` outside `cache.py`** (excluding tests, which use their own `CacheManager` instances).

### Phase 2: Connect/disconnect per operation

1. **Remove persistent `self.conn`** from `__init__`. Store only `self.db_path`.

2. **Add `_connect()` context manager:**

```python
from contextlib import contextmanager

@contextmanager
def _connect(self):
    conn = duckdb.connect(self.db_path)
    try:
        for ext in self._extensions:
            conn.execute(f"LOAD {ext}")
        yield conn
    finally:
        conn.close()
```

3. **Wrap every `CacheManager` method** in `with self._connect() as conn:`. Connection granularity is per **public method**, not per SQL statement. Multi-statement methods like `store_resource()` use a single connection for atomicity.

4. **Handle extension loading.** `INSTALL` is persistent (saved to the database file). `LOAD` is per-connection. During `initialize()`, install extensions and record which ones succeeded in `self._extensions: list[str]`. The `_connect()` context manager then LOADs them on each connection.

```python
def initialize(self):
    self._extensions = []
    with self._connect_raw() as conn:  # no LOAD yet
        # Create tables...
        for ext in ["httpfs", "json", "spatial"]:
            try:
                conn.execute(f"INSTALL {ext}")
                conn.execute(f"LOAD {ext}")
                self._extensions.append(ext)
            except Exception:
                if ext == "spatial":
                    logger.info("spatial extension not available")
    self._has_spatial = "spatial" in self._extensions
```

Use `_connect_raw()` (no extension loading) for `initialize()`, and `_connect()` (with loading) everywhere else.

5. **Add retry logic for writes** (~8 lines, no libraries):

```python
import time

def _with_write_retry(self, fn, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            with self._connect() as conn:
                return fn(conn)
        except duckdb.IOException:
            if attempt == max_attempts - 1:
                raise
            time.sleep(0.1 * (attempt + 1))
```

Apply to write-path methods: `store_resource`, `remove_resource`, `remove_all`, `update_expires_at`, `store_dataset_metadata`.

6. **Delete `close()`** entirely. Remove `cache.close()` from `server.py` lifespan. No persistent connection = nothing to close.

### Phase 3: Update tests

Tests in `test_cache.py`, `test_staleness.py`, `test_utils.py` access `cache.conn` directly. Update to use the new methods or the context manager. Tests use their own `CacheManager` instances so concurrency isn't a concern — just API compatibility.

## Edge cases addressed

- **Concurrent `initialize()`**: Two processes starting simultaneously both run `CREATE TABLE IF NOT EXISTS` and `INSTALL`. Both are idempotent — safe with retry.
- **Concurrent `store_resource` for same resource**: Both pass `is_cached()` check, both download, one wins the write lock. Second retries and overwrites — functionally correct, just wasteful. Acceptable for a local tool.
- **`_has_spatial` accuracy**: Set during `initialize()` based on which extensions installed successfully. Stable for the process lifetime.
- **asyncio concurrency within a single process**: FastMCP may dispatch multiple tool calls concurrently. Within the same process, `duckdb.connect()` to the same file shares state — two connections from the same process do not contend. This is safe.

## Not in scope

- Multi-user shared cache (different OS users) — each user gets their own cache dir
- Distributed caching — this is a local tool, not a service
- Option E (WAL mode) — viable future refinement if read overhead matters, but Option D is simpler and sufficient given that all operations are dominated by network latency
