# Architecture Alternatives & Design Critique

**Date:** 2026-03-19
**Context:** Analysis of the current ontario-data-mcp design and potential improvements.

---

## Current Architecture Summary

- **FastMCP server** with 23 tools across 7 categories
- **Two portal clients** (CKANClient, ArcGISHubClient) behind a thin `PortalClient` protocol
- **DuckDB** as a single file serving both metadata tables and analytics data tables
- **Fan-out pattern** for discovery: all portals queried in parallel, results merged
- **Two-step workflow** for analysis: `download_resource` → `query_cached`

---

## Issue 1: The Remote/Local Query Split Creates a Confusing 3-Tool API

**Problem:** Three tools do fundamentally similar things with different backends:
- `query_resource` — CKAN Datastore API (remote, structured)
- `sql_query` — CKAN SQL endpoint (remote, raw SQL)
- `query_cached` — local DuckDB (requires prior download)

The model must understand when each applies, and `sql_query` can only target one portal at a time (ArcGIS portals are excluded entirely). Both remote tools are rate-limited and ephemeral; `query_cached` is fast and composable.

**Alternative: one smart `query` tool that auto-caches**

```python
async def query(resource_id: str, sql: str) -> ToolResult:
    """Query a resource. Downloads and caches automatically on first call."""
    if not cache.is_cached(resource_id):
        await _download_resource(resource_id)  # transparent
    table = cache.get_table_name(resource_id)
    return cache.query(sql.replace("{table}", f'"{table}"'))
```

The model writes SQL once and doesn't need to know whether data is local yet. This eliminates `query_resource`, `sql_query`, and `preview_data` (replaced by `LIMIT 10`). **23 tools → ~15.**

**Trade-off:** The implicit download hides latency and quota usage. Could surface this as a note in the response ("Downloaded 45,000 rows to cache") rather than requiring an explicit prior call.

---

## Issue 2: The `PortalClient` Protocol Is a Leaky Abstraction

**Problem:** `protocols.py` defines only 3 methods (`package_search`, `package_show`, `resource_show`). But tools call `ckan.datastore_search()`, `ckan.datastore_sql()`, `ckan.datastore_search_all()` — all CKAN-specific methods not in the protocol — and guard against ArcGIS portals with scattered `arcgis_guard()` / `is_arcgis_portal()` checks throughout the tool modules.

**Alternative: capability queries on the client**

```python
class PortalClient(Protocol):
    def supports(self, capability: str) -> bool: ...
    # "datastore_sql", "datastore_search", "direct_download"
```

Replace every `is_arcgis_portal(ctx, portal)` guard with `if not client.supports("datastore_sql")`. The capability check is co-located with the client that knows the truth, not scattered across 5 tool files.

**Stronger alternative:** Fold datastore query into the protocol itself. ArcGIS client returns `None` for unsupported operations. Tools check for `None` once.

---

## Issue 3: `fan_out` + `first_match=True` Is Fragile for Resource Lookups

**Problem:** When a bare (unprefixed) resource ID is passed to a resource-level tool, `fan_out` queries all portals and returns the first success. A transient 429 or 5xx on the matching portal causes silent "not found" — the error is swallowed and other portals return their own "not found." This is the P2 bug in `known_issues.md`.

**Alternative: require prefixed IDs for resource operations**

Fan out only for discovery (`search_datasets`, `list_organizations`). For any operation on a specific resource or dataset, require the `portal:id` prefix. If a bare ID is passed, return an error asking for the prefixed form with a suggestion:

```
Resource ID must be prefixed: e.g. "ontario:abc123", "toronto:abc123".
Use search_datasets to find the dataset and its prefixed ID.
```

This eliminates the ambiguity entirely and makes failures explicit rather than silent.

---

## Issue 4: DuckDB as Both Metadata Store and Analytics Store

**Problem:** `_cache_metadata` and `_dataset_metadata` live in the same DuckDB file as all the data tables. Every `cache_info` or `is_cached()` call must open a DuckDB connection. Per-operation connections avoid lock contention but add setup overhead. Metadata is OLTP access patterns (point reads, updates); data tables are OLAP (analytical scans).

**Alternative: split the stores**

- **SQLite** for `_cache_metadata` and `_dataset_metadata` — OLTP-appropriate, WAL mode, genuinely concurrent, trivially fast for point reads
- **DuckDB** for data tables only — OLAP-appropriate, can be opened read-only for queries

This also makes `cache_info` and `is_cached()` fast without touching DuckDB at all, and allows multiple DuckDB readers simultaneously (attach read-only).

**Implementation note:** DuckDB can `ATTACH` a SQLite file directly via the `sqlite` extension, so cross-store queries remain possible if needed.

---

## Issue 5: Validate-After-Claim Is Optional and Therefore Often Skipped

**Problem:** `validate_result` is a separate tool the model can call after making a data claim. The friction of an extra tool call means it's frequently skipped — which defeats its purpose.

**Alternative: fold validation into `query_cached` as an optional parameter**

```python
async def query_cached(sql: str, claim: str | None = None) -> ToolResult:
    results = cache.query(sql)
    if claim:
        validation = _validate_claim(results, claim)
        results.structured["validation"] = validation
    return results
```

Validation becomes zero additional tool calls and always returns as structured output. The model passes its intended claim alongside the SQL. The response can flag mismatches inline.

---

## Issue 6: Unit Normalization Is a Manual Gotcha

**Problem:** The `known_issues.md` and MCP server instructions warn models to check unit columns before comparing datasets. This is implicit knowledge that must be re-communicated every session and still fails when the model misses it.

**Alternative: detect and surface unit conflicts automatically**

At `store_resource` time, detect unit columns (names matching `unit`, `units`, `uom`, `unit_of_measure`) and record them in cache metadata. In `_generate_query_warnings`, when a JOIN query references two tables, check if they share column names that have conflicting units in metadata:

```
⚠ Tables "ds_ontario_water_..." and "ds_toronto_water_..." both have a
"concentration" column but use different units (mg/L vs µg/L).
Values are not directly comparable.
```

This moves domain knowledge from documentation into runtime feedback.

---

## Issue 7: 23 Tools Is Too Many for Reliable Tool Selection

**Problem:** LLM tool-use performance degrades noticeably past ~15 tools in a single context window. The current 23 tools have significant overlap: 3 query tools, 2 spatial-adjacent tools, 4 cache management tools, and `list_resources` / `get_resource_schema` that largely duplicate what `get_dataset_info` returns.

**Proposed consolidation to 14 tools:**

| Group | Tools |
|-------|-------|
| Discovery (4) | `search_datasets`, `list_organizations`, `list_topics`, `find_related_datasets` |
| Inspect (3) | `get_dataset_info`, `get_resource_schema`, `compare_datasets` |
| Query (2) | `query` (merged auto-caching), `spatial_query` |
| Cache (3) | `cache_info`, `cache_manage`, `refresh_cache` |
| Quality (2) | `check_freshness`, `profile_data` |

**Eliminated:** `list_resources` (→ `get_dataset_info`), `preview_data` (→ `LIMIT 10` in `query`), `load_geodata` (→ `spatial_query` triggers auto-load), `query_resource` + `sql_query` (→ merged into `query`), `list_geo_datasets` (→ `search_datasets` with format filter), `validate_result` (→ optional param on `query`).

---

## Longer-Term: Two Separate MCPs

The server currently does two fundamentally different things:

1. **Data discovery** — fan out to all portals, return metadata. Network-bound, high latency, external dependency.
2. **Data analysis** — local DuckDB, SQL. CPU-bound, low latency, no external dependency.

These have different failure modes, different caching strategies, and different upgrade cadences. The cleanest long-term architecture splits them:

- **`ontario-portals-mcp`** — discovery and metadata only, no local state, stateless
- **`duckdb-analysis-mcp`** — analytics over local Parquet/DuckDB files, portal-agnostic

The analysis MCP becomes reusable for any data source, not just Ontario portals. The discovery MCP becomes simpler (no cache, no DuckDB dependency). The two communicate by the discovery MCP returning download URLs that the analysis MCP fetches and caches.

**Trade-off:** Two MCPs means more configuration overhead for users, and the tight integration (searching then immediately querying) becomes a multi-MCP workflow. Worth considering only if the current tool-count problem proves acute or if the analysis layer needs to grow independently (e.g., cross-city comparisons with non-Ontario portals).

---

## Priority Order

1. **Auto-caching query tool** (Issue 1) — eliminates the two-step workflow, reduces tool count, biggest UX improvement for the model
2. **Prefixed IDs required for resource ops** (Issue 3) — eliminates a class of silent bugs with a simple API constraint
3. **Capability-based protocol** (Issue 2) — cleans up scattered `arcgis_guard` calls throughout tool modules
4. **SQLite metadata / DuckDB data split** (Issue 4) — concurrency and performance improvement
5. **Inline validation** (Issue 5) and **unit conflict warnings** (Issue 6) — incremental improvements to existing tools
