# DuckDB 1.5 + TRY() + RENAME + Structured Output + asyncio Implementation Plan

> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Upgrade DuckDB to 1.5, add column name normalization, extend TRY() type detection, add structured output to two key tools, and unblock the event loop during large downloads.

**Architecture:** Five independent improvements that can be applied sequentially to the existing tool modules. Items 1-3 touch `cache.py` and `pyproject.toml`. Item 4 touches `tools/querying.py` and `tools/metadata.py`. Item 5 touches `tools/retrieval.py` and `cache.py`.

**Tech Stack:** fastmcp 3.0, DuckDB 1.5, pandas, asyncio, ToolResult (fastmcp.tools.tool)

---

## Files Modified

- `pyproject.toml` — bump duckdb constraint
- `src/ontario_data/cache.py` — column stripping, sci-notation detection, async store_resource
- `src/ontario_data/tools/querying.py` — ToolResult structured output for query_cached
- `src/ontario_data/tools/metadata.py` — ToolResult structured output for get_resource_schema
- `src/ontario_data/tools/retrieval.py` — asyncio.to_thread for blocking store calls
- `tests/test_cache.py` — extend for new normalization + sci-notation
- `tests/test_tools_unit.py` — extend for ToolResult return type
- `tests/test_server.py` — verify outputSchema on key tools

---

## Chunk 1: DuckDB bump + Column Name Normalization + TRY() Extension

### Task 1: Bump DuckDB to >=1.5.0

**Files:**
- Modify: `pyproject.toml`

- [ ] **Step 1: Update constraint**

In `pyproject.toml`, change:
```toml
"duckdb>=1.1.0",
```
to:
```toml
"duckdb>=1.5.0",
```

- [ ] **Step 2: Install**

```bash
uv sync
```

Expected: DuckDB 1.5.0 installed (or 1.4.4 if 1.5 isn't on PyPI yet — check with `uv pip show duckdb`).

- [ ] **Step 3: Run existing tests**

```bash
uv run pytest tests/ -v --ignore=tests/test_integration.py
```

Expected: All pass. If DuckDB 1.5 introduced breaking changes, failures will show here.

- [ ] **Step 4: Commit**

```bash
git add pyproject.toml uv.lock
git commit -m "feat: bump duckdb to >=1.5.0 for geometry type, non-blocking checkpoints, read_duckdb()"
```

---

### Task 2: Strip whitespace from column names at store time

**Files:**
- Modify: `src/ontario_data/cache.py` (around line 234 in `store_resource`)
- Test: `tests/test_cache.py`

Open data portals frequently return column names with leading/trailing whitespace (e.g., `" Count "` instead of `"Count"`). Stripping at storage time prevents downstream quoting issues.

- [ ] **Step 1: Write the failing test**

In `tests/test_cache.py`, add to `TestStoreDataFrame`:

```python
def test_strips_whitespace_from_column_names(self, cache):
    """Columns with leading/trailing spaces are normalized at storage time."""
    import pandas as pd
    df = pd.DataFrame([{"  Count  ": 1, " Name ": "Alice"}])
    cache.store_resource("res1", "ds1", "test_ws", df, "http://example.com")
    rows = cache.query("SELECT * FROM test_ws")
    assert rows[0] == {"Count": 1, "Name": "Alice"}

def test_strips_whitespace_does_not_alter_internal_content(self, cache):
    """Whitespace inside column values is left unchanged."""
    import pandas as pd
    df = pd.DataFrame([{"Name": "  Alice  "}])
    cache.store_resource("res2", "ds1", "test_ws2", df, "http://example.com")
    rows = cache.query('SELECT Name FROM test_ws2')
    assert rows[0]["Name"] == "  Alice  "
```

- [ ] **Step 2: Run tests to confirm failure**

```bash
uv run pytest tests/test_cache.py::TestStoreDataFrame::test_strips_whitespace_from_column_names -v
```

Expected: FAIL — column name still has spaces.

- [ ] **Step 3: Implement the fix**

In `src/ontario_data/cache.py`, inside `store_resource()`, immediately before the `CREATE TABLE` call (line ~234):

```python
# Normalize column names: strip leading/trailing whitespace
df = df.copy()
df.columns = df.columns.str.strip()
```

The full block should look like:

```python
# Normalize column names: strip leading/trailing whitespace
df = df.copy()
df.columns = df.columns.str.strip()

# Create table from DataFrame
conn.execute(f'CREATE TABLE "{table_name}" AS SELECT * FROM df')
```

- [ ] **Step 4: Run tests**

```bash
uv run pytest tests/test_cache.py::TestStoreDataFrame -v
```

Expected: All pass including the two new tests.

- [ ] **Step 5: Update query_cached docstring to mention RENAME**

In `src/ontario_data/tools/querying.py`, the `query_cached` docstring already has gotchas. Add one more line:

Current docstring includes:
```
Column names vary across resources in the same dataset — always DESCRIBE first.
```

Append after that line:
```
Use SELECT * RENAME ("old col" AS new_col) to normalize column names across datasets in a JOIN.
```

- [ ] **Step 6: Commit**

```bash
git add src/ontario_data/cache.py src/ontario_data/tools/querying.py tests/test_cache.py
git commit -m "feat: strip whitespace from column names at store time, document SELECT * RENAME"
```

---

### Task 3: Extend TRY() type detection for scientific notation

**Files:**
- Modify: `src/ontario_data/cache.py` (`_detect_numeric_varchars`, `store_resource`)
- Test: `tests/test_cache.py`

DuckDB 1.3+ generalizes `TRY()` to any expression, not just CAST. The current numeric detection handles plain decimals and comma-formatted numbers. Extend it to handle scientific notation strings (e.g., `"1.23e-5"`, `"4.5E+10"`) which appear in environmental and measurement datasets.

- [ ] **Step 1: Write the failing test**

```python
def test_detects_scientific_notation_numerics(self, cache):
    """Columns with scientific notation strings are cast to DOUBLE."""
    import pandas as pd
    df = pd.DataFrame([
        {"measurement": "1.23e-5", "label": "trace"},
        {"measurement": "4.56E+10", "label": "large"},
        {"measurement": "7.89e0", "label": "unit"},
    ])
    cache.store_resource("res_sci", "ds1", "test_sci", df, "http://example.com")
    rows = cache.query('SELECT measurement FROM test_sci WHERE measurement < 0.001')
    assert len(rows) == 1
    assert abs(rows[0]["measurement"] - 1.23e-5) < 1e-10
```

- [ ] **Step 2: Run to confirm failure**

```bash
uv run pytest tests/test_cache.py -k test_detects_scientific_notation -v
```

Expected: FAIL — `measurement` stays VARCHAR.

- [ ] **Step 3: Add sci-notation regex to `_detect_numeric_varchars`**

In `src/ontario_data/cache.py`, `_detect_numeric_varchars` currently has:

```python
plain_re = re.compile(r"^-?\d+\.?\d*$")
comma_re = re.compile(r"^-?\d{1,3}(,\d{3})+(\.\d+)?$")
```

Add a third regex after those two lines:

```python
sci_re = re.compile(r"^-?\d+\.?\d*[eE][+-]?\d+$")
```

Then update the counting logic:

Current:
```python
plain_count = sum(1 for v in values if plain_re.match(v))
comma_count = sum(1 for v in values if comma_re.match(v))
numeric_count = plain_count + comma_count
if numeric_count / len(values) > 0.8:
    suspects.append({"name": col_name, "has_commas": comma_count > 0})
```

Replace with:
```python
plain_count = sum(1 for v in values if plain_re.match(v))
comma_count = sum(1 for v in values if comma_re.match(v))
sci_count = sum(1 for v in values if sci_re.match(v))
numeric_count = plain_count + comma_count + sci_count
if numeric_count / len(values) > 0.8:
    suspects.append({
        "name": col_name,
        "has_commas": comma_count > 0,
        "has_sci": sci_count > 0,
    })
```

- [ ] **Step 4: Update the ALTER TABLE expression in `store_resource`**

Current ALTER TABLE section in `store_resource`:
```python
for col_info in numeric_varchars:
    c = col_info["name"].replace('"', '""')
    if col_info["has_commas"]:
        expr = f'TRY_CAST(REPLACE("{c}", \',\', \'\') AS DOUBLE)'
    else:
        expr = f'TRY_CAST("{c}" AS DOUBLE)'
    try:
        conn.execute(
            f'ALTER TABLE "{table_name}" ALTER "{c}" '
            f'TYPE DOUBLE USING {expr}'
        )
```

Replace with:
```python
for col_info in numeric_varchars:
    c = col_info["name"].replace('"', '""')
    if col_info["has_commas"]:
        expr = f'TRY_CAST(REPLACE("{c}", \',\', \'\') AS DOUBLE)'
    elif col_info.get("has_sci"):
        # TRY() wraps any expression — handles "1.23e-5" style strings
        expr = f'TRY(CAST("{c}" AS DOUBLE))'
    else:
        expr = f'TRY_CAST("{c}" AS DOUBLE)'
    try:
        conn.execute(
            f'ALTER TABLE "{table_name}" ALTER "{c}" '
            f'TYPE DOUBLE USING {expr}'
        )
```

- [ ] **Step 5: Run tests**

```bash
uv run pytest tests/test_cache.py -v
```

Expected: All pass including new sci-notation test.

- [ ] **Step 6: Commit**

```bash
git add src/ontario_data/cache.py tests/test_cache.py
git commit -m "feat: detect scientific notation columns and cast with TRY() in store_resource"
```

---

## Chunk 2: Structured Output for query_cached and get_resource_schema

fastmcp 3.0 supports returning `ToolResult(content=..., structured_content=...)` directly from tool functions (passthrough path in `convert_result`). This lets Claude read markdown while programmatic clients parse typed JSON.

### Task 4: Structured output for query_cached

**Files:**
- Modify: `src/ontario_data/tools/querying.py`
- Test: `tests/test_tools_unit.py`, `tests/test_server.py`

The `output_schema` dict is passed to `@mcp.tool()`. The tool returns `ToolResult` directly with both content (markdown) and structured_content (typed dict).

- [ ] **Step 1: Write the test**

In `tests/test_tools_unit.py`, add (alongside existing query_cached tests):

```python
from fastmcp.tools.tool import ToolResult

async def test_query_cached_returns_structured_content(make_mock_context):
    """query_cached returns ToolResult with structured_content alongside text."""
    import pandas as pd
    from ontario_data.tools.querying import query_cached

    cache = make_mock_context.cache
    df = pd.DataFrame([{"city": "Toronto", "pop": 2930000}])
    cache.store_resource("r1", "ds1", "test_cities", df, "http://example.com")

    ctx = make_mock_context(cache)
    result = await query_cached('SELECT * FROM "test_cities"', ctx=ctx)

    assert isinstance(result, ToolResult)
    assert result.structured_content is not None
    assert "rows" in result.structured_content
    assert result.structured_content["rows"][0]["city"] == "Toronto"
    assert any("Toronto" in str(block) for block in result.content)  # text still present
```

In `tests/test_server.py`, add:

```python
async def test_query_cached_has_output_schema(self):
    """query_cached declares outputSchema for structured clients."""
    async with Client(mcp) as client:
        tools = await client.list_tools()
        qc = next(t for t in tools if t.name == "query_cached")
        assert qc.outputSchema is not None
        assert qc.outputSchema.get("type") == "object"
        assert "rows" in qc.outputSchema.get("properties", {})
```

- [ ] **Step 2: Run to confirm failures**

```bash
uv run pytest tests/test_tools_unit.py -k query_cached -v
uv run pytest tests/test_server.py -k output_schema -v
```

Expected: Both fail — current return is `str`, no outputSchema.

- [ ] **Step 3: Implement**

At the top of `src/ontario_data/tools/querying.py`, add import:

```python
from fastmcp.tools.tool import ToolResult
```

Define the output schema constant near the top of the file (after imports):

```python
_QUERY_CACHED_OUTPUT_SCHEMA = {
    "type": "object",
    "properties": {
        "sql": {"type": "string", "description": "The SQL query that was executed"},
        "rows": {
            "type": "array",
            "items": {"type": "object"},
            "description": "Result rows as JSON objects",
        },
        "columns": {
            "type": "array",
            "items": {"type": "string"},
            "description": "Column names in result order",
        },
        "total_rows": {
            "type": "integer",
            "description": "Total rows matched (may exceed rows if truncated)",
        },
        "truncated": {
            "type": "boolean",
            "description": "True if results were truncated to MAX_QUERY_ROWS",
        },
        "warnings": {
            "type": "array",
            "items": {"type": "string"},
            "description": "Heuristic warnings (e.g. COUNT(*) vs SUM suggestion)",
        },
    },
    "required": ["sql", "rows", "columns", "truncated", "warnings"],
}
```

Change the decorator for `query_cached`:

```python
@mcp.tool(annotations=READONLY, output_schema=_QUERY_CACHED_OUTPUT_SCHEMA)
async def query_cached(
    sql: str,
    ctx: Context = None,
) -> ToolResult:
```

At the end of `query_cached`, replace the `return "\n".join(parts)` with:

```python
        markdown = "\n".join(parts)
        col_names = [f["name"] for f in fields] if fields else []
        return ToolResult(
            content=markdown,
            structured_content={
                "sql": sql,
                "rows": results,
                "columns": col_names,
                "total_rows": truncated_total if truncated_total is not None else len(results),
                "truncated": truncated_total is not None,
                "warnings": warnings,
            },
        )
```

The `except` block at the bottom still raises `InvalidQueryError` — no change needed there (error path doesn't return ToolResult).

- [ ] **Step 4: Run tests**

```bash
uv run pytest tests/test_tools_unit.py tests/test_server.py -v
```

Expected: All pass.

- [ ] **Step 5: Commit**

```bash
git add src/ontario_data/tools/querying.py tests/test_tools_unit.py tests/test_server.py
git commit -m "feat: add structured output (outputSchema + ToolResult) to query_cached"
```

---

### Task 5: Structured output for get_resource_schema

**Files:**
- Modify: `src/ontario_data/tools/metadata.py`
- Test: `tests/test_server.py`

- [ ] **Step 1: Write the test**

In `tests/test_server.py`, add:

```python
async def test_get_resource_schema_has_output_schema(self):
    """get_resource_schema declares outputSchema with fields array."""
    async with Client(mcp) as client:
        tools = await client.list_tools()
        grs = next(t for t in tools if t.name == "get_resource_schema")
        assert grs.outputSchema is not None
        assert "fields" in grs.outputSchema.get("properties", {})
```

- [ ] **Step 2: Run to confirm failure**

```bash
uv run pytest tests/test_server.py -k resource_schema -v
```

Expected: FAIL.

- [ ] **Step 3: Implement**

At the top of `src/ontario_data/tools/metadata.py`, add:

```python
from fastmcp.tools.tool import ToolResult
```

Add the output schema constant near imports:

```python
_RESOURCE_SCHEMA_OUTPUT_SCHEMA = {
    "type": "object",
    "properties": {
        "resource_id": {"type": "string"},
        "total_records": {"type": "integer"},
        "num_columns": {"type": "integer"},
        "fields": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "type": {"type": "string"},
                    "sample_values": {
                        "type": "array",
                        "items": {"type": "string"},
                    },
                },
                "required": ["name", "type"],
            },
        },
        "datastore_active": {"type": "boolean"},
        "hint": {"type": "string"},
    },
    "required": ["resource_id"],
}
```

Change the decorator for `get_resource_schema`:

```python
@mcp.tool(annotations=READONLY, output_schema=_RESOURCE_SCHEMA_OUTPUT_SCHEMA)
async def get_resource_schema(
    resource_id: str,
    sample_size: int = 5,
    ctx: Context = None,
) -> ToolResult:
```

Change the final `return md_response(**data)` to:

```python
    markdown = md_response(**data)
    return ToolResult(content=markdown, structured_content=data)
```

The `data` dict already has the right shape (`resource_id`, `total_records`, `num_columns`, `fields` or `datastore_active`/`hint`) to match the schema.

- [ ] **Step 4: Run tests**

```bash
uv run pytest tests/test_server.py -v
```

Expected: All pass.

- [ ] **Step 5: Commit**

```bash
git add src/ontario_data/tools/metadata.py tests/test_server.py
git commit -m "feat: add structured output (outputSchema + ToolResult) to get_resource_schema"
```

---

## Chunk 3: Unblock Event Loop During Downloads

### Task 6: asyncio.to_thread for blocking store_resource calls

**Files:**
- Modify: `src/ontario_data/tools/retrieval.py`
- Test: `tests/test_tools_unit.py`

`cache.store_resource()` and `cache.store_dataset_metadata()` are synchronous DuckDB operations. For large DataFrames (XLSX files with 100k+ rows), they block the entire asyncio event loop for several seconds. `asyncio.to_thread()` offloads them to a thread pool.

Note: Full MCP Tasks (SEP-1686) via `task=TaskConfig(mode="optional")` requires installing `pydocket` (a Redis-backed job queue). That's external infrastructure not appropriate for a local server. `asyncio.to_thread()` is the right fix for a local stdio server.

- [ ] **Step 1: Write the test**

In `tests/test_tools_unit.py`, add:

```python
import asyncio

async def test_download_resource_uses_thread_for_store(make_mock_context, monkeypatch):
    """store_resource is called in a thread, not blocking the event loop."""
    import pandas as pd
    from ontario_data.tools import retrieval

    calls = []

    def fake_store(**kwargs):
        # Record that we're NOT on the main thread
        calls.append(asyncio.get_event_loop() if False else "thread_call")

    # Patch the download to return a small DataFrame
    async def fake_download(ckan, resource_id, http_client):
        df = pd.DataFrame([{"col": 1}])
        resource = {"id": resource_id, "package_id": "ds1", "format": "CSV",
                    "url": "http://x.com", "datastore_active": False}
        dataset = {"id": "ds1", "name": "test", "title": "Test", "resources": [resource]}
        return df, resource, dataset

    cache = make_mock_context.cache
    ctx = make_mock_context(cache)
    monkeypatch.setattr(retrieval, "_download_resource_data", fake_download)
    monkeypatch.setattr(cache, "store_resource", fake_store)

    await retrieval.download_resource("ontario:abc123", ctx=ctx)
    assert len(calls) == 1  # store_resource was called
```

- [ ] **Step 2: Run to confirm it passes (behavior test, not threading assertion)**

```bash
uv run pytest tests/test_tools_unit.py -k download_resource -v
```

Note: This test verifies `store_resource` is called; strict thread-isolation testing is complex. The goal is to verify no regression.

- [ ] **Step 3: Implement in download_resource**

In `src/ontario_data/tools/retrieval.py`, add import at top:

```python
import asyncio
```

In `download_resource`, replace the synchronous calls:

Current (lines ~156-164):
```python
    table_name = make_table_name(dataset.get("name", ""), bare_id, portal=portal)
    cache.store_resource(
        resource_id=bare_id,
        dataset_id=dataset.get("id", ""),
        table_name=table_name,
        df=df,
        source_url=resource.get("url", ""),
    )
    cache.store_dataset_metadata(dataset.get("id", ""), dataset)
```

Replace with:
```python
    table_name = make_table_name(dataset.get("name", ""), bare_id, portal=portal)
    await asyncio.to_thread(
        cache.store_resource,
        resource_id=bare_id,
        dataset_id=dataset.get("id", ""),
        table_name=table_name,
        df=df,
        source_url=resource.get("url", ""),
    )
    await asyncio.to_thread(
        cache.store_dataset_metadata,
        dataset.get("id", ""),
        dataset,
    )
```

- [ ] **Step 4: Apply same fix in refresh_cache**

In `refresh_cache`, the `store_resource` call inside the loop (line ~280):

Current:
```python
            cache.store_resource(
                resource_id=item["resource_id"],
                dataset_id=item["dataset_id"],
                table_name=item["table_name"],
                df=df,
                source_url=item["source_url"],
            )
```

Replace with:
```python
            await asyncio.to_thread(
                cache.store_resource,
                resource_id=item["resource_id"],
                dataset_id=item["dataset_id"],
                table_name=item["table_name"],
                df=df,
                source_url=item["source_url"],
            )
```

- [ ] **Step 5: Run all tests**

```bash
uv run pytest tests/ -v --ignore=tests/test_integration.py
```

Expected: All pass.

- [ ] **Step 6: Commit**

```bash
git add src/ontario_data/tools/retrieval.py tests/test_tools_unit.py
git commit -m "feat: offload store_resource to thread pool to unblock event loop during downloads"
```

---

## Summary

| Item | Change | Files |
|---|---|---|
| 1 | DuckDB >=1.5.0 | `pyproject.toml` |
| 2 | TRY() for sci-notation | `cache.py` |
| 3 | Strip column whitespace + RENAME docs | `cache.py`, `querying.py` |
| 4 | Structured output (query_cached) | `querying.py`, `test_server.py` |
| 4 | Structured output (get_resource_schema) | `metadata.py`, `test_server.py` |
| 5 | asyncio.to_thread for store_resource | `retrieval.py` |

**Python version:** No change needed. fastmcp 3.0 requires ≥3.10, DuckDB ≥3.9. Current `>=3.10` covers all dependencies.

**Note on full MCP Tasks (SEP-1686):** Enabling `task=TaskConfig(mode="optional")` on `download_resource` requires installing `fastmcp[tasks]` (pydocket + Redis). For a local stdio server, `asyncio.to_thread()` is the appropriate solution.
