# Function Traces — ontario-data-mcp

End-to-end trace of every function in the codebase, organized by module.

---

## Table of Contents

1. [Entry Points](#1-entry-points)
2. [Server & Lifespan](#2-server--lifespan)
3. [Portal Configuration](#3-portal-configuration)
4. [Logging](#4-logging)
5. [CKAN Client](#5-ckan-client)
6. [ArcGIS Hub Client](#6-arcgis-hub-client)
7. [Cache Manager](#7-cache-manager)
8. [Utilities](#8-utilities)
9. [Staleness](#9-staleness)
10. [Formatting](#10-formatting)
11. [CLI](#11-cli)
12. [Tools — Discovery](#12-tools--discovery)
13. [Tools — Metadata](#13-tools--metadata)
14. [Tools — Retrieval](#14-tools--retrieval)
15. [Tools — Querying](#15-tools--querying)
16. [Tools — Quality](#16-tools--quality)
17. [Tools — Geospatial](#17-tools--geospatial)
18. [Prompts](#18-prompts)
19. [Resources](#19-resources)
20. [Frontend — carousel.js](#20-frontend--carouseljs)
21. [Frontend — sql-highlight.js](#21-frontend--sql-highlightjs)

---

## 1. Entry Points

### `__main__.py:main()` (line 1)
**Signature:** `(none)` — module-level import + call
**Purpose:** CLI entry point. Imports and calls `server.main()`.
**Flow:**
1. `from ontario_data.server import main` — triggers full server module load (FastMCP init, tool registration).
2. Calls `main()` → dispatches to either MCP server or cache CLI.

---

## 2. Server & Lifespan

### `server.py:lifespan(server)` (line 26)
**Signature:** `async context manager (server) → yields dict`
**Purpose:** FastMCP lifespan — initializes shared state for the server's lifetime.
**Flow:**
1. Calls `setup_logging()` → configures rotating file logger.
2. Creates `httpx.AsyncClient(timeout=ONTARIO_DATA_TIMEOUT or 30)` — shared HTTP client for all portals.
3. Creates `CacheManager()` → resolves `~/.cache/ontario-data/ontario_data.duckdb`.
4. Calls `cache.initialize()` → creates metadata tables, installs DuckDB extensions (httpfs, json, spatial).
5. Creates empty `portal_clients: dict` — lazily populated by `get_deps()`.
6. **Yields** state dict: `{http_client, portal_configs, portal_clients, cache}`.
7. On shutdown: closes all portal clients, closes httpx client.

### `server.py:main()` (line 84)
**Signature:** `() → None`
**Purpose:** Top-level dispatcher between MCP server mode and cache CLI mode.
**Flow:**
1. If `sys.argv[1] == "cache"` → imports `cli.run`, calls `run(sys.argv[2:])`.
2. Otherwise → calls `mcp.run()` which starts the FastMCP server (stdio transport).

### Module-level side effects (lines 50–81)
- `sys.modules.setdefault("ontario_data.server", ...)` — ensures single FastMCP instance when loaded via `fastmcp run`.
- Creates `mcp = FastMCP(...)` with server name, instructions, version, lifespan.
- Defines `READONLY` and `DESTRUCTIVE` ToolAnnotations constants.
- Imports all tool modules (`discovery`, `metadata`, `retrieval`, `querying`, `quality`, `geospatial`) and `prompts`, `resources` — their `@mcp.tool`/`@mcp.prompt`/`@mcp.resource` decorators register handlers.

---

## 3. Portal Configuration

### `portals.py:PortalType` (line 7)
**Type:** `StrEnum` with values `CKAN`, `ARCGIS_HUB`.
**Purpose:** Discriminator for which client class to instantiate per portal.

### `portals.py:PortalConfig` (line 12)
**Type:** Frozen dataclass with fields: `name`, `base_url`, `portal_type`, `description`, `licence_name`, `licence_url`.
**Purpose:** Immutable configuration for a single data portal.

### `portals.py:PORTALS` (line 22)
**Type:** `dict[str, PortalConfig]` — three entries: `"ontario"`, `"toronto"`, `"ottawa"`.
**Purpose:** Registry of all configured portals. Used by `get_deps()`, `fan_out()`, `parse_portal_id()`.

---

## 4. Logging

### `logging_config.py:setup_logging()` (line 8)
**Signature:** `() → logging.Logger`
**Purpose:** Configures JSON-line rotating file logger.
**Flow:**
1. Reads `LOG_LEVEL` env var (default `"WARNING"`).
2. Gets/creates `"ontario_data"` logger.
3. If no handlers exist: creates `~/.cache/ontario-data/server.log` with `RotatingFileHandler(5MB, 3 backups)`.
4. Sets JSON-line formatter: `{"time":"...","level":"...","module":"...","message":"..."}`.
5. Returns the configured logger.

**Called by:** `lifespan()` at server startup.

---

## 5. CKAN Client

### `ckan_client.py:CKANClient.__init__(...)` (line 26)
**Signature:** `(base_url, timeout, http_client, max_retries, base_delay, rate_limit)`
**Flow:**
1. Strips trailing `/` from `base_url`, builds `api_url = base_url + "/api/3/action"`.
2. Reads timeout from env `ONTARIO_DATA_TIMEOUT` (default 30s) if not provided.
3. Stores httpx client ref; tracks `_owns_client` for cleanup ownership.
4. Reads rate limit from env `ONTARIO_DATA_RATE_LIMIT` (default 10 req/s), computes `_min_interval`.

### `ckan_client.py:CKANClient._get_client()` (line 49)
**Signature:** `async () → httpx.AsyncClient`
**Flow:** Lazily creates an `httpx.AsyncClient` if none was injected. Sets `_owns_client = True`.

### `ckan_client.py:CKANClient._rate_limit()` (line 55)
**Signature:** `async () → None`
**Flow:** Enforces minimum interval between requests. Calculates elapsed time since last request; sleeps the difference if too soon.

### `ckan_client.py:CKANClient._request(action, params)` (line 64)
**Signature:** `async (action: str, params: dict | None) → Any`
**Purpose:** Core HTTP method for all CKAN API calls. Handles retry + rate limiting.
**Flow:**
1. Gets httpx client via `_get_client()`.
2. Builds URL: `{api_url}/{action}`.
3. Loop up to `max_retries + 1` attempts:
   a. Calls `_rate_limit()` — enforces request spacing.
   b. `await client.get(url, params=params)`.
   c. If status in `{429, 500, 502, 503, 504}` and retries remain → exponential backoff (`base_delay * 2^attempt + jitter`), continue.
   d. `response.raise_for_status()`.
   e. Parses JSON; if `success=False` → raises `CKANError(message)`.
   f. Returns `data["result"]`.
4. On `ConnectError`/`ReadTimeout`/`WriteTimeout` → same exponential backoff retry; re-raises on final attempt.

### `ckan_client.py:CKANClient.close()` (line 103)
**Signature:** `async () → None`
**Flow:** Closes httpx client only if this instance owns it (`_owns_client`). Sets `_http_client = None`.

### `ckan_client.py:CKANClient.package_search(...)` (line 109)
**Signature:** `async (query, filters, sort, rows, start, facet_fields) → dict`
**Flow:**
1. Builds params dict: `{q, rows, start}`.
2. If filters → builds `fq` string from `{k}:{v}` pairs.
3. If sort → adds `sort` param.
4. If facet_fields → adds `facet.field` and `facet=true`.
5. Calls `_request("package_search", params)` → returns CKAN search result dict.

### `ckan_client.py:CKANClient.package_search_all(...)` (line 129)
**Signature:** `async (query, filters, sort, page_size) → list[dict]`
**Purpose:** Auto-paginating wrapper around `package_search`.
**Flow:**
1. Loop with incrementing `start`:
   a. Calls `package_search(...)`.
   b. Extends `all_results` with `result["results"]`.
   c. Breaks when `len(all_results) >= result["count"]`.
2. Returns flat list of all dataset dicts.

### `ckan_client.py:CKANClient.package_show(id)` (line 149)
**Signature:** `async (id: str) → dict`
**Flow:** Calls `_request("package_show", {"id": id})`. Returns full dataset metadata dict.

### `ckan_client.py:CKANClient.resource_show(id)` (line 152)
**Signature:** `async (id: str) → dict`
**Flow:** Calls `_request("resource_show", {"id": id})`. Returns resource metadata dict.

### `ckan_client.py:CKANClient.resource_search(query, order_by, limit, offset)` (line 155)
**Signature:** `async (query, order_by, limit, offset) → dict`
**Flow:** Builds params, calls `_request("resource_search", params)`.

### `ckan_client.py:CKANClient.datastore_search(...)` (line 171)
**Signature:** `async (resource_id, filters, fields, sort, limit, offset) → dict`
**Flow:**
1. Builds params: `{resource_id, limit, offset}`.
2. If filters → JSON-serializes to `params["filters"]`.
3. If fields → comma-joins to `params["fields"]`.
4. If sort → adds sort param.
5. Calls `_request("datastore_search", params)`.

### `ckan_client.py:CKANClient.datastore_search_all(...)` (line 194)
**Signature:** `async (resource_id, filters, fields, sort, page_size) → dict`
**Purpose:** Auto-paginating `datastore_search`.
**Flow:**
1. Loop: calls `datastore_search(offset=offset)`, extends `all_records`.
2. Captures `fields` and `total` from first page.
3. Breaks when `len(all_records) >= total` or empty page.
4. Returns `{records, fields, total}`.

### `ckan_client.py:CKANClient.datastore_sql(sql)` (line 229)
**Signature:** `async (sql: str) → dict`
**Flow:** Calls `_request("datastore_search_sql", {"sql": sql})`. Remote SQL execution.

### `ckan_client.py:CKANClient.tag_list(query, all_fields)` (line 232)
**Signature:** `async (query, all_fields) → list`
**Flow:** Calls `_request("tag_list", params)`. Returns list of tag strings or dicts.

### `ckan_client.py:CKANClient.organization_list(sort, all_fields, include_dataset_count)` (line 238)
**Signature:** `async (sort, all_fields, include_dataset_count) → list`
**Flow:** Calls `_request("organization_list", params)`. Returns list of org dicts.

### `ckan_client.py:CKANClient.group_list(sort, all_fields, include_dataset_count)` (line 250)
**Signature:** `async (sort, all_fields, include_dataset_count) → list`
**Flow:** Calls `_request("group_list", params)`. Returns list of group dicts.

### `ckan_client.py:CKANClient.package_list(limit, offset)` (line 262)
**Signature:** `async (limit, offset) → list[str]`
**Flow:** Calls `_request("package_list", params)`. Returns list of package name strings.

---

## 6. ArcGIS Hub Client

### `arcgis_client.py:ArcGISHubClient.__init__(...)` (line 20)
**Signature:** `(base_url, http_client, org_name, org_title)`
**Flow:** Stores base URL (stripped trailing `/`), HTTP client, org metadata. Sets `_owns_client` flag.

### `arcgis_client.py:ArcGISHubClient._get_client()` (line 32)
**Signature:** `async () → httpx.AsyncClient`
**Flow:** Lazily creates httpx client with 30s timeout if none injected.

### `arcgis_client.py:ArcGISHubClient.close()` (line 38)
**Signature:** `async () → None`
**Flow:** Closes httpx client if owned.

### `arcgis_client.py:ArcGISHubClient.package_search(...)` (line 45)
**Signature:** `async (query, filters, sort, rows, start, facet_fields) → dict`
**Purpose:** Searches via OGC Records API, returns CKAN-shaped `{count, results}`.
**Flow:**
1. Builds params: `{limit: rows}`, optional `startindex` and `q`.
2. GET `{base_url}/api/search/v1/collections/all/items`.
3. For each feature in response:
   a. Extracts `id` from properties; appends `_0` suffix for layered types (Feature Service, Map Service) via `_is_layered_type()`.
   b. Builds CKAN-compatible dataset dict: `{id, name, title, notes, tags, resources, ...}`.
   c. Slugifies title via `_slugify()`.
4. Returns `{count: numberMatched, results: [...]}`.

### `arcgis_client.py:ArcGISHubClient.package_show(id)` (line 118)
**Signature:** `async (id: str) → dict`
**Purpose:** Gets dataset metadata via Hub v3 API, returns CKAN-like dict.
**Flow:**
1. GET `{base_url}/api/v3/datasets/{id}`.
2. If 404 and `id` has no `_` → retries with `{id}_0` (Feature Service layer convention).
3. Extracts `attributes` from response.
4. Builds CKAN-compatible dict with: `id, name, title, notes, tags, resources, update_frequency, license_title, geographic_coverage`.

### `arcgis_client.py:ArcGISHubClient.resource_show(id)` (line 168)
**Signature:** `async (id: str) → dict`
**Purpose:** Synthesizes a CKAN-like resource dict from dataset metadata.
**Flow:**
1. Calls `self.package_show(id)`.
2. Searches resources for matching `id`; adds `package_id`.
3. If no match → returns minimal dict with `{id, package_id, name, format, url, datastore_active}`.

### `arcgis_client.py:ArcGISHubClient.organization_list(...)` (line 189)
**Signature:** `async (...) → list[dict]`
**Flow:** Returns single-element list with org metadata (single-org portal stub).

### `arcgis_client.py:ArcGISHubClient.tag_list(...)` (line 201)
**Signature:** `async (...) → list`
**Flow:** Returns empty list (ArcGIS Hub has no tags endpoint).

### `arcgis_client.py:ArcGISHubClient.get_download_url(dataset_id, fmt)` (line 206)
**Signature:** `async (dataset_id: str, fmt: str) → str | None`
**Purpose:** Gets bulk download URL from Downloads API.
**Flow:**
1. GET `{base_url}/api/v3/datasets/{dataset_id}/downloads?spatialRefId=4326&format={fmt}`.
2. If 404 → returns `None`.
3. Iterates response `data[]`, finds entry matching format with a URL.
4. Returns URL string or `None`.

### `arcgis_client.py:_is_layered_type(item_type)` (line 233)
**Signature:** `(item_type: str) → bool`
**Flow:** Returns `True` if `item_type` is `"Feature Service"` or `"Map Service"`.

### `arcgis_client.py:_slugify(title)` (line 238)
**Signature:** `(title: str) → str`
**Flow:** Lowercases, replaces non-alphanumeric with `-`, collapses runs, truncates to 80 chars.

---

## 7. Cache Manager

### `cache.py:_has_semicolons_outside_strings(sql)` (line 26)
**Signature:** `(sql: str) → bool`
**Purpose:** SQL injection defense — detects semicolons not inside string literals.
**Flow:** State-machine parser tracking `in_string` state. Handles SQL-standard `''` escaping. Returns `True` if `;` found outside quotes.

### `cache.py:_validate_sql(sql)` (line 49)
**Signature:** `(sql: str) → None` (raises `InvalidQueryError`)
**Purpose:** Ensures SQL is read-only and injection-safe.
**Flow:**
1. Strips comments (`/* */` and `--`).
2. Checks for semicolons outside strings via `_has_semicolons_outside_strings()`.
3. Extracts first word; rejects if not in `_ALLOWED_PREFIXES` (`select`, `with`, `explain`, `describe`, `show`, `pragma`, `summarize`).

### `cache.py:CacheManager.__init__(db_path)` (line 82)
**Signature:** `(db_path: str | None)`
**Flow:**
1. If `db_path` is None → resolves to `~/.cache/ontario-data/ontario_data.duckdb` (via env `ONTARIO_DATA_CACHE_DIR`).
2. Creates cache directory if needed.
3. Initializes `_extensions = []`, `_has_spatial = False`.

### `cache.py:CacheManager._connect_raw()` (line 94)
**Signature:** `context manager → yields duckdb.Connection`
**Flow:** Opens raw connection to `self.db_path`, yields it, closes on exit. No extension loading.

### `cache.py:CacheManager._connect()` (line 101)
**Signature:** `context manager → yields duckdb.Connection`
**Flow:** Opens connection, loads all registered `_extensions`, yields, closes.

### `cache.py:CacheManager._with_retry(fn, max_attempts)` (line 111)
**Signature:** `(fn: Callable[[Connection], T], max_attempts=3) → T`
**Purpose:** Retries with linear backoff on DuckDB file lock contention (`IOException`).
**Flow:** Loop up to `max_attempts`: calls `fn(conn)` via `_connect()`. On `IOException` → sleeps `0.1 * (attempt + 1)`. Re-raises on final attempt.

### `cache.py:CacheManager.initialize()` (line 123)
**Signature:** `() → None`
**Purpose:** One-time schema + extension setup.
**Flow:**
1. Opens raw connection.
2. Creates `_cache_metadata` table (resource_id PK, dataset_id, table_name, downloaded_at, row_count, size_bytes, source_url, expires_at).
3. Migration: `ALTER TABLE ADD COLUMN type_warnings JSON` (swallows error if exists).
4. Creates `_dataset_metadata` table (dataset_id PK, metadata JSON, cached_at).
5. Installs + loads `httpfs`, `json` extensions.
6. Attempts `spatial` extension; sets `_has_spatial = True/False`.

### `cache.py:CacheManager.has_spatial_extension` (line 179)
**Signature:** `@property → bool`
**Flow:** Returns `self._has_spatial`.

### `cache.py:CacheManager._detect_numeric_varchars(conn, table_name)` (line 183)
**Signature:** `@staticmethod (conn, table_name: str) → list[str]`
**Purpose:** Finds VARCHAR columns with >80% numeric values.
**Flow:**
1. `DESCRIBE "{table_name}"` → get columns.
2. For each VARCHAR column: samples 100 distinct non-null values.
3. Regex-tests each value against `^-?\d+\.?\d*$`.
4. If >80% match → column is a suspect.
5. Returns list of suspect column names.

### `cache.py:CacheManager.store_resource(...)` (line 205)
**Signature:** `(resource_id, dataset_id, table_name, df: DataFrame, source_url)`
**Purpose:** Upserts a resource into the cache.
**Flow (inside `_with_retry`):**
1. Checks for existing entry by `resource_id` → drops old table + metadata if found.
2. `CREATE TABLE "{table_name}" AS SELECT * FROM df` — ingests DataFrame.
3. Calls `_detect_numeric_varchars(conn, table_name)` → gets type warnings.
4. Inserts metadata row: resource_id, dataset_id, table_name, now(), row_count, size_bytes, source_url, type_warnings JSON.

### `cache.py:CacheManager.is_cached(resource_id)` (line 246)
**Signature:** `(resource_id: str) → bool`
**Flow:** SELECT 1 from `_cache_metadata` where resource_id matches. Returns `True`/`False`.

### `cache.py:CacheManager.get_table_name(resource_id)` (line 253)
**Signature:** `(resource_id: str) → str | None`
**Flow:** SELECT table_name from `_cache_metadata`. Returns string or `None`.

### `cache.py:CacheManager.list_cached()` (line 260)
**Signature:** `() → list[dict]`
**Flow:** SELECT all from `_cache_metadata` ordered by `downloaded_at DESC`. Returns list of dicts.

### `cache.py:CacheManager.remove_resource(resource_id)` (line 279)
**Signature:** `(resource_id: str) → None`
**Flow (inside `_with_retry`):**
1. Looks up table_name by resource_id.
2. `DROP TABLE IF EXISTS "{table_name}"`.
3. `DELETE FROM _cache_metadata WHERE resource_id = ?`.

### `cache.py:CacheManager.remove_all()` (line 293)
**Signature:** `() → None`
**Flow (inside `_with_retry`):**
1. SELECT all table_names from metadata.
2. DROP TABLE for each.
3. DELETE all from `_cache_metadata`.

### `cache.py:CacheManager.get_stats()` (line 304)
**Signature:** `() → dict`
**Flow:** Aggregates `count(*)`, `sum(row_count)`, `sum(size_bytes)` from `_cache_metadata`. Returns `{table_count, total_rows, total_size_bytes, db_path}`.

### `cache.py:CacheManager.query(sql)` (line 317)
**Signature:** `(sql: str) → list[dict]`
**Purpose:** Validated read-only query for user/LLM-supplied SQL.
**Flow:** Delegates to `query_with_meta(sql)`, returns just the rows.

### `cache.py:CacheManager.query_with_meta(sql)` (line 328)
**Signature:** `(sql: str) → tuple[list[dict], list[dict]]`
**Purpose:** Validated query returning (rows, field_metadata).
**Flow:**
1. `_validate_sql(sql)` — rejects unsafe SQL.
2. Opens connection, `conn.execute(sql)`.
3. Extracts column names + types from `result.description`.
4. Fetches all rows, zips into dicts.
5. Returns `(rows, [{name, type}, ...])`.

### `cache.py:CacheManager.query_df(sql)` (line 345)
**Signature:** `(sql: str) → DataFrame`
**Flow:** Validates SQL, opens connection, `conn.execute(sql).fetchdf()`. Returns pandas DataFrame.

### `cache.py:CacheManager.update_expires_at(resource_id, expires_at)` (line 351)
**Signature:** `(resource_id: str, expires_at: datetime)`
**Flow (inside `_with_retry`):** `UPDATE _cache_metadata SET expires_at = ? WHERE resource_id = ?`.

### `cache.py:CacheManager.store_dataset_metadata(dataset_id, metadata)` (line 360)
**Signature:** `(dataset_id: str, metadata: dict)`
**Flow (inside `_with_retry`):** `INSERT OR REPLACE INTO _dataset_metadata` with JSON-serialized metadata and current timestamp.

### `cache.py:CacheManager.get_dataset_metadata(dataset_id)` (line 371)
**Signature:** `(dataset_id: str) → dict | None`
**Flow:** SELECT metadata JSON from `_dataset_metadata`, parses and returns, or `None`.

### `cache.py:CacheManager.execute_sql(sql, params)` (line 380)
**Signature:** `(sql: str, params=None) → list[tuple]`
**Purpose:** **Unvalidated** SQL for internal/programmatic use.
**Flow:** Opens connection, executes with params, returns raw tuples.

### `cache.py:CacheManager.execute_sql_dict(sql, params)` (line 391)
**Signature:** `(sql: str, params=None) → list[dict]`
**Purpose:** **Unvalidated** SQL returning dicts. Internal use.
**Flow:** Opens connection, executes, zips column names with rows into dicts.

### `cache.py:CacheManager.get_tables_metadata(table_names)` (line 402)
**Signature:** `(table_names: list[str]) → list[dict]`
**Flow:** Builds `IN (?, ?, ...)` query against `_cache_metadata`. Returns list of metadata dicts with keys: resource_id, dataset_id, table_name, downloaded_at, row_count, expires_at.

### `cache.py:CacheManager.get_resource_meta(resource_id)` (line 418)
**Signature:** `(resource_id: str) → dict | None`
**Flow:** SELECT full row from `_cache_metadata`. Parses `type_warnings` from JSON. Returns dict or `None`.

---

## 8. Utilities

### `utils.py:ResourceNotCachedError` (line 18)
**Type:** Exception. Raised when a tool requires cached data that doesn't exist.

### `utils.py:DatastoreNotAvailableError` (line 23)
**Type:** Exception. Raised when a resource has no datastore.

### `utils.py:SpatialExtensionError` (line 28)
**Type:** Exception. Raised when DuckDB spatial extension is unavailable.

### `utils.py:_lifespan_state(ctx)` (line 33)
**Signature:** `(ctx: Context) → dict`
**Flow:** Returns `ctx.lifespan_context` — the state dict yielded by `lifespan()`.

### `utils.py:get_deps(ctx, portal)` (line 37)
**Signature:** `(ctx: Context, portal: str) → tuple[CKANClient | ArcGISHubClient, CacheManager]`
**Purpose:** Dependency injection — lazily creates portal client on first use.
**Flow:**
1. Gets state via `_lifespan_state(ctx)`.
2. Validates `portal` exists in configs; raises `ValueError` if not.
3. If client not yet created for this portal:
   a. CKAN portals → `CKANClient(base_url, http_client)`.
   b. ArcGIS Hub portals → `ArcGISHubClient(base_url, http_client, org_name, org_title)`.
4. Returns `(client, cache)`.

### `utils.py:parse_portal_id(id_str, known_portals)` (line 72)
**Signature:** `(id_str: str, known_portals: set[str]) → tuple[str | None, str]`
**Purpose:** Splits `"portal:bare_id"` into portal key and bare ID.
**Flow:** If `:` in string and prefix matches a known portal → returns `(prefix, rest)`. Otherwise returns `(None, id_str)`.

### `utils.py:fan_out(ctx, portal, fn, *, first_match)` (line 81)
**Signature:** `async (ctx, portal, fn, first_match=False) → list[tuple[str, T | None, str | None]]`
**Purpose:** Runs async function across portals (parallel or sequential).
**Flow:**
- If `portal` specified → single-portal list.
- If `first_match=True` → sequential loop; returns on first success, collects errors on all-fail.
- If `first_match=False` → `asyncio.gather` all portals in parallel; wraps each in `_safe()` to catch exceptions.
- Returns `[(portal_key, result_or_None, error_or_None), ...]`.

### `utils.py:unwrap_first_match(results, bare_id, entity_type)` (line 127)
**Signature:** `(results, bare_id, entity_type) → tuple[str, T]`
**Purpose:** Extracts first success from `fan_out(first_match=True)`.
**Flow:** If first result has no error → returns `(portal, result)`. Otherwise joins all errors into a `ValueError`.

### `utils.py:resolve_dataset(ctx, dataset_id)` (line 150)
**Signature:** `async (ctx, dataset_id: str) → tuple[str, str, dict]`
**Purpose:** Resolves a possibly-bare dataset ID to `(portal, bare_id, dataset_dict)`.
**Flow:**
1. Calls `parse_portal_id()` to split prefix.
2. If portal known → directly calls `client.package_show(bare_id)`.
3. If no portal → calls `fan_out(first_match=True)` with `package_show` across all portals.
4. Uses `unwrap_first_match()` to extract result.

### `utils.py:resolve_resource_portal(ctx, resource_id)` (line 176)
**Signature:** `async (ctx, resource_id: str) → tuple[str, str]`
**Purpose:** Resolves a possibly-bare resource ID to `(portal, bare_id)`.
**Flow:**
1. If portal prefix exists → returns immediately.
2. Otherwise → `fan_out(first_match=True)` calling `client.resource_show(bare_id)` on each portal.
3. Returns first portal that responds successfully.

### `utils.py:get_cache(ctx)` (line 201)
**Signature:** `(ctx: Context) → CacheManager`
**Flow:** Returns `_lifespan_state(ctx)["cache"]`.

### `utils.py:strip_internal_fields(records)` (line 205)
**Signature:** `(records: list[dict]) → list[dict]`
**Flow:** Filters out keys starting with `_` (CKAN bookkeeping columns like `_id`, `_full_text`).

### `utils.py:_slugify(name, fallback, max_len)` (line 211)
**Signature:** `(name: str, fallback="unknown", max_len=40) → str`
**Flow:** Lowercases, replaces non-alphanumeric with `_`, collapses runs, strips leading/trailing `_`, truncates.

### `utils.py:infer_portal_from_table(table_name)` (line 218)
**Signature:** `(table_name: str) → str`
**Purpose:** Inverse of `make_table_name` — extracts portal from `ds_toronto_...` or `geo_ottawa_...`.
**Flow:** Splits on `_`, checks prefix (`ds`/`geo`) and second part against `PORTALS`. Falls back to `"ontario"`.

### `utils.py:make_table_name(dataset_name, resource_id, portal)` (line 230)
**Signature:** `(dataset_name: str, resource_id: str, portal="ontario") → str`
**Flow:** Returns `"ds_{portal}_{slugified_name}_{resource_id[:8]}"`.

### `utils.py:make_geo_table_name(dataset_name, resource_id, portal)` (line 237)
**Signature:** `(dataset_name: str, resource_id: str, portal="ontario") → str`
**Flow:** Returns `"geo_{portal}_{slugified_name}_{resource_id[:8]}"`. Distinguishes spatial tables.

### `utils.py:require_cached(cache, resource_id)` (line 244)
**Signature:** `(cache: CacheManager, resource_id: str) → str`
**Purpose:** Returns table name or raises `ResourceNotCachedError` with download hint.
**Flow:**
1. Strips portal prefix via `parse_portal_id()`.
2. Calls `cache.get_table_name(bare_id)`.
3. If `None` → raises `ResourceNotCachedError` with download command suggestion.

### `utils.py:json_response(**kwargs)` (line 261)
**Signature:** `(**kwargs) → str`
**Flow:** `json.dumps(kwargs, indent=2, default=str)`. Serializes with `str` fallback for non-JSON types.

---

## 9. Staleness

### `staleness.py:FREQUENCY_DAYS` (line 8)
**Type:** `dict[str, int]` — maps CKAN `update_frequency` values to expected update intervals in days.

### `staleness.py:compute_expires_at(downloaded_at, update_frequency)` (line 20)
**Signature:** `(downloaded_at: datetime, update_frequency: str | None) → datetime`
**Flow:** Looks up `update_frequency` in `FREQUENCY_DAYS` (default 30 days). Returns `downloaded_at + timedelta(days=days)`.

### `staleness.py:get_staleness_info(cache, resource_id)` (line 28)
**Signature:** `(cache: CacheManager, resource_id: str) → dict | None`
**Purpose:** Gets staleness status for a cached resource.
**Flow:**
1. Calls `cache.get_resource_meta(resource_id)` → returns `None` if not cached.
2. Extracts `downloaded_at` and `expires_at` from metadata.
3. If `expires_at` is `None` → defaults to `downloaded_at + 30 days`.
4. Ensures timezone-aware comparisons.
5. Returns `{resource_id, downloaded_at, expires_at, is_stale: bool, age_hours: float}`.

---

## 10. Formatting

### `formatting.py:_escape_cell(value)` (line 12)
**Signature:** `(value: Any) → str`
**Flow:** Stringifies value, escapes `|` → `\|` and `\n` → ` ` for markdown table safety.

### `formatting.py:md_table(headers, rows)` (line 20)
**Signature:** `(headers: list[str], rows: list[list[Any]]) → str`
**Purpose:** Builds a plain markdown table.
**Flow:**
1. Pads short rows / truncates long rows to match header count.
2. Escapes all cells via `_escape_cell()`.
3. Builds header line, separator line (`---`), and data lines.
4. Joins with `\n`.

### `formatting.py:md_response(**kwargs)` (line 40)
**Signature:** `(**kwargs: Any) → str`
**Purpose:** Drop-in replacement for `json_response()` — renders kwargs as markdown.
**Flow:** For each kwarg:
- `list[dict]` → markdown table via `md_table()`.
- `dict` → nested bullet list.
- `list` → comma-separated inline.
- scalar → `- **key:** value`.

### `formatting.py:format_records(records, row_count, *, source, total, preview, fields)` (line 80)
**Signature:** `(records, row_count, *, source, total, preview, fields) → str`
**Purpose:** Formats tabular query results with count header + markdown table.
**Flow:**
1. Builds header line: row count, total, source table info.
2. If `fields` → adds schema line listing column types.
3. If records exist → builds table via `md_table()`.
4. Joins parts.

### `formatting.py:format_search_results(query, portals_searched, results, skipped)` (line 124)
**Signature:** `(query, portals_searched, results, skipped) → str`
**Purpose:** Formats search results grouped by portal.
**Flow:**
1. Counts total returned vs total available.
2. Builds header with count summary.
3. For each portal result: prints portal header + bullet list of datasets with formats and org.
4. For skipped portals: prints skip reason.

---

## 11. CLI

### `cli.py:_muted(text)` (line 19)
**Signature:** `(text: str) → str`
**Flow:** Wraps text in ANSI gray escape codes for dim terminal output.

### `cli.py:_make_cache(quiet)` (line 23)
**Signature:** `(quiet: bool) → CacheManager`
**Flow:** Creates and initializes `CacheManager()`. Prints db path (dimmed) unless quiet.

### `cli.py:_human_size(nbytes)` (line 31)
**Signature:** `(nbytes: int) → str`
**Flow:** Converts bytes to human-readable string (B, KB, MB, GB, TB).

### `cli.py:_print_table(headers, rows)` (line 39)
**Signature:** `(headers: list[str], rows: list[list[str]]) → None`
**Flow:** Calculates column widths, prints aligned header + separator + data rows.

### `cli.py:cmd_list(args)` (line 51)
**Signature:** `(args: Namespace) → None`
**Purpose:** `ontario-data-mcp cache list` command.
**Flow:**
1. Creates cache, lists all cached resources.
2. If empty → prints "Cache is empty."
3. If `--json` → enriches with staleness info, prints JSON.
4. Otherwise → prints formatted table with resource_id, table_name, rows, size, downloaded_at, stale status.

### `cli.py:cmd_stats(args)` (line 83)
**Signature:** `(args: Namespace) → None`
**Purpose:** `ontario-data-mcp cache stats` command.
**Flow:** Gets cache stats, prints as JSON or human-readable (tables, total rows, total size).

### `cli.py:cmd_query(args)` (line 96)
**Signature:** `(args: Namespace) → None`
**Purpose:** `ontario-data-mcp cache query <sql>` command.
**Flow:** Runs validated SQL via `cache.query()`. Prints results as table or "No results."

### `cli.py:cmd_remove(args)` (line 116)
**Signature:** `(args: Namespace) → None`
**Purpose:** `ontario-data-mcp cache remove <resource_id>` command.
**Flow:** Strips portal prefix, checks if cached, calls `cache.remove_resource()`.

### `cli.py:cmd_clear(args)` (line 127)
**Signature:** `(args: Namespace) → None`
**Purpose:** `ontario-data-mcp cache clear` command.
**Flow:** Lists cached, prompts for confirmation (unless `--yes`), calls `cache.remove_all()`.

### `cli.py:cmd_refresh(args)` (line 144)
**Signature:** `(args: Namespace) → None`
**Purpose:** `ontario-data-mcp cache refresh <resource_id>` command.
**Flow:**
1. Strips portal prefix, checks if cached.
2. Gets resource metadata, infers portal from table name.
3. Async inner function `_do_refresh()`:
   a. Creates httpx client with 120s timeout.
   b. If ArcGIS portal → calls `_download_arcgis_resource_data()`.
   c. If CKAN portal → calls `_download_resource_data()`.
   d. Calls `cache.store_resource()` to upsert.
   e. Computes and sets `expires_at`.
4. Runs async function via `asyncio.run()`.

### `cli.py:build_parser()` (line 202)
**Signature:** `() → argparse.ArgumentParser`
**Flow:** Builds argparse parser with subcommands: `list`, `stats`, `query`, `remove`, `clear`, `refresh`. Each sets a `func` default.

### `cli.py:run(argv)` (line 236)
**Signature:** `(argv: list[str] | None) → None`
**Flow:** Parses args via `build_parser()`, calls `args.func(args)`.

---

## 12. Tools — Discovery

### `discovery.py:search_datasets(query, organization, resource_format, update_frequency, sort_by, limit, portal, ctx)` (line 20)
**Signature:** MCP tool, async
**Purpose:** Searches datasets across all portals. Primary entry point for data exploration.
**Flow:**
1. Builds CKAN filters from optional params (organization, resource_format, update_frequency).
2. Defines `_search_one(portal_key)`:
   a. Gets client via `get_deps()`.
   b. Calls `client.package_search(query, filters, sort, rows)`.
   c. Extracts: id (portal-prefixed), name, title, organization, description, formats, num_resources, last_modified, update_frequency.
   d. Returns portal result dict with total_count and datasets.
3. Calls `fan_out(ctx, portal, _search_one)` — parallel across all portals (or single if specified).
4. Separates successful results from skipped (errored) portals.
5. Formats via `format_search_results()` → markdown.

### `discovery.py:list_organizations(include_counts, portal, ctx)` (line 102)
**Signature:** MCP tool, async
**Purpose:** Lists government orgs with dataset counts across portals.
**Flow:**
1. Defines `_list_orgs(portal_key)` → calls `client.organization_list(all_fields=True)`, extracts org details, sorts by dataset_count desc.
2. `fan_out()` across portals.
3. Merges all org lists.
4. Returns `md_response(organizations=all_orgs)`.

### `discovery.py:list_topics(query, portal, limit, ctx)` (line 138)
**Signature:** MCP tool, async
**Purpose:** Lists top tags/topics ranked by dataset count.
**Flow:**
1. Defines `_list_tags(portal_key)` → calls `client.tag_list(query, all_fields=True)`.
2. Handles two response shapes: list of dicts or list of strings.
3. `fan_out()`, merges, sorts by count desc, truncates to limit.
4. Returns `md_response(summary=..., topics=...)`.

### `discovery.py:find_related_datasets(dataset_id, limit, ctx)` (line 175)
**Signature:** MCP tool, async
**Purpose:** Finds datasets related by shared tags and organization.
**Flow:**
1. Resolves dataset via `resolve_dataset()`.
2. Gets client for that portal.
3. Searches by top 5 tags (OR query) → collects related datasets with shared_tags.
4. Searches by organization → adds datasets not already seen.
5. Returns `md_response(source=..., related=...)`.

### `discovery.py:list_portals(ctx)` (line 228)
**Signature:** MCP tool, async
**Purpose:** Lists all available data portals.
**Flow:** Reads `portal_configs` from lifespan state. Builds list of portal dicts. Returns `md_response(portals=...)`.

---

## 13. Tools — Metadata

### `metadata.py:get_dataset_info(dataset_id, ctx)` (line 19)
**Signature:** MCP tool, async
**Purpose:** Gets full dataset metadata including all resources.
**Flow:**
1. Resolves dataset via `resolve_dataset()`.
2. Caches dataset metadata via `cache.store_dataset_metadata()`.
3. Extracts resource details: id, name, format, size, url, last_modified, datastore_active.
4. Returns `md_response(id, name, title, description, organization, tags, update_frequency, resources, ...)`.

### `metadata.py:list_resources(dataset_id, ctx)` (line 62)
**Signature:** MCP tool, async
**Purpose:** Lists all resources (files) in a dataset.
**Flow:**
1. Resolves dataset.
2. Extracts resource details including `data_range`.
3. Returns `md_response(dataset, dataset_id, num_resources, resources)`.

### `metadata.py:get_resource_schema(resource_id, sample_size, ctx)` (line 94)
**Signature:** MCP tool, async
**Purpose:** Gets column schema and sample values for a datastore resource.
**Flow:**
1. Parses portal prefix.
2. If ArcGIS Hub → returns not-available message (no remote schema API).
3. Defines `_schema(pk)`:
   a. Calls `client.resource_show(bare_id)`.
   b. If not `datastore_active` → returns hint to download first.
   c. Calls `client.datastore_search(bare_id, limit=sample_size)`.
   d. Extracts fields (skipping `_`-prefixed), samples values.
   e. Returns schema dict.
4. If portal known → direct call. Otherwise → `fan_out(first_match=True)`.
5. Returns `md_response(**data)`.

### `metadata.py:compare_datasets(dataset_ids, ctx)` (line 151)
**Signature:** MCP tool, async
**Purpose:** Side-by-side metadata comparison for 2-5 datasets (cross-portal capable).
**Flow:**
1. For each dataset_id (up to 5):
   a. Resolves via `resolve_dataset()`.
   b. Extracts: id, title, organization, num_resources, formats, update_frequency, last_modified, tags, license, geographic_coverage.
2. Computes `shared_tags` (intersection of all tag sets).
3. Returns `md_response(datasets=..., shared_tags=...)`.

### `metadata.py:check_freshness(dataset_id, ctx)` (line 116 in quality.py)
**Signature:** MCP tool, async
**Purpose:** Checks if a dataset is current vs its declared update frequency.
**Flow:**
1. Resolves dataset.
2. Parses `metadata_modified` to datetime, computes days since update.
3. Looks up expected interval from `FREQUENCY_DAYS`.
4. Determines staleness: `days_since_update > expected`.
5. Collects per-resource freshness info.
6. Returns `md_response(dataset, update_frequency, last_modified, days_since_update, is_stale, resources)`.

---

## 14. Tools — Retrieval

### `retrieval.py:_download_resource_data(ckan, resource_id)` (line 29)
**Signature:** `async (ckan: CKANClient, resource_id: str) → tuple[DataFrame, dict, dict]`
**Purpose:** Downloads resource data from CKAN, preferring datastore API.
**Flow:**
1. Calls `ckan.resource_show(resource_id)` → gets resource metadata.
2. Calls `ckan.package_show(dataset_id)` → gets dataset metadata.
3. If `datastore_active`:
   a. Calls `ckan.datastore_search_all(resource_id)` → all records.
   b. Creates DataFrame, drops `_`-prefixed columns.
4. Otherwise → direct file download:
   a. `httpx.AsyncClient(timeout=120s).get(url)`.
   b. Format-specific parsing: CSV/TXT → `pd.read_csv`, XLS/XLSX → `pd.read_excel`, JSON → `pd.read_json`, GEOJSON → `gpd.read_file` with geometry → WKT conversion.
5. Returns `(df, resource_dict, dataset_dict)`.

### `retrieval.py:_download_arcgis_resource_data(client, resource_id, http_client)` (line 75)
**Signature:** `async (client, resource_id, http_client) → tuple[DataFrame, dict, dict]`
**Purpose:** Downloads ArcGIS Hub resource via Downloads API (bulk CSV).
**Flow:**
1. Calls `client.package_show(resource_id)` → dataset metadata.
2. Calls `client.get_download_url(resource_id, fmt="csv")` → CSV URL.
3. If URL found → downloads with 120s timeout, parses CSV to DataFrame.
4. If no URL → raises `ValueError` with portal link.
5. Returns `(df, synthesized_resource_dict, dataset_dict)`.

### `retrieval.py:download_resource(resource_id, ctx)` (line 107)
**Signature:** MCP tool, async
**Purpose:** Downloads and caches a resource in DuckDB. Core workflow tool.
**Flow:**
1. Parses portal prefix.
2. If already cached:
   a. Gets table_name, metadata, staleness info.
   b. Returns `md_response(status="already_cached", ...)` with type_warnings if any.
3. If portal unknown → `resolve_resource_portal()`.
4. Gets client via `get_deps()`.
5. Reports progress (0%).
6. Dispatches download:
   a. ArcGIS → `_download_arcgis_resource_data()`.
   b. CKAN → `_download_resource_data()`.
7. Reports progress (70%).
8. Builds table name via `make_table_name()`.
9. Calls `cache.store_resource()` → ingests DataFrame.
10. Calls `cache.store_dataset_metadata()` → caches dataset metadata.
11. Computes `expires_at` via `compute_expires_at()`, sets via `cache.update_expires_at()`.
12. Reports progress (100%).
13. Returns `md_response(status="downloaded", table_name, row_count, columns, dtypes, type_warnings, hint)`.

### `retrieval.py:cache_info(ctx)` (line 200)
**Signature:** MCP tool, async
**Purpose:** Returns cache statistics and all cached datasets.
**Flow:**
1. Gets stats + list of cached resources.
2. Enriches each with staleness info and size_mb.
3. Returns `md_response(**stats, total_size_mb, datasets)`.

### `retrieval.py:cache_manage(action, resource_id, ctx)` (line 233)
**Signature:** MCP tool, async
**Purpose:** Cache management — remove single or clear all.
**Flow:**
- `action="remove"` → strips portal prefix, calls `cache.remove_resource()`.
- `action="clear"` → counts cached, calls `cache.remove_all()`.
- Invalid action → raises `ValueError`.

### `retrieval.py:refresh_cache(resource_id, ctx)` (line 264)
**Signature:** MCP tool, async
**Purpose:** Re-downloads cached resources for fresh data.
**Flow:**
1. If `resource_id` specified → filters to single resource.
2. For each cached resource:
   a. Reports progress.
   b. Infers portal from table name.
   c. Dispatches download (ArcGIS or CKAN).
   d. Re-stores via `cache.store_resource()`.
   e. Updates `expires_at`.
   f. On error → records error in results.
3. Returns `md_response(refreshed=results)`.

---

## 15. Tools — Querying

### `querying.py:_detect_numeric_varchars(fields, rows)` (line 33)
**Signature:** `(fields: list[dict], rows: list[dict]) → list[str]`
**Purpose:** Detects VARCHAR fields with >80% numeric values from query results.
**Flow:** Same algorithm as `CacheManager._detect_numeric_varchars` but operates on query result dicts instead of DuckDB columns.

### `querying.py:_generate_query_warnings(sql, results, fields, cache)` (line 55)
**Signature:** `(sql, results, fields, cache) → list[str]`
**Purpose:** Post-query heuristic warnings to catch common accuracy traps.
**Flow:**
1. Extracts table names from SQL via regex `FROM "([^"]+)"`.
2. **COUNT(*) warning:** If SQL uses `COUNT(*)` and table has quantity-like columns (count, quantity, number, total, amount...) → warns to use `SUM()` instead.
3. **Empty result warning:** If 0 rows but table is non-empty → warns about WHERE/JOIN conditions.
4. **Low GROUP BY cardinality:** If 1-3 groups from >1000 rows → warns about GROUP BY columns.

### `querying.py:query_resource(resource_id, filters, fields, sort, limit, offset, ctx)` (line 112)
**Signature:** MCP tool, async
**Purpose:** Queries a resource via CKAN Datastore API (remote, no download needed).
**Flow:**
1. Parses portal prefix.
2. If ArcGIS Hub → returns not-available message.
3. Defines `_query(pk)` → `client.datastore_search(bare_id, filters, fields, sort, limit, offset)`.
4. If portal known → direct call. Otherwise → `fan_out(first_match=True)`.
5. Extracts field_info (skipping `_`-prefixed), strips internal fields from records.
6. Returns `format_records(clean_records, row_count, total, fields)`.

### `querying.py:sql_query(sql, portal, ctx)` (line 164)
**Signature:** MCP tool, async
**Purpose:** Runs SQL against CKAN Datastore (remote). One-off queries only.
**Flow:**
1. Gets client for specified portal.
2. If ArcGIS Hub → returns not-available message.
3. Calls `client.datastore_sql(sql)`.
4. Strips internal fields.
5. Returns `format_records(clean_records, row_count, fields)`.

### `querying.py:query_cached(sql, ctx)` (line 196)
**Signature:** MCP tool, async
**Purpose:** Runs SQL against locally cached DuckDB data. Primary analysis tool.
**Flow:**
1. Gets cache via `get_cache()`.
2. Try block:
   a. `cache.query_with_meta(sql)` → validates SQL, executes, returns `(results, fields)`.
   b. **Truncation:** If `>= MAX_QUERY_ROWS (2000)` → counts full result, truncates to 2000.
   c. **Numeric VARCHAR detection:** `_detect_numeric_varchars(fields, results)`.
   d. **Post-query warnings:** `_generate_query_warnings(sql, results, fields, cache)`.
   e. Builds response parts:
      - SQL echo: `` **Query:** `{sql}` ``
      - Results table via `format_records()`.
      - Truncation warning if applicable.
      - All warnings prefixed with `⚠`.
      - **Data provenance:** Looks up table metadata (resource_id, downloaded_at, staleness status).
3. Except block: On error, lists available tables + contextual hints (type casting, quoting).

### `querying.py:preview_data(resource_id, rows, ctx)` (line 317)
**Signature:** MCP tool, async
**Purpose:** Quick preview of first N rows (remote, no download).
**Flow:**
1. Parses portal prefix.
2. If ArcGIS Hub → returns not-available message.
3. Defines `_preview(pk)` → `client.datastore_search(bare_id, limit)`.
4. Direct call or fan_out.
5. Returns `format_records(clean_records, row_count, total, preview=True, fields)`.

---

## 16. Tools — Quality

### `quality.py:check_data_quality(resource_id, ctx)` (line 21)
**Signature:** MCP tool, async
**Purpose:** Comprehensive data quality analysis on cached data.
**Flow:**
1. Gets cache + table name via `require_cached()`.
2. Gets total row count.
3. `DESCRIBE "{table_name}"` → column info.
4. For each column:
   a. Counts NULLs → `null_count`, `null_pct`.
   b. Counts distinct values → `distinct_count`, `cardinality_pct`.
   c. If numeric type (INT, FLOAT, DOUBLE, DECIMAL, NUMERIC) → computes `min`, `max`, `avg`, `stddev`.
   d. If VARCHAR → samples 100 values, checks if >80% numeric:
      - Adds `type_note` warning.
      - Computes numeric stats via `TRY_CAST(col AS DOUBLE)`.
5. Duplicate row check via window function `COUNT(*) OVER (PARTITION BY all_columns)`.
6. Returns `md_response(resource_id, table_name, total_rows, duplicate_rows, columns=quality_report)`.

### `quality.py:check_freshness(dataset_id, ctx)` (line 116)
**Signature:** MCP tool, async
**Purpose:** Checks dataset currency against declared update frequency.
**Flow:**
1. Resolves dataset via `resolve_dataset()`.
2. Parses `metadata_modified` → computes `days_since_update`.
3. Looks up expected interval from `FREQUENCY_DAYS`.
4. Computes `is_stale = days_since_update > expected`.
5. Collects per-resource last_modified info.
6. Returns `md_response(dataset, update_frequency, last_modified, days_since_update, is_stale, resources)`.

### `quality.py:profile_data(resource_id, ctx)` (line 162)
**Signature:** MCP tool, async
**Purpose:** Statistical profile using DuckDB `SUMMARIZE`.
**Flow:**
1. Gets cache + table name via `require_cached()`.
2. Runs `SUMMARIZE SELECT * FROM "{table_name}"` via `cache.execute_sql_dict()`.
3. Gets row count.
4. Returns `md_response(resource_id, table_name, row_count, columns=summary)`.

---

## 17. Tools — Geospatial

### `geospatial.py:load_geodata(resource_id, force_refresh, ctx)` (line 27)
**Signature:** MCP tool, async
**Purpose:** Downloads and caches geospatial data (SHP, KML, GeoJSON) with spatial support.
**Flow:**
1. Parses portal prefix.
2. If ArcGIS Hub → returns not-available message.
3. If cached and no force_refresh → returns already_cached.
4. Resolves resource portal if needed.
5. Gets resource + dataset metadata.
6. Downloads file with 120s timeout.
7. Format-specific parsing:
   a. GEOJSON → `gpd.read_file(driver="GeoJSON")`.
   b. KML → `gpd.read_file(driver="KML")`.
   c. SHP/ZIP → extracts to temp dir, `gpd.read_file(tmpdir)`.
8. Converts geometry to WKT, extracts geometry_type, CRS, bounding box.
9. Drops `geometry` column, stores as regular DataFrame.
10. Builds geo table name via `make_geo_table_name()`.
11. Stores via `cache.store_resource()`.
12. Returns `md_response(status, table_name, row_count, columns, geometry_types, bounds, crs, hint)`.

### `geospatial.py:spatial_query(resource_id, operation, latitude, longitude, radius_km, bbox, limit, ctx)` (line 128)
**Signature:** MCP tool, async
**Purpose:** Runs spatial queries against cached geospatial data.
**Flow:**
1. Gets cache + table name via `require_cached()`.
2. Checks `cache.has_spatial_extension` → raises `SpatialExtensionError` if not.
3. **Validates coordinates:**
   - Latitude: -90 to 90.
   - Longitude: -180 to 180.
   - Radius: must be > 0.
   - Bbox: must have 4 values within valid ranges.
4. Builds parameterized SQL based on operation:
   a. `"contains_point"` → `ST_Contains(geom, ST_Point(?, ?))` with distance.
   b. `"within_radius"` → `ST_DWithin(geom, point, degree_radius)` with `distance_km`, ordered by distance.
   c. `"within_bbox"` → `ST_Intersects(geom, ST_MakeEnvelope(?, ?, ?, ?))`.
5. Executes via `cache.execute_sql_dict(sql, params)`.
6. If no results → provides context (table has N features but none match query).
7. Returns `format_records(records, row_count)`.

### `geospatial.py:list_geo_datasets(format_filter, limit, portal, ctx)` (line 244)
**Signature:** MCP tool, async
**Purpose:** Finds datasets containing geospatial resources.
**Flow:**
1. Builds format list: specific filter or `["SHP", "KML", "GEOJSON"]`.
2. Defines `_list_geo(portal_key)`:
   a. For each geo format → `client.package_search(filters={"res_format": fmt})`.
   b. Deduplicates by dataset ID.
   c. Extracts geo resources (format in SHP/KML/GEOJSON/ZIP).
3. `fan_out()` across portals.
4. Merges results, truncates to limit.
5. Returns `md_response(total, datasets)`.

---

## 18. Prompts

### `prompts.py:_format_cached_context(cache)` (line 14)
**Signature:** `(cache: CacheManager) → str`
**Flow:** Lists up to 10 cached datasets with table_name, row_count, downloaded_at. Returns formatted string or empty.

### `prompts.py:_get_topic_context(ctx, topic)` (line 24)
**Signature:** `async (ctx, topic: str) → str`
**Purpose:** Pre-fetches search results from each portal for prompt context.
**Flow:** For each portal: calls `client.package_search(query=topic, rows=5)`, extracts top titles. Handles errors gracefully.

### `prompts.py:explore_topic(topic, ctx)` (line 46)
**Signature:** MCP prompt, async
**Purpose:** Guided topic exploration across all portals.
**Flow:**
1. Gets cache context + topic context (pre-fetched search results).
2. Returns `Message(role="user")` with exploration instructions: search, summarize, get details, suggest downloads.

### `prompts.py:data_investigation(dataset_id, ctx)` (line 74)
**Signature:** MCP prompt, async
**Purpose:** Deep investigation workflow for a specific dataset.
**Flow:**
1. Resolves dataset title (tolerates errors).
2. Gets cache context.
3. Returns `Message` with step-by-step investigation workflow: get_dataset_info → list_resources → get_resource_schema → download → check_data_quality → profile_data → insights.

### `prompts.py:compare_data(dataset_ids, ctx)` (line 112)
**Signature:** MCP prompt, async
**Purpose:** Side-by-side analysis of multiple datasets.
**Flow:**
1. Splits comma-separated IDs.
2. Gets cache context.
3. Returns `Message` with comparison workflow: compare_datasets → download → profile → query_cached for joins → summarize.

---

## 19. Resources

### `resources.py:cache_index(ctx)` (line 12)
**Signature:** MCP resource `ontario://cache/index`, async
**Purpose:** Lists all cached datasets with freshness info.
**Flow:** Gets stats + cached list from cache. Returns JSON with total_cached, total_rows, total_size_mb, datasets.

### `resources.py:dataset_metadata(dataset_id, ctx)` (line 26)
**Signature:** MCP resource `ontario://dataset/{dataset_id}`, async
**Purpose:** Full metadata for a specific dataset.
**Flow:**
1. Checks local cache first via `cache.get_dataset_metadata()`.
2. If miss → resolves via `resolve_dataset()`, caches under both canonical_id and bare_id.
3. Returns JSON.

### `resources.py:portal_stats(ctx)` (line 43)
**Signature:** MCP resource `ontario://portal/stats`, async
**Purpose:** Overview statistics across all portals.
**Flow:** For each portal: searches (rows=0) for total count, lists top 5 orgs. Returns JSON.

### `resources.py:schema_resource(table_name, ctx)` (line 73)
**Signature:** MCP resource `ontario://schema/{table_name}`, async
**Purpose:** Column schema, types, samples, and type warnings for a cached table.
**Flow:**
1. `DESCRIBE "{table_name}"` for columns.
2. `SELECT * LIMIT 3` for samples.
3. Regex-checks VARCHAR columns for numeric values → type_warnings.
4. Returns JSON with fields (name, type, sample_values, type_warning) and hints.

### `resources.py:duckdb_sql_guide()` (line 123)
**Signature:** MCP resource `ontario://guides/duckdb-sql`, async
**Purpose:** Static DuckDB SQL reference for Ontario data analysis.
**Flow:** Returns JSON with title, reference URL, and tips array.

---

## 20. Frontend — carousel.js

### `carousel.js:esc(s)` (line 8)
**Signature:** `(s: string) → string`
**Flow:** Creates a div, sets `textContent`, reads back `innerHTML`. HTML-escapes any string safely.

### `carousel.js:renderSources(sources)` (line 13)
**Signature:** `(sources: array) → string`
**Flow:** Builds "Source: <link>" or "Sources: <link>, <link>" HTML from sources array with optional org.

### `carousel.js:renderSteps(steps)` (line 25)
**Signature:** `(steps: array) → string`
**Flow:** Maps each step to `<div class="hood-step"><span class="tool-name">tool</span> → description</div>`.

### `carousel.js:renderHood(hood)` (line 33)
**Signature:** `(hood: object) → string`
**Purpose:** Renders "Under the hood" expandable section.
**Flow:** Builds `<details><summary>` with steps via `renderSteps()` and SQL via `highlightSQL()`.

### `carousel.js:renderFollowup(fu)` (line 43)
**Signature:** `(fu: object) → string`
**Purpose:** Renders follow-up question/answer as expandable `<details>`.
**Flow:** Builds question summary, answer with optional punchline bold, and sources.

### `carousel.js:renderCard(ex)` (line 58)
**Signature:** `(ex: object) → string`
**Purpose:** Renders a single example card.
**Flow:** Builds div with tag, question, answer+punchline, sources, hood, followup.

### `carousel.js:render(examples)` (line 72)
**Signature:** `(examples: array) → void`
**Purpose:** Injects all cards into the carousel track.
**Flow:**
1. Finds `.carousel-track` element.
2. Sets `innerHTML` to all cards joined.
3. Sets ARIA attributes for accessibility.
4. Shows carousel section.

### `carousel.js:init()` (line 85)
**Signature:** `() → void`
**Flow:** Fetches `examples.json`, calls `render()`. Silently ignores fetch failures.

### IIFE wrapper (lines 92-97)
**Flow:** If DOM still loading → adds `DOMContentLoaded` listener. Otherwise calls `init()` immediately.

---

## 21. Frontend — sql-highlight.js

### `sql-highlight.js:esc(ch)` (line 31)
**Signature:** `(ch: string) → string`
**Flow:** HTML-escapes `&`, `<`, `>`, `"`.

### `sql-highlight.js:escStr(s)` (line 41)
**Signature:** `(s: string) → string`
**Flow:** Replaces all `&<>"` in a string using `esc()`.

### `sql-highlight.js:highlightSQL(sql)` (line 45)
**Signature:** `(sql: string) → string`
**Purpose:** State-machine SQL syntax highlighter producing HTML with span classes.
**Flow:** Character-by-character scan:
1. `--` line comments → `<span class="comment">`.
2. `'...'` string literals (handles `''` escape) → `<span class="str">`.
3. Numeric literals (digits, optional dot) → `<span class="num">`.
4. Words → checked against `KEYWORDS` set → `<span class="kw">`, or `FUNCTIONS` set → `<span class="fn">`, or plain text.
5. Everything else (operators, parens, whitespace) → escaped directly.
6. Returns complete HTML string.

**Exported as:** `root.highlightSQL = highlightSQL` (global function, used by `carousel.js:renderHood()`).

### Module constants
- `KEYWORDS`: 72 SQL keywords (SELECT, FROM, WHERE, JOIN, WITH, ...).
- `FUNCTIONS`: 46 SQL functions (AVG, SUM, TRY_CAST, ST_CONTAINS, ...).

---

## Cross-Cutting Patterns

### Dependency Injection
All MCP tools receive `ctx: Context`. State flows:
```
lifespan() → yields {http_client, portal_configs, portal_clients, cache}
  → _lifespan_state(ctx) → state dict
  → get_deps(ctx, portal) → (client, cache)  [lazy client creation]
  → get_cache(ctx) → cache
```

### Portal Routing
```
parse_portal_id("toronto:abc123") → ("toronto", "abc123")
fan_out(ctx, portal=None, fn) → runs fn on all portals (parallel)
fan_out(ctx, portal=None, fn, first_match=True) → runs fn sequentially, stops on first success
resolve_dataset(ctx, "abc123") → tries all portals, returns (portal, bare_id, dataset_dict)
```

### Cache Workflow (preferred path)
```
search_datasets(query) → portal-prefixed IDs
  → download_resource(resource_id) → stores in DuckDB, returns table_name
    → query_cached(sql) → validates SQL, executes on DuckDB, returns markdown
```

### SQL Safety Chain
```
User SQL → _validate_sql()
  → strip comments
  → _has_semicolons_outside_strings() → reject injection
  → check first word against _ALLOWED_PREFIXES → reject mutations
  → execute read-only query
```

### Response Formatting
```
Old: json_response(**kwargs) → JSON string
New: md_response(**kwargs) → markdown (tables, bullets, inline)
     format_records(records, ...) → markdown table with count header
     format_search_results(query, ...) → portal-grouped markdown
```
