# Turnstone — Progress Tracker

## Backlog

### Infrastructure

- [x] **Copilot review fixes** — ARIA `role="list"` fix and `renderStatusBar` cache key fix (included in PR #1 merge)
- [x] **Test coverage reporting** — pytest-cov + coverage config in pyproject.toml, CI uploads XML artifacts. Baseline: 41% (482 tests pass)
- [x] **PyPI trusted publisher** — Trusted publisher configured on pypi.org. Validate with v0.3.0 tag push after PR #10 merges
- [x] **Pre-commit hooks** — `.pre-commit-config.yaml` with ruff (check + format) and mypy hooks. Install with `pre-commit install`

### Features

- [x] **Streaming tool output** — Bash tool now uses Popen with line-by-line stdout streaming via `on_tool_output_chunk` SSE events; web UI renders incrementally with pulsing border
- [x] **Streaming call_id routing** — `call_id` now included in `tool_info`/`approve_request`/`tool_result` SSE payloads; `data-call-id` set on `.approval-tool` DOM elements; streaming chunks and final results route to correct tool div during parallel execution
- [x] **Session persistence/resume** — Config persistence (temperature, reasoning_effort, etc.), interrupted session repair, prune_sessions test coverage
- [x] **Console → server deep linking** — Clickable workstream rows open server UI with `?ws_id=` for auto-selection; URL cleaned after navigation; external-link indicator on hover; `server_url` injected into workstream dicts by collector
- [x] **Tool extensibility / MCP** — MCP client manager with background asyncio event loop; connects to stdio and HTTP MCP servers; tool schemas converted to OpenAI format with `mcp__{server}__{tool}` prefixing; `--mcp-config` CLI/TOML/JSON config; `/mcp` introspection command; MCP tools available in main session and task agents
- [x] **Multi-model support** — ModelRegistry with named model configs (`[models.*]` in config.toml), per-workstream model selection (`/model <alias>`, API `model` param), fallback routing chain, optional agent model override; MODEL column in both dashboard UIs; 43 new tests

### Operational

- [x] **Prometheus /metrics verification** — 16 metrics fully wired: 11 original + 5 new (`turnstone_sse_connections_active`, `turnstone_ratelimit_rejected_total`, `turnstone_backend_up`, `turnstone_circuit_state`, `turnstone_workstreams_evicted_total`); per-workstream metrics labeled by `ws_id`; comprehensive tests in `test_server_live.py`
- [x] **Graceful health degradation** — `BackendHealthMonitor` probes `client.models.list()` on a background thread; circuit breaker state machine (CLOSED/OPEN/HALF_OPEN); `/health` returns `"status": "degraded"` + `"backend"` field when LLM unreachable; Docker healthcheck accepts degraded; session `_create_stream_with_retry` fast-fails on open circuit; both UIs show degradation indicator; 15 new tests
- [x] **Rate limiting / backpressure** — Token-bucket per-IP rate limiter (`turnstone/core/ratelimit.py`); 429 + `Retry-After` responses; `/health` and `/metrics` exempt; configurable via `[ratelimit]` config; workstream auto-eviction of oldest idle at capacity (`max_workstreams` configurable); both UIs handle 429 with retry + toast, show eviction notifications; 10 new tests

### Roadmap Phase 1 — MVP (Early Adopters)
_Core platform hardening and the minimum feature set to onboard first paying users._

- [x] **HTTP server → Starlette/ASGI migration** — Replace stdlib `http.server` with Starlette/uvicorn; enables proper SSE via `sse-starlette`, async request handling, and `httpx-sse` for bridge SSE consumption
- [x] **Multi-provider API support** — Abstract the LLM client layer to support OpenAI Responses API and Anthropic API natively alongside the current Chat Completions API. Currently all interaction goes through the `openai` SDK; Responses API uses a different request/response shape (stateful, built-in tool execution), and Anthropic has its own message format and tool_use blocks. Requires a provider adapter layer in front of `ChatSession`
- [x] **Provider-native web search** — Replace Tavily-based `web_search` tool with provider-native search: Anthropic's `web_search_20250305` server-side tool and OpenAI's `web_search_options` / search-model integration. Eliminates the Tavily API key dependency, uses provider domain filtering, and keeps search execution server-side. Requires handling new response content block types (`web_search_tool_result` for Anthropic, `url_citation` annotations for OpenAI) in `_stream_response` and message history. Falls back to Tavily for local/vLLM providers that lack native search
- [x] **Console workstream creation + server proxy** — Console dashboard now supports creating workstreams on specific nodes (or auto-selected by headroom) via MQ dispatch through the existing bridge pipeline. Reverse proxy at `/node/{node_id}/` serves the server UI through the console port using JS shim injection (overrides `fetch()`/`EventSource()` to route through proxy) and HTML rewriting. SSE streaming proxied via `httpx.AsyncClient`. Console-return banner injected at top of proxied pages. 22 new tests
- [x] **Database abstraction** — `turnstone/core/storage/` package with `StorageBackend` protocol (21 methods), SQLAlchemy Core schema (`_schema.py` — single source of truth), SQLite backend (`_sqlite.py` — FTS5 search, message reconstruction, incomplete turn repair), PostgreSQL backend (`_postgresql.py` — tsvector/ILIKE search, connection pooling), singleton registry (`_registry.py` — `init_storage`/`get_storage`/`reset_storage`), and programmatic Alembic migrations (`_migrate.py` + `migrations/001_initial_schema.py`). `memory.py` reduced from 647→190 lines as thin facade. `get_tavily_key()` moved to `config.py`. Session.py `open_db()` calls replaced with generic KV methods (`kv_set`/`kv_delete`/`kv_search`). `[database]` config section with env var support (`TURNSTONE_DB_BACKEND`, `TURNSTONE_DB_URL`). 39 new tests (934 total)
- [x] **User identity and API tokens** — JWT-based auth architecture with three token types: config-file tokens (hmac, backward-compat), API tokens (`ts_` prefix, SHA-256 hashed, DB-backed), and JWTs (HS256, 24h expiry, validated locally). Username:password login via bcrypt. Hierarchical scopes: `read` < `write` < `approve`. New tables: `users` (user_id, username, display_name, password_hash), `api_tokens` (token_hash, user_id, scopes, expires), `channel_users` (for future channel integrations). `user_id` column added to `sessions` and `workstreams`. Console owns admin CRUD (6 endpoints under `/api/admin/`); server validates JWTs only. Public `/api/auth/setup` endpoint for first-time admin creation (atomic: create user + return JWT, only works when zero users exist). `turnstone-admin` CLI for user/token management. Auth status endpoint drives UI setup wizard detection. Server-side validation: username 1-64 ASCII, password min 8 chars, scopes validated against whitelist. Python + TypeScript SDKs updated with `login(username, password)`, `authStatus()`, `setup()`. OpenAPI specs updated. Alembic migration `002_user_identity.py`. New `docs/security.md` + diagram `15-auth-architecture.puml`. Dependencies: PyJWT, bcrypt. 64 new tests (1023 total)
- [x] **Communication channel integrations** — Unified `turnstone-channel` gateway microservice with `ChannelAdapter` protocol for bidirectional platform adapters. Discord adapter implemented first using discord.py v2.4+ (gateway WebSocket, MESSAGE_CONTENT intent). Thread-per-@mention model: bot creates Discord thread + workstream on mention, streams responses via edit-in-place (1.5s interval), tool approvals via persistent button views (Approve/Reject/Always). Slash commands: `/link` (API token → channel_users mapping), `/unlink`, `/ask`, `/status`, `/close`. `AsyncRedisBroker` (redis.asyncio) for async MQ integration on discord.py event loop. `ChannelRouter` manages channel/thread → workstream mapping via `channel_routes` table (persistent, survives restart). Message formatter with code-block-aware chunking (2000 char Discord limit). Unlinked users silently dropped. New tables: `channel_routes` (migration 003). Storage CRUD: 4 channel_user + 4 channel_route methods on both SQLite/PostgreSQL backends. Optional dep: `pip install turnstone[discord]`. Compose service: `channel` on production profile. 83 new tests (1106 total). Protocol designed for future Slack/Teams adapters
- [x] **Node version tracking / drift detection** — VER column added to console dashboard node table; `get_overview()` returns `version_drift` + `versions`; `get_nodes()` promotes `version` to top-level field; `get_version_info()` method for per-node map; group headers show "mixed" + yellow drift badge when versions differ; status bar shows DRIFT warning or single version; console `/health` includes drift fields; 10 new tests
- [x] **Deployment packaging** — Docker Compose `production` profile with PostgreSQL service, resource limits, and `TURNSTONE_DB_BACKEND`/`TURNSTONE_DB_URL` env vars. Dockerfile updated with `libpq5`, `[postgres]` extras, and `entrypoint.sh` migration runner. Helm chart (`deploy/helm/turnstone/`) with bitnami PostgreSQL/Redis subcharts, 3 deployments (server, bridge, console), migration Job (pre-install/pre-upgrade hook), ingress, configmap, secrets, NOTES.txt. Terraform AWS ECS/Fargate module (`deploy/terraform/modules/aws-ecs/`) with RDS PostgreSQL, ElastiCache Redis, ALB, security groups, IAM, Secrets Manager; example in `deploy/terraform/examples/aws-ecs-basic/`. `.env.example` for documented environment variables
- [x] **API versioning / OpenAPI spec** — All API endpoints versioned under `/v1/` prefix via Starlette `Mount`. OpenAPI 3.1 spec generated from Pydantic v2 models at `/openapi.json`; Swagger UI at `/docs`. New `turnstone/api/` package with schemas, spec builder, and endpoint catalogs for both server and console. Auth middleware strips version prefix for path classification. Console proxy handles `/v1/api/` upstream forwarding. 35 new tests (823 total)
- [x] **Client SDK** — Python SDK (`turnstone/sdk/`) with sync + async clients for both server and console APIs. Returns Pydantic models directly, streams SSE events as 27 typed dataclasses with registry-based deserialization. High-level `send_and_wait()` for request-response patterns. TypeScript SDK (`sdk/typescript/`) with zero browser dependencies, fetch + ReadableStream SSE parsing, discriminated union event types with type guards. 63 Python tests, 21 TypeScript tests (896 total)
- [x] **Admin console UI (basic)** — Admin panel in console dashboard with Users and Tokens tabs. Users tab: grid table (username, display name, created, actions), create user modal (username, display name, password with confirmation), delete with cascade confirmation. Tokens tab: user selector dropdown, grid table (prefix, name, scope badges, created, expires, actions), create token modal (name, scopes dropdown, expiry), revoke with confirmation, "token created" show-once modal with copy-to-clipboard. Full keyboard accessibility: focus traps on all modals, Escape to close, backdrop click dismiss, arrow keys for tab switching, ARIA tablist/tabpanel roles, sr-only labels. Responsive column hiding at 700px. Scope badges with semantic colors (read=dim, write=cyan, approve=amber). Custom select chevrons, focus-visible styles, prefers-reduced-motion support. `admin.js` (~500 lines) + HTML + CSS additions

- [ ] **Channel notification tool** — Allow the LLM to send notifications to Discord channels or users via a `notify` tool. Target format: `discord:channel_id` or `discord:user_id` (snowflake IDs for v1, name resolution as follow-up). **New data flow**: server → bridge → Redis → channel gateway (reverse of current channel→server direction). **What exists**: `ChannelAdapter.send(channel_id, content)` protocol method declared but not implemented on `TurnstoneBot`; tool system has established `_prepare_*`/`_exec_*` pattern; `AsyncRedisBroker` pub/sub infrastructure; discord.py `get_channel()`/`fetch_user()` APIs for arbitrary sends. **What's needed**: (1) `tools/notify.json` schema file, (2) `_exec_notify()` in session.py emitting a new SSE event, (3) `ChannelNotifyEvent` dataclass in `mq/protocol.py`, (4) bridge relay logic to publish notify events to a new `{prefix}:notify:{channel_type}` pub/sub channel, (5) channel gateway subscription on the notify channel, (6) `ChannelRouter.notify()` method for adapter dispatch, (7) `TurnstoneBot.send()` implementing the `ChannelAdapter` protocol (`bot.get_channel(id).send(content)` + DM via `bot.fetch_user(id)`). **Open questions**: should the tool support rich embeds or plain text only? Should it verify the channel exists before sending (sync validation vs fire-and-forget)? Should notifications include the workstream context (ws_id, session_id) for traceability? Rate limiting on outbound notifications to prevent abuse?

### Roadmap Phase 2 — SME (Small-to-Medium Enterprise)
_Identity, governance, and observability required for teams and departments adopting at scale._

- [ ] **Enterprise auth (OIDC/SAML, RBAC)** — Extend Phase 1 user identity with pluggable identity provider support (OpenID Connect, SAML 2.0) for SSO integration and full role-based access control. Builds on the existing `users`/`api_tokens` tables. Requires auth provider abstraction, session/JWT management, role hierarchy (admin / operator / viewer), per-workstream access scoping, and immutable audit logging of authenticated actions
- [ ] **Multi-tenancy / workspace isolation** — Logical tenant boundaries so separate teams or departments get isolated environments with their own models, tools, quotas, and data. Goes beyond RBAC roles — requires tenant-scoped storage, per-tenant config overrides, cross-tenant visibility controls, and tenant-aware routing in the MQ layer
- [ ] **Usage metering / token accounting** — Track per-user and per-team token consumption for chargeback, quota enforcement, and usage analytics. Requires metering hooks in the LLM call path, a usage store (tied to database abstraction), quota policies, and reporting API/UI in the console dashboard
- [ ] **Audit logging (immutable)** — Dedicated immutable audit trail of every action: who, what, when, on which workstream, with what outcome. Exportable for SOC 2, ISO 27001, HIPAA compliance reporting. Append-only store with tamper-evident checksums, configurable export formats (JSON, CSV, SIEM integration), and admin query UI in the console
- [ ] **OpenTelemetry / distributed tracing** — Request tracing across the full console → bridge → server → LLM backend path. Requires OpenTelemetry SDK integration, trace propagation through MQ messages and SSE streams, span instrumentation for key operations (LLM calls, tool execution, MQ routing), and exporter config for Jaeger/Zipkin/OTLP backends
- [ ] **Content guardrails / DLP** — Input/output scanning pipeline for PII detection, sensitive data masking, prompt injection defense, and compliance policy violations. Pluggable filter chain that runs before LLM submission and before response delivery. Integration points for external DLP engines (e.g. Microsoft Purview, Google DLP API). Configurable per-tenant policies with block/redact/flag actions
- [ ] **Tool governance** — Org-level approve/deny lists for tools, sandboxing policies, and per-tool usage tracking. Security teams need visibility into what the AI can execute, who authorized each tool, and audit trails of tool invocations. Requires a tool policy engine, admin approval workflow, and reporting in the console
- [ ] **Admin console UI (full)** — Extend the admin console with governance features: tenant management, RBAC role/permission editing, tool approve/deny policies, prompt template management, usage dashboards, and audit log viewer. Builds on the basic admin UI from Phase 1 and the auth/tenancy/governance backends from Phase 2
- [ ] **Prompt governance / template library** — Versioned, org-approved system prompt templates with change control. Prevents teams from deploying arbitrary instructions. Requires a template registry, version history, approval workflow, and per-workstream template binding with override policies
- [ ] **Centralized configuration + secret management** — Replace per-node TOML/env config with a shared config table in the database for cluster-wide settings. Two-tier secret model: **infrastructure secrets** (DB password, Redis password, secret manager credentials) stay in environment variables / mounted files as the bootstrap root of trust — these are needed before the config system is reachable. **Application secrets** (LLM API keys, auth tokens, webhook URLs) stored in the config table via a reference pattern (`env://VAR_NAME`, `vault://path/to/secret`, `awssm://secret-name`) rather than plaintext. A `SecretResolver` protocol with pluggable backends (env vars, HashiCorp Vault, AWS Secrets Manager, K8s Secrets) fetches actual values at runtime. Config table rows are safe to inspect, export, and replicate because they contain pointers, not secrets. Requires: config schema with `is_secret` flag, resolver chain with caching/TTL, admin UI for config editing (mask secret values), and change propagation to running nodes (via MQ notification or polling). Ship with five resolver backends: `env://` (environment variables — universal, zero deps), `file://` (read from path — works with K8s secret mounts, SOPS-decrypted files, Docker secrets, systemd-creds, Conjur-injected files), `pgcrypto://` (encrypted column in config table via PostgreSQL pgcrypto — self-contained, no external service), `vault://` (HashiCorp Vault — optional dep), and cloud backends `awssm://` / `gcpsm://` (AWS/GCP secret managers — optional deps). The `file://` backend is the universal non-cloud answer since every secret injection mechanism ultimately writes to a file
- [ ] **Operational alerting / webhook notifications** — Configurable outbound webhooks for operational events: circuit breaker trips, workstream failures, health degradation, evictions, quota threshold alerts. Support PagerDuty, OpsGenie, generic HTTP endpoints, and the communication channel adapters (Slack/Teams/Discord) from Phase 1. Requires an event subscription model, delivery queue with retry, severity routing, and per-channel configuration. Distinct from the bidirectional channel integrations — this is outbound-only ops alerting
- [ ] **MQ backend abstraction** — Wrap Redis MQ layer behind an abstract interface to support alternative backends. Currently `RedisBroker` is used directly throughout bridge/broker/console with no abstraction layer. Redis pub/sub is fire-and-forget with no delivery guarantees; Redis Streams adds consumer groups and replay but Redis itself remains a single point of failure with a painful clustering story and no native multi-datacenter support. Long-term scaling considerations: **NATS/JetStream** (lightweight, built for service mesh, persistent streams, multi-cluster federation — most natural upgrade path); **RabbitMQ** (full AMQP broker, mature but heavier operationally); **in-memory** (for testing and single-node dev). The abstraction should separate two distinct communication patterns: (1) internal control plane (workstream routing, health, state sync — low volume, needs reliability) and (2) inbound event edge (external triggers from Kafka/SQS/CloudEvents — needs standard connectors and backpressure). Building the interface in Phase 2 allows swapping backends when operational pain is concrete rather than speculative

### Roadmap Phase 3 — Enterprise (Large-Scale / Regulated)
_Compliance, resilience, and operational maturity for large deployments and regulated industries._

- [ ] **Centralized configuration** — Replace per-node TOML/env config with a single source of truth for cluster-wide settings. Options include a config table in the shared database, etcd/consul KV store, or a config distribution protocol over the existing MQ layer
- [ ] **Data retention policies** — Configurable retention rules for session history, conversation logs, metrics, and audit trails. TTL-based auto-pruning, archival to cold storage, compliance-driven deletion (GDPR right-to-erasure). Requires retention policy config, background cleanup jobs, optional archive export, and integration with the auth layer for per-user data management
- [ ] **Backup / disaster recovery** — Automated backups with configurable schedules, cross-region replication for HA deployments, and point-in-time recovery. Defined RTO/RPO targets per data tier (session state vs. audit logs vs. config). Requires backup orchestration, restore verification, and integration with the database abstraction and object storage layers
- [ ] **Session replay** — Step-by-step replay of any completed session for debugging, compliance review, training, or incident investigation. Capture full interaction timeline: user messages, LLM responses, tool calls with inputs/outputs, approval gates, timing data. Requires a structured session recording format, replay UI in the console, and access controls tied to RBAC
- [ ] **High availability / failover** — Redundant console and server instances with automatic failover. Leader election for singleton processes (fanout thread, cleanup jobs), session handoff when a node goes down, and graceful drain for rolling deployments. Requires health-aware load balancing, state replication or shared-nothing architecture, and integration with the MQ layer for node presence tracking

### Technical Debt

- [ ] **MCP resources & prompts** — Support MCP resource primitives (inject context) and prompt templates (slash commands from MCP servers); currently only tools are implemented
- [ ] **MCP dynamic tool refresh** — Add `/mcp refresh` command and `notifications/tools/list_changed` handler so tools update without restart
- [ ] **MCP bridge integration** — Extend MQ bridge protocol to proxy MCP tool calls over Redis so bridge-connected workstreams can use MCP tools
- [ ] **MCP per-server auto-approve** — Allow `auto_approve = true` per MCP server in config for trusted read-only servers; currently all MCP tools require approval
- [x] **Circuit breaker HALF_OPEN single-request permit** — `acquire_request_permit()` (renamed from property to method) now allows exactly one probe request in HALF_OPEN via `_half_open_permit` flag. Session uses `BaseException` catch to ensure `record_failure()` always fires. Permit explicitly reset on all state transitions
- [x] **Rate limiter X-Forwarded-For support** — `resolve_client_ip()` with rightmost-untrusted XFF parsing. Configurable `trusted_proxies` (comma-separated CIDRs). IPv4-mapped IPv6 normalization for dual-stack. Clientless requests pass through instead of sharing a bucket. Startup log shows proxy config. 15 new tests
- [x] **Anthropic web search multi-turn encrypted content** — Raw provider content blocks (including `encrypted_content`/`encrypted_index`) captured during streaming via `_block_to_dict()`, stored as `_provider_content` on assistant messages, passed through verbatim in `_convert_messages()`. Persisted to SQLite via `provider_data` column (auto-migrated). Thinking block text accumulated. `model_dump(exclude_none=True)` prevents Anthropic API rejection. 5 new tests
- [x] **Frontend shared code refactor** — Extracted shared design system into `turnstone/shared_static/`: `base.css` (design tokens, reset, typography, shared components), `auth.js` (login overlay + authFetch), `theme.js` (toggle + localStorage), `toast.js`, `utils.js` (escapeHtml, formatTokens, ctxClass, formatUptime), and `kb.js` (keyboard shortcuts). Both UIs import at `/shared/` via Starlette StaticFiles mount. Console proxy updated: shim injection moved to inline HTML `<script>`, `/shared/` paths rewritten and proxied. ~1540 lines removed from page-specific files, 775 lines in shared package. 13 new tests
- [x] **Channel auto_approve implementation** — `ChannelConfig.auto_approve` and `auto_approve_tools` passed through to `CreateWorkstreamMessage` and `SendMessage`. `_on_ws_event` intercepts `ApprovalRequestEvent`: if blanket `auto_approve` or all tools in `auto_approve_tools`, auto-responds and notifies thread instead of showing buttons. `_should_auto_approve()` checks ALL tool names in event items against config list
- [x] **Channel session resume on reactivation** — When a stale route is detected (evicted workstream), `get_or_create_workstream` captures old ws_id, creates new workstream, then looks up old session via `get_session_id_by_ws()` and sends `/resume <session_id>` command through MQ. New `get_session_id_by_ws()` added to StorageBackend protocol + both backends. Thread notified with "Workstream reactivated." as resume happens asynchronously
- [x] **Channel session resume robustness** — Replaced fragile two-step resume (create workstream + fire-and-forget `/resume` CommandMessage) with atomic flow: `CreateWorkstreamMessage.resume_session` field passes session_id to server, which resumes atomically during `POST /v1/api/workstreams/new` (same HTTP request). Bridge emits `SessionResumedEvent` to per-workstream channel for confirmation. Discord bot displays "Session resumed: {name} ({count} messages restored)". Server UI `dashboardResumeSession` simplified to single request. Pruned sessions fall back gracefully to fresh start (`resumed: false`). `WorkstreamCreatedEvent` extended with `resumed`/`session_id`/`message_count` fields. Discord thread message replay deferred (threads preserve their own history)
- [x] **Bridge/console service auth** — Bridge and console silently failed all server API calls (401) when `TURNSTONE_AUTH_TOKEN` was empty and JWT auth was enabled. Both now auto-mint a service JWT from `TURNSTONE_JWT_SECRET` at startup: bridge gets `approve` scope (needs it for tool approval proxy), console collector gets `read`, console proxy gets `write`. 1-week expiry, refreshed on restart
- [x] **AsyncRedisBroker pub/sub race condition** — Multiple per-channel listener tasks polled `get_message()` on the same `PubSub` connection, causing tasks to steal each other's messages. Replaced with single `_dispatch_loop` that routes messages to callbacks by channel name. Callbacks fired as independent `asyncio.Task` so slow callbacks don't block other channels
- [x] **Discord approval format** — Formatter expected nested `item["function"]["name"]` but server SSE sends flat `item["func_name"]`. Fixed to read both formats. Also fixed `_handle_create_ws` not tracking initial message in `_active_sends`, which prevented `TurnCompleteEvent` from firing and left streaming messages unfinalized
- [ ] **Channel bidirectional message routing** — Messages sent via the node server UI are not forwarded to the linked Discord thread. The channel adapter only subscribes to MQ events that originate from channel-initiated workstreams, but doesn't relay responses triggered by server UI input. When a user then sends a message from Discord, both the queued server-side response and the new Discord-triggered response arrive simultaneously, causing duplicate/stomped replies. Need to either: (a) subscribe the channel adapter to all SSE events for linked workstreams regardless of origin, ensuring server UI messages and their responses are mirrored to the Discord thread; or (b) make the bridge publish a `MessageEvent` on the global/ws channel whenever a server UI message is sent, so the channel adapter can echo it. Must also handle dedup so responses aren't rendered twice when the same workstream is active in both surfaces
- [x] **Service JWT auto-refresh** — `ServiceTokenManager` in `core/auth.py` auto-rotates service JWTs (1h expiry, refresh at 80% lifetime). Bridge uses httpx event hooks (`_inject_auth`) for transparent per-request header injection. Console stores manager on `app.state` for collector and proxy tokens. Hard fail (`SystemExit(1)`) when `TURNSTONE_JWT_SECRET` is missing and no `--auth-token` provided
- [ ] **Channel Slack adapter** — Implement `SlackChannelAdapter` using `slack_bolt` (asyncio mode) or Slack socket mode. Reuse `ChannelAdapter` protocol, `ChannelRouter`, and `_formatter`. Thread-per-conversation model maps to Slack threads. App Home tab for `/link`. Optional dep: `pip install turnstone[slack]`
- [ ] **Channel Teams adapter** — Microsoft Teams adapter using Bot Framework SDK. Webhook receiver + proactive messaging. Adaptive Cards for approval flows instead of buttons. Optional dep: `pip install turnstone[teams]`
- [ ] **Channel admin endpoints for console API spec** — Add the 3 channel user admin endpoints to `turnstone/api/console_spec.py` for OpenAPI documentation
- [ ] **Channel integration tests** — Test `get_or_create_workstream` success path, `_on_global_event` handling, `_on_ws_event` dispatch, `_recover_routes`, admin channel endpoints, `/link` token validation, concurrent workstream creation race, PostgreSQL channel CRUD
- [x] **Admin confirm dialogs** — Replaced browser `confirm()` with styled `#confirm-overlay` modal matching instrument panel aesthetic. Reusable `showConfirmModal(title, message, actionLabel, callback)` with red confirm button, focus trap, Escape/backdrop dismiss. Applied to delete user, revoke token, and unlink channel across all three admin tabs. Descriptive messages explain consequences (e.g. cascade deletes, JWT validity window)
- [x] **Admin focus return on modal close** — All admin modal hide functions (`hideCreateUserModal`, `hideCreateTokenModal`, `hideTokenCreatedModal`, `hideCreateChannelModal`, `hideConfirmModal`) now return focus to their respective trigger button/element. Confirm modal tracks `_confirmTriggerEl` via `document.activeElement` at open time
- [ ] **SDK integration tests** — Current SDK tests use mocked httpx transports only. Add TestClient-based integration tests that verify the SDK against a real Starlette server app: `send_and_wait` with SSE streaming, `stream_events`/`stream_global_events`/`stream_cluster_events` generators, auth flow round-trips, and error code handling. Would catch contract drift between SDK and server
- [x] **SDK `_SyncRunner.run_iter` robustness** — `_safe_anext()` sentinel wrapper catches `StopAsyncIteration` inside the coroutine boundary, returns `_STOP` sentinel. `run_iter` checks for sentinel instead of catching exception across thread boundary. Empty generator edge case test added
- [ ] **Storage PostgreSQL integration tests** — Current PostgreSQL backend tests are skipped by default (no `@pytest.mark.postgres` tests written yet). Add CI PostgreSQL service and integration tests that verify all 21 `StorageBackend` methods against real PostgreSQL, including tsvector full-text search parity with SQLite FTS5
- [ ] **Storage MariaDB/MySQL backend** — `StorageBackend` protocol supports additional backends. MariaDB/MySQL would need a `_mysql.py` backend with `FULLTEXT` index for search and `ON DUPLICATE KEY UPDATE` for upserts. Requires `mysqlclient` or `aiomysql` optional dependency
- [ ] **Alembic offline SQL generation** — Current migrations run online only. Add `alembic upgrade --sql` support for environments where DDL must be reviewed/approved before execution (common in regulated enterprises)
- [ ] **Air-gapped deployment bundling** — `deploy/air-gap/bundle.sh` script to download Swagger UI dist, pip wheels, Docker images into a single offline tarball with `install.sh` for network-isolated environments. Swagger UI CDN already configurable via `swagger_ui_base_url`
- [ ] **Approval timeout shows as approved in UI** — When the bridge's approval timeout expires, the server-side `_approval_event.wait()` returns with a denial, but the UI renders the tool call block as if it was approved (green/success styling) instead of showing a timeout/denied state. The `approve_request` SSE event is cleared but no corresponding `tool_result` with a timeout indicator is sent to the browser, so the approval block stays in its pending visual state or flips to approved. Need to emit a distinct `approval_timeout` or `tool_result` event with an error/timeout status so the UI can render it correctly (red/warning styling, "Approval timed out" label)
- [x] **SSE proxy: use httpx-sse instead of manual parsing** — Replaced manual chunk buffering / `\r` stripping / `\n\n` splitting with `httpx_sse.aconnect_sse()`. Eliminates fragile hand-rolled parser and chunk-boundary bugs. Event types now forwarded correctly
- [x] **SSE proxy event type forwarding** — Resolved by the httpx-sse refactor above; `sse.event` is always forwarded in the yield dict
- [ ] **Docker Compose integration test harness** — Manual Docker Compose testing revealed multiple regressions that unit tests didn't catch: unversioned collector API paths, SSE `\r\n` line ending mismatch through proxy, workstream state not updating for dashboard polling. Add a CI-friendly integration test that spins up the production compose stack (postgres + redis + server + bridge + console), verifies health endpoints, sends a message, and checks state propagation end-to-end. Could use `docker compose up -d` + `pytest` with httpx against exposed ports
- [x] **Console proxy duplicate branding** — Removed "turnstone" span from `_CONSOLE_BANNER_TEMPLATE`. Banner is now purely navigational: `← Console │ node-id`
- [x] **Console proxy back-navigation** — Removed `target="_blank"` from node link, changed `window.open` to `window.location.href` for same-tab navigation with working back button
- [ ] **Workstream-scoped config** — `session_config` is keyed by `(session_id, key)` with no direct `ws_id` column. Workstream-level config (settings that persist across session rotations within a workstream) requires a join through `sessions.ws_id`. Consider adding `ws_id` to `session_config` or a separate `workstream_config` table if workstream-scoped settings become a common access pattern
- [ ] **Per-user scope storage** — Password login in `auth_login` grants all scopes (`read,write,approve`) to every database user regardless of intent. The `users` table has no `scopes` column — all authenticated users are effectively admins. Add a `scopes` column to `users` (default `read,write`) and use it in `auth_login` instead of hardcoding `frozenset({"read", "write", "approve"})`. The `CreateUserRequest` schema already has no scopes field; one should be added. This is a prerequisite for non-admin user accounts
- [x] **`escapeHtml` does not encode single quotes** — Added `'` → `&#39;` replacement to `escapeHtml()` in `shared_static/utils.js`
- [x] **Username character validation too permissive** — Added `is_valid_username()` in `core/auth.py` restricting to `[a-zA-Z0-9._-]{1,64}`. Used in `server.py`, `console/server.py` (auth_setup + admin_create_user), and `admin.py` CLI
- [x] **Structured logging** — `turnstone/core/log.py` with structlog + stdlib integration. `configure_logging(level, json_output, service)` replaces all `basicConfig()` calls. JSON output when stderr is not a TTY (production/Docker), colored console otherwise. ContextVars (`ctx_node_id`, `ctx_ws_id`, `ctx_user_id`, `ctx_request_id`) auto-injected into every log event. `LogContextMiddleware` sets request_id + ws_id per HTTP request with token-based reset. Bridge `_run_in_context()` propagates ctx_node_id to child threads. Env var overrides: `TURNSTONE_LOG_FORMAT`, `TURNSTONE_LOG_LEVEL`. All 6 entry points retrofitted with `--log-level` and `--log-format` flags. Server operational `print()` calls replaced with structured `log.info()`. 18 new tests (959 total)
