# Delphi Panel Review: Close Structural Test Coverage Gaps

**Date:** 2026-02-25
**Spec Version:** 1.0 (Post-Clarify)
**Panelists:** Senior Test Engineer, OSS Maintainer, Security Engineer, Tech Lead

## Executive Summary

The spec correctly identifies the highest-risk untested code -- the orchestration entry points and safety boundaries -- and proposes a sound mock-boundary strategy (mock HTTP+LLM, keep real ContextVar+DuckDB+RecipeStore). However, the orchestrator integration tests will require careful design around the `provider` lazy singleton proxy (`api_agent/agent/model.py` lines 57-64), the `trace_metadata` context manager, and the deeply nested tool-function closures inside `process_query`/`process_rest_query`. The panel approves with conditions: the spec needs explicit guidance on how to mock the LLM provider's `run_tool_loop` for orchestrator tests, and the security test scope should be expanded to cover two bypass vectors the current regex misses.

## Panelist Reviews

### Senior Test Engineer

**Verdict:** Approve with Conditions

#### Strengths

1. **Mock boundary is well-chosen.** The decision to mock at HTTP transport (httpx) and LLM (`provider.run_tool_loop`) while keeping real ContextVar, DuckDB, and RecipeStore is the right call. This tests the actual wiring without flakiness from network or LLM non-determinism.

2. **The existing `test_executor.py` and `test_rest_client.py` patterns are solid models.** `test_executor.py` (lines 107-158) tests `execute_sql` with real DuckDB against in-memory data -- the same approach should work for the orchestrator tests when they invoke SQL steps. `test_rest_client.py` (lines 78-183) demonstrates clean method-blocking tests that verify safety without actual HTTP calls. The spec implicitly asks for the same pattern.

3. **Replacing `test_recipe_runner.py` is correct.** The current tests (lines 29-43, 76-90) monkeypatch `execute_recipe_steps` itself, which means they test that `execute_recipe_tool` calls the executor but never verify the executor's actual behavior with real recipe structures. The `fake_execute_recipe_steps` function at line 29 returns a hardcoded tuple -- it never invokes the `api_step_executor` callback, never runs SQL steps, never touches ContextVar. This is exactly the "mock the thing you're testing" anti-pattern.

#### Concerns

1. **The `provider` proxy is a testing hazard.** In `api_agent/agent/model.py` lines 57-64, `provider` is a module-level `_ProviderProxy` that lazily creates a real LLM provider on first attribute access. Both `process_query` (graphql_agent.py line 629) and `process_rest_query` (rest_agent.py line 742) call `provider.run_tool_loop()`. If tests import these modules without first patching the provider, the proxy will attempt to create a real OpenAI/Anthropic client, which will fail without API keys. The spec doesn't address this.

   **Recommendation:** The spec should require either (a) monkeypatching `api_agent.agent.model.provider` before any orchestrator test, or (b) monkeypatching `api_agent.agent.model._provider` with a fake `LLMProvider` subclass that returns scripted `RunResult` objects. Option (b) is cleaner because it avoids triggering `get_provider()` entirely.

2. **`trace_metadata` will silently pass in tests but could mask import issues.** The `trace_metadata` context manager (tracing.py lines 49-56) is a no-op when tracing is disabled, so it won't block tests. However, the `with trace_metadata(...)` wrapper around `provider.run_tool_loop()` at graphql_agent.py line 628 and rest_agent.py line 741 means the mock must be set up before entering that context manager. This is fine but worth documenting in test setup comments.

3. **ContextVar isolation between tests needs explicit management.** Both orchestrators set ContextVars at the top of `process_query`/`process_rest_query` (graphql_agent.py lines 582-588, rest_agent.py lines 672-678). Since pytest-asyncio runs each test in its own event loop iteration, ContextVars should be isolated. But if tests share the same event loop (the `session` scope mode), ContextVars will bleed between tests. The spec should mandate `@pytest.mark.asyncio` with default (function-level) scope for all new tests.

4. **Tool closures capture `ctx` and ContextVars by reference.** The `_create_graphql_query_tool` function (graphql_agent.py line 331) creates a closure over `ctx` and the module-level ContextVars. When `provider.run_tool_loop` executes tool calls, it invokes these closures. If the test mocks `run_tool_loop` to return a canned `RunResult` without actually calling tools, the ContextVars (`_graphql_queries`, `_last_result`, etc.) will remain at their reset values from lines 582-586. This means tests won't verify the ContextVar -> result assembly path unless the mock actually invokes the tools.

   **Recommendation:** For orchestrator tests, use a `FakeLLMProvider` that actually calls the tool functions with scripted arguments (simulating what an LLM would do), then returns a final `RunResult`. This preserves the ContextVar flow. Alternatively, the spec should split the orchestrator tests into two categories: (a) wiring tests that mock `run_tool_loop` at a high level and verify the setup/teardown around it, and (b) tool-execution tests that invoke individual tool functions directly with pre-set ContextVars.

5. **The `_fetch_schema_context` function (graphql_agent.py line 231) makes a real HTTP call to fetch the GraphQL introspection schema.** This call happens before `run_tool_loop` is invoked (line 591). The spec says to mock httpx, but the introspection fetch uses `graphql_fetch` which is `execute_query` from `graphql/client.py`, which creates its own `httpx.AsyncClient` (client.py line 47). The mock must intercept this specific httpx call with a canned introspection response. Similarly, `fetch_schema_context` in rest_agent.py line 681 calls `rest/schema_loader.py` which also creates its own httpx client (schema_loader.py line 34). The spec should clarify whether to use `pytest-httpx`, `respx`, or `monkeypatch` for these mocks, and provide guidance on fixture structure for canned schema responses.

#### Recommendations

- Add a "Test Infrastructure" section to the spec defining: (1) how to mock the provider, (2) how to mock httpx for schema fetches, (3) ContextVar scope requirements.
- Consider creating a `FakeLLMProvider(LLMProvider)` test helper class that can be scripted to return specific `LLMResponse` sequences and actually execute tool calls, living in a `tests/conftest.py` or `tests/helpers/` module.
- Ensure recipe runner behavioral tests create real `Recipe` dicts with actual step structures (not empty `steps: []`), use a real `RecipeStore.save_recipe()` to populate the store, and mock only `graphql_fetch`/`execute_request` at the httpx boundary.

---

### Open-Source Maintainer

**Verdict:** Approve with Conditions

#### Strengths

1. **The 3-PR strategy is practical and contributor-friendly.** PR 1 (critical gaps: orchestrators, GraphQL client, execute tool, config) addresses the highest-impact items first. A contributor evaluating this project can look at PR 1 alone and gain confidence that the main code paths are tested. PR 2 (weak coverage) and PR 3 (cleanup) can follow without blocking.

2. **Config contract tests (FR-008, SC-004) are the single most contributor-friendly addition.** Open-source contributors routinely break deployments by changing env var names or defaults. The current `config.py` (72 lines) has `AliasChoices` for `API_KEY` (line 28: `AliasChoices("API_AGENT_API_KEY", "OPENAI_API_KEY", "ANTHROPIC_API_KEY")`) and `BASE_URL` (line 34: `AliasChoices("API_AGENT_BASE_URL", "OPENAI_BASE_URL")`). These alias chains are the most likely source of contributor-introduced regressions. Snapshot tests for every env var name and default will catch these immediately.

3. **FR-013 (deterministic tests) and FR-015 (don't break existing tests) are essential constraints.** Contributors will run `uv run pytest tests/ -v` and expect a clean pass. Any flakiness or dependency on live APIs would destroy trust.

4. **The acceptance scenarios in User Story 1 are concrete and testable.** Scenarios 1-5 map directly to code paths in `process_query` and `process_rest_query`. A contributor reading these scenarios can understand what the orchestrator is supposed to do without reading the implementation.

#### Concerns

1. **Missing: contributor onboarding for writing new tests.** The spec defines what tests to write but not how a new contributor would add a test. The existing test files don't have a consistent pattern -- `test_executor.py` uses classes, `test_recipe_runner.py` uses standalone functions, `test_rest_client.py` mixes both. The spec should recommend a consistent style for new test files.

2. **Missing: fixture organization strategy.** The orchestrator tests will need canned GraphQL introspection responses, canned LLM responses (scripted `RunResult` sequences), and canned REST OpenAPI specs. Where do these live? Inline in the test file? In a `tests/fixtures/` directory? In `conftest.py`? For an OSS project, fixture discoverability matters. If a contributor needs to add a new schema variant, they should know exactly where to put it.

3. **SC-008 (90% regression detection via second reverse-TDD review) is subjective.** The initial 60% assessment was presumably a manual judgment call. Repeating it will yield a different number depending on who does it and their interpretation. For OSS credibility, consider supplementing this with a concrete metric: statement coverage percentage via `pytest-cov`. Not as a replacement (coverage != regression detection), but as a secondary, reproducible signal.

4. **The spec doesn't mention CI integration for coverage reporting.** The existing CI (`.github/workflows/test.yml`) runs tests but likely doesn't report coverage. Adding `pytest-cov` to CI with a minimum threshold (e.g., 80% statement coverage) would give contributors a clear, automated signal. This is table stakes for 5k+ star projects.

5. **Edge case from spec: "What happens when the GraphQL schema introspection returns an empty or malformed schema?"** This is a real contributor concern. Looking at `_fetch_schema_context` (graphql_agent.py lines 231-259), if `graphql_fetch` returns `{"success": False}`, the function returns `""` (empty string). Then `schema_ctx` is empty (line 591), `_raw_schema` is never set, `search_recipes` gets an empty string for `raw_schema` and returns no suggestions. The agent runs with no schema context. This is tested implicitly by the "schema fetch fails" acceptance scenario (US1-6), but the spec should ensure the test verifies the agent still produces a reasonable error or falls back gracefully, not just "doesn't crash."

#### Recommendations

- Add a "Test Style Guide" section: use classes for related scenarios, `@pytest.mark.asyncio` for all async tests, descriptive docstrings on every test method (following `test_executor.py` pattern).
- Create a `tests/fixtures/` directory with canned responses (GraphQL introspection JSON, OpenAPI spec JSON, LLM response sequences) as `.json` files. Reference these from tests via `pathlib.Path(__file__).parent / "fixtures" / "..."`.
- Add `pytest-cov` to the dev dependencies and CI, with a floor of 75% statement coverage (conservative but enforceable). This supplements the manual reverse-TDD review with a reproducible metric.

---

### Security Engineer

**Verdict:** Approve with Conditions

#### Strengths

1. **The mutation blocker is correctly identified as a security-critical path.** The `_MUTATION_PATTERN` regex at `graphql/client.py` line 12 is `re.compile(r"^\s*mutation\b", re.IGNORECASE | re.MULTILINE)`. The spec's SC-002 requires testing 5 mutation format variants. This is the right priority.

2. **REST method blocking is already well-tested in `test_rest_client.py`.** The existing tests (lines 82-183) cover POST, PUT, DELETE, PATCH blocking and the `allow_unsafe_paths` glob matching. This is solid and doesn't need additional coverage.

3. **FR-005 (mutation variants) is specific and actionable.** The spec explicitly calls out "standard, mixed case, leading whitespace, multiline" variants.

#### Concerns

1. **The mutation regex has a known bypass vector: queries containing mutation in a non-operation context.** The regex `^\s*mutation\b` with `re.MULTILINE` will match any line that starts with "mutation" -- including inside comments or string literals within a query. However, the more dangerous bypass is a query that wraps a mutation inside a query operation:

   ```graphql
   query {
     __typename
   }
   mutation {
     deleteUser(id: "1") { id }
   }
   ```

   This is actually a **valid concern the regex handles correctly** because `re.MULTILINE` makes `^` match at the start of each line, and the second line `mutation {` would match. Good.

   However, consider this bypass:

   ```graphql
   {
     mutation: __typename
   }
   ```

   This is a valid query using a field alias named "mutation". The regex would NOT match this because `mutation` doesn't appear at the start of a line preceded only by whitespace. This is a **false negative that's actually correct behavior** -- this is a query, not a mutation. The regex is sound for its purpose.

2. **Batch queries (arrays) are a real bypass vector that the spec doesn't address.** Some GraphQL servers support batched operations via JSON arrays:

   ```json
   [
     {"query": "{ users { id } }"},
     {"query": "mutation { deleteUser(id: \"1\") { id } }"}
   ]
   ```

   Looking at `execute_query` in `client.py` lines 43-44, the payload is always `{"query": query}` -- a single string. The mutation check happens on that single string (line 36). If a caller passes a batch query as a string, the mutation check would catch it because `_MUTATION_PATTERN.search(query)` scans the entire string with `re.MULTILINE`. However, the batch format is JSON-level, not string-level -- it would need to be handled at the HTTP payload level, not the query string level. Since `execute_query` always sends `{"query": query}` (not an array), batch injection is not possible through this code path. Good.

3. **The `_execute` MCP tool (execute.py) bypasses the agent loop but still uses `execute_query`.** At execute.py line 57, the `_execute` tool calls `execute_query(query, variables, ctx.target_url, ctx.target_headers)`, which goes through the mutation check. This is correct. However, the test spec (FR-007) should explicitly verify that `_execute` with a mutation query returns an error. This is implied but not called out in the acceptance scenarios.

4. **`X-Allow-Unsafe-Paths` header is a security-relevant control surface.** The spec mentions this in the edge cases but doesn't require explicit tests. A contributor could accidentally change the glob matching logic in `rest/client.py` and break the safety model. However, this is already covered by `test_rest_client.py` (lines 135-183). The spec should explicitly note that existing REST safety tests are sufficient and don't need to be re-written.

5. **GraphQL subscriptions are mentioned in edge cases but not tested.** Looking at the regex `^\s*mutation\b`, a subscription operation (`subscription { ... }`) would NOT be blocked. The `execute_query` function would attempt to send it as a POST to the GraphQL endpoint, which would fail because subscriptions require WebSocket. This is a minor concern -- the worst case is a failed HTTP request, not a security bypass. But the spec should decide whether to block subscriptions explicitly.

6. **The GraphQL response parsing at client.py lines 56-58 has a subtle security-adjacent concern.** If the response contains both `errors` and `data`, the current code returns `{"success": False, "error": result["errors"]}` at line 57, discarding the `data` field entirely. This is conservative (safe), but the spec's FR-006 says "errors are surfaced alongside any partial data." This is a contradiction with the current implementation. The tests should verify the **actual** behavior (errors = failure, data discarded), not the spec's aspirational behavior, unless the implementation is also being changed.

   **This is important.** Line 56: `if "errors" in result: return {"success": False, "error": result["errors"]}`. The `data` field is ignored. The spec's acceptance scenario US2-4 says "errors are surfaced in the return value alongside any partial data." If the test is written to match the spec, it will fail against the current code. If the test is written to match the current code, it doesn't match the spec. The spec needs to clarify: is this a behavior change or a documentation of current behavior?

#### Recommendations

- Add a test case for `_execute` with a mutation query to explicitly verify the safety boundary holds through the MCP tool layer, not just the GraphQL client directly.
- Clarify FR-006 / US2-4: does "errors alongside partial data" describe desired future behavior or current behavior? Current implementation at client.py line 56-57 discards data when errors are present.
- Add a test for subscription operations (`subscription { ... }`) to document the current behavior (passes regex, fails at HTTP level).
- Add a note that existing REST safety tests in `test_rest_client.py` are sufficient and out of scope for this effort.
- Consider adding a test for the `_MUTATION_PATTERN` against `fragment` operations to ensure they're not falsely blocked: `fragment UserFields on User { id name }` should not trigger the mutation check (it won't, because the regex checks for `mutation\b`, not `fragment\b`, but explicit verification is good documentation).

---

### Pragmatic Tech Lead

**Verdict:** Approve with Conditions

#### Strengths

1. **The priority ordering is correct.** The orchestration entry points are where regressions will actually originate. Testing `_format_type` helpers (already tested) is low-value; testing `process_query` wiring is high-value. The spec correctly front-loads the highest-risk items.

2. **The 31-point estimate is reasonable for 8 work items across 2-3 PRs.** The largest item (orchestrator integration tests, 8 points) is appropriately sized -- it requires the most mock setup and has the most code paths.

3. **The "replace, don't keep alongside" decision for recipe runner tests is pragmatic.** Maintaining two test suites for the same code is a maintenance burden. The current tests in `test_recipe_runner.py` (102 lines) provide false confidence -- they verify call-through behavior but not actual recipe execution. Replacing them is the right call.

#### Concerns

1. **The LLM provider `complete()` tests (5 points, FR-011) are the lowest-value item and could be deferred.** Looking at the three providers:

   - `openai_provider.py` (lines 24-67): `complete()` is 27 lines. It calls the OpenAI SDK and parses the response. The parsing is trivial (lines 44-53: iterate `message.tool_calls`, JSON-parse arguments). Testing this with httpx mocks requires mocking the OpenAI SDK's internal HTTP layer, which is fragile and couples tests to SDK internals.

   - `anthropic_provider.py` (lines 25-82): `complete()` is 57 lines. It has more logic (system message extraction, content block parsing), but still fundamentally wraps the Anthropic SDK. Same mock fragility issue.

   - `openai_compat.py` (lines 32-82): `complete()` has the retry-without-tools fallback (lines 51-58), which is the only genuinely interesting logic. This one is worth testing.

   **Recommendation:** Test only `openai_compat.py`'s retry-without-tools path (2 points instead of 5). The OpenAI and Anthropic providers' `complete()` methods are thin SDK wrappers; their `format_*` methods are already tested. Defer full `complete()` testing until a provider SDK upgrade creates a concrete risk.

2. **The middleware non-recipe path tests (3 points, FR-012) overlap with the orchestrator tests.** If the orchestrator tests verify that `process_query` is called with the correct arguments (which requires the query handler to route correctly, which requires the middleware to resolve tool names), then middleware tests provide limited additional value. However, the middleware `on_call_tool` non-recipe path (middleware.py lines 246-266) does have its own logic: validating the tool name prefix, extracting the suffix, and creating a modified context. This is worth 1-2 tests, not a full 3-point item.

   **Recommendation:** Reduce to 2 points. Test: (a) valid tool name resolves to correct internal name, (b) invalid tool name prefix raises `NotFoundError`. Skip testing the `call_next` forwarding -- that's framework behavior.

3. **The query handler tests (3 points, FR-010) could be simplified.** Looking at `query.py` (74 lines), the `query()` function is 30 lines. It does: get context, reset recipe flag, route to GraphQL or REST, check recipe changes, format response. The routing is a simple `if/else` on `req_ctx.api_type`. The recipe notification logic is 4 lines. The CSV detection is 2 lines. This is 2 points of work, not 3.

4. **Missing from scope: the `_should_stop` callback.** Both orchestrators define a `_should_stop` closure (graphql_agent.py lines 617-621, rest_agent.py lines 730-734) that checks `_return_directly_flag`. This is the mechanism for `return_directly=True` to short-circuit the agent loop. If this callback breaks, the agent will continue looping unnecessarily, wasting LLM tokens. The spec mentions `return_directly` in the recipe context but doesn't explicitly test the `_should_stop` -> `__DIRECT_RETURN__` path through the orchestrator.

5. **The spec doesn't prioritize within PR 1.** If PR 1 takes longer than expected, which items should ship first? I'd argue: (a) GraphQL client mutation tests (2 points, highest safety value per point), (b) config contract tests (2 points, highest contributor protection per point), (c) execute tool tests (3 points), (d) orchestrator tests (8 points). This way, even a partial PR 1 delivers meaningful safety and contributor protection.

6. **Consider adding a "minimum viable" column to the effort table.** For each work item, what's the minimum number of tests that delivers 80% of the regression detection value? For orchestrators: 4 tests (success, MaxTurnsExceeded, no output, error). For mutation blocker: 3 tests (standard mutation, mixed case, valid query passes). For config: 2 tests (defaults, alias priority). This gives a fallback plan if the effort runs over.

#### Recommendations

- Defer full LLM provider `complete()` tests to a separate effort. Test only the `openai_compat` retry path now (saves 3 points).
- Reduce middleware non-recipe path to 2 points (2 focused tests).
- Reduce query handler to 2 points.
- Add explicit prioritization within PR 1 for partial-delivery safety.
- Revised total: ~25 points (down from 31), with a "minimum viable" floor of ~16 points that covers orchestrators + safety + config.

---

## Points of Consensus

1. **All panelists agree the orchestrator tests are the highest-priority gap.** `process_query` and `process_rest_query` are 700+ and 800+ lines respectively with zero test coverage. This is the most dangerous blind spot.

2. **All panelists agree the mock boundary (HTTP+LLM mocked, ContextVar+DuckDB+RecipeStore real) is correct.** This tests the wiring without introducing flakiness.

3. **All panelists agree the existing `test_recipe_runner.py` should be replaced.** The current monkeypatch-everything approach provides false confidence.

4. **All panelists agree FR-013 (deterministic tests) and FR-015 (don't break existing tests) are non-negotiable constraints.**

5. **All panelists agree the config contract tests deliver outsized value relative to their complexity.** 2 points for protecting every deployment from env var renames.

## Points of Disagreement

| Topic | Position A | Position B |
|-------|-----------|-----------|
| LLM provider `complete()` tests | **Security + Test Engineer:** Test all 3 providers; SDK upgrades are a real risk vector and the response parsing is provider-specific. | **Tech Lead:** Only test `openai_compat` retry path; the other two are thin SDK wrappers and mocking SDK internals is fragile. **OSS Maintainer** leans toward Position A for contributor confidence but acknowledges the fragility concern. |
| FR-006 / US2-4 (errors alongside partial data) | **Security Engineer:** The spec says "errors alongside partial data" but the code discards data when errors present (client.py line 56-57). This must be resolved before writing tests. | **Test Engineer + Tech Lead:** Test the current behavior (errors = failure). If the behavior should change, that's a separate feature, not a test gap. |
| Middleware non-recipe path effort | **Test Engineer:** 3 points is fair; the tool name validation and context forwarding have subtle edge cases. | **Tech Lead:** 2 points; the `call_next` forwarding is framework behavior that doesn't need our tests. |
| Coverage metrics in CI | **OSS Maintainer:** Add `pytest-cov` with a floor; this is table stakes for credible OSS. | **Tech Lead:** Coverage metrics are a vanity number; the reverse-TDD review is more meaningful. **Test Engineer:** Both -- use coverage as a floor, reverse-TDD as the quality bar. |

## Consolidated Recommendations

Prioritized by impact and urgency:

1. **[MUST] Add a "Test Infrastructure" section to the spec** defining how to mock the LLM provider (`_ProviderProxy` at model.py line 64), how to mock httpx for schema fetches (introspection in graphql_agent.py line 233, OpenAPI spec in rest_agent.py line 681), and ContextVar scope requirements (function-level `@pytest.mark.asyncio`).

2. **[MUST] Resolve FR-006 / US2-4 ambiguity.** The current code at `client.py` lines 56-57 discards `data` when `errors` is present. The spec's acceptance scenario says "errors alongside partial data." Decide: test current behavior or change behavior first? Recommendation: test current behavior, file a separate issue for the behavior change if desired.

3. **[MUST] Expand mutation blocker test scope** to include: (a) subscription operations, (b) fragment-only documents, (c) a mutation query sent through the `_execute` MCP tool (not just the GraphQL client directly). This costs ~30 minutes of additional work and closes real security verification gaps.

4. **[SHOULD] Create a `FakeLLMProvider` test helper** that extends `LLMProvider`, returns scripted `LLMResponse` sequences, and optionally executes tool calls to preserve the ContextVar -> result assembly flow. This is a one-time investment (~2 points) that dramatically simplifies all orchestrator tests.

5. **[SHOULD] Establish fixture organization** with a `tests/fixtures/` directory containing canned GraphQL introspection JSON, OpenAPI spec JSON, and LLM response sequences. This improves contributor discoverability and test maintainability.

6. **[SHOULD] Add explicit prioritization within PR 1:** (a) mutation blocker, (b) config contract, (c) execute tool, (d) orchestrators. This ensures partial delivery still provides meaningful value.

7. **[COULD] Reduce LLM provider tests to `openai_compat` retry path only** (saves ~3 points). Defer full provider testing to a future SDK upgrade cycle.

8. **[COULD] Add `pytest-cov` to CI** with a 75% floor as a secondary, reproducible coverage signal alongside the manual reverse-TDD review.

9. **[COULD] Add a "Test Style Guide" note** recommending class-based test organization (following `test_executor.py`), descriptive docstrings, and consistent import patterns for new test files.

## Risk Register

| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| `_ProviderProxy` triggers real SDK client creation during test import, causing test failures without API keys | High | High | Mock `api_agent.agent.model._provider` in `conftest.py` before any orchestrator module import |
| ContextVar bleed between tests if pytest-asyncio uses session-scoped event loop | Medium | High | Mandate function-scoped `@pytest.mark.asyncio` for all new tests; verify `asyncio_mode` in `pyproject.toml` |
| Orchestrator tests become brittle if they depend on exact `RunResult.final_output` string formatting | Medium | Medium | Assert on structural properties (`ok=True`, `queries` is non-empty list) not exact string matches |
| Mocking httpx for introspection/OpenAPI fetches creates tight coupling to httpx internals | Medium | Medium | Use `respx` library (mock at transport level) or monkeypatch `execute_query`/`load_openapi_spec` at the function level instead of httpx internals |
| FR-006 ambiguity causes test to be written for wrong behavior, then immediately broken by a "fix" PR | Medium | Low | Resolve ambiguity in spec before implementation (Recommendation #2) |
| `openai_compat` retry-without-tools test requires mocking exception types from the OpenAI SDK | Low | Medium | Use a generic `Exception("tool calling not supported")` to trigger the retry path -- the code checks for "tool" or "function" in the error string (openai_compat.py line 53) |
| Recipe runner behavioral tests need a valid recipe with real steps; constructing these fixtures is non-trivial | Medium | Low | Create 2 canonical recipe fixtures (one GraphQL, one REST) in `tests/fixtures/` with known inputs and expected outputs |

## Final Verdict

**Approve with Conditions**

The spec is well-structured, correctly prioritized, and grounded in a real analysis of the codebase's coverage gaps. It can proceed to implementation after addressing these conditions:

1. **Add a Test Infrastructure section** covering provider mocking strategy, httpx mocking approach, and ContextVar isolation requirements. (Blocks PR 1 implementation.)

2. **Resolve the FR-006 / US2-4 discrepancy** between spec ("errors alongside partial data") and code (errors discard data). (Blocks GraphQL client test implementation.)

3. **Expand SC-002 mutation test variants** to include subscription operations and a test through the `_execute` MCP tool layer. (Can be addressed during implementation.)

All other recommendations are improvements that can be incorporated during implementation without requiring a spec revision.
