# Command Provider Tool Calling

## Problem

The command provider runs an external agent (`claude -p`, `codex`, any stdin→stdout agent) as the "LLM." These agents have their own tools internally. But Swival also has tools the agent can't access — MCP servers, A2A agents, skills. The agent needs a way to call those Swival-exclusive tools through a convention that's distinct from its own internal tool calling.

## Scope (v1)

v1 exposes only tools the external agent **cannot have on its own**:

- MCP tools (`mcp__*`)
- A2A tools (`a2a__*`)
- Skills (`use_skill`)

Explicitly **out of scope** for v1:

- `think`, `todo`, `snapshot` — tightly coupled to Swival's compaction and state machinery.
- `read_file`, `write_file`, `edit_file`, `grep`, `list_files`, `run_command` — the agent already has its own versions.
- A `--command-tools all` mode — exposing write/edit/run_command through the minimal command-provider system prompt without the normal operating instructions is a weaker safety envelope.

## Convention: XML Blocks with Call IDs

The agent requests a Swival tool by emitting a text block in its stdout:

```
<swival:call id="c1" name="mcp__db__query">
{"sql": "SELECT count(*) FROM users"}
</swival:call>
```

- `swival:` namespace won't collide with agent output
- `id` attribute is **required** — a short caller-chosen string. Correlates requests with results. Needed because the agent may call the same tool multiple times in one response.
- JSON body = tool arguments
- Multiple calls per response supported

### Results

Swival feeds results back in the transcript keyed by call id:

```
[swival_result id="c1" name="mcp__db__query"]
[{"count": 42}]
```

### System Prompt

Swival prepends tool instructions to the transcript when it has exposable tools:

```
In addition to your own tools, you have access to external tools provided by the
orchestrator. To call one, emit a block in your response:

<swival:call id="UNIQUE_ID" name="tool_name">
{"param": "value"}
</swival:call>

Each call must have a unique id (e.g. c1, c2, c3). Do NOT use your own tool-calling
mechanism for these — they must appear as literal text in your response. The
orchestrator will execute them and provide results in [swival_result] sections.

Continue working until you can give a final answer with no <swival:call> blocks.

Available external tools:

- mcp__db__query: Run a SQL query against the project database.
  Parameters: {"sql": string}
```

Tool descriptions are generated from the actual tool schemas — same `description` and `parameters` fields, rendered as text.

## Execution

### Post-Tool Bookkeeping

`handle_tool_call()` (agent.py:1414) only parses args, calls `dispatch()`, and returns `(tool_msg, tool_meta)`. The important bookkeeping happens **after** it returns, in `run_agent_loop()` (agent.py:4159–4256):

- `EVENT_TOOL_START` — emitted **before** `handle_tool_call()` (agent.py:4160)
- `handle_tool_call()` — executes the tool
- `EVENT_TOOL_FINISH` / `EVENT_TOOL_ERROR` — emitted **after** (agent.py:4183)
- Report logging (`report.record_tool_call()`) — after (agent.py:4202)
- Snapshot dirty tracking (`snapshot_state.mark_dirty()`) — after (agent.py:4219)
- Error guardrails (consecutive error counting, nudge/stop interventions) — after (agent.py:4225)

Extract only the **post-call** bookkeeping into a shared helper. `EVENT_TOOL_START` stays at the call site — it fires before execution, not after.

```python
def _post_tool_bookkeeping(
    tool_msg, tool_meta, turn, turn_offset,
    report, snapshot_state, consecutive_errors,
    verbose, _emit,
):
    """Post-tool-call bookkeeping shared by run_agent_loop() and command provider.

    Handles: post-call event emission, report logging, snapshot dirty tracking,
    error guardrail tracking.

    EVENT_TOOL_START is NOT included — callers emit it before execution.

    Returns list of intervention strings.
    """
    interventions = []
    name = tool_meta["name"]

    # Post-call events
    if tool_meta["succeeded"]:
        _emit(EVENT_TOOL_FINISH, {
            "name": name, "turn": turn, "elapsed": tool_meta["elapsed"],
        })
    else:
        _emit(EVENT_TOOL_ERROR, {
            "name": name, "turn": turn, "error": tool_msg["content"][:500],
        })

    # Report logging
    if report:
        report.record_tool_call(
            turn + turn_offset, name, tool_meta["arguments"],
            tool_meta["succeeded"], tool_meta["elapsed"],
            len(tool_msg["content"]),
            error=tool_msg["content"] if not tool_meta["succeeded"] else None,
        )

    # Snapshot dirty tracking
    if snapshot_state is not None:
        snapshot_state.mark_dirty(name)

    # Error guardrails
    result = tool_msg["content"]
    if result.startswith("error:"):
        canonical = _canonical_error(result)
        prev_err, prev_count = consecutive_errors.get(name, ("", 0))
        count = prev_count + 1 if canonical == prev_err else 1
        consecutive_errors[name] = (canonical, count)
        if count >= 3:
            interventions.append(
                f"STOP: `{name}` failed {count} times with the same error. "
                "Use different arguments or a different approach."
            )
            if report:
                report.record_guardrail(turn + turn_offset, name, "stop")
        elif count >= 2:
            interventions.append(
                f"IMPORTANT: `{name}` failed {count} times: {canonical}\n"
                "Fix the arguments or use a different approach."
            )
            if report:
                report.record_guardrail(turn + turn_offset, name, "nudge")
        if verbose:
            fmt.guardrail(name, count, canonical)
    else:
        consecutive_errors.pop(name, None)

    return interventions
```

`run_agent_loop()` is refactored to: emit `EVENT_TOOL_START`, call `handle_tool_call()`, then call `_post_tool_bookkeeping()`. The think/todo nudges stay in `run_agent_loop()` — they don't apply to the v1 command-provider tool set.

### Execution Loop

The outer agent loop's `turns` counter is plumbed into the command provider so that reports and events are keyed to the outer turn, not the inner retry round.

`_call_command_with_tools()` returns a **three-tuple**: `(msg, "stop", activity_summary)` where `activity_summary` is a list of `{"name": str, "succeeded": bool}` dicts (empty list if no tools were called). The `msg.content` is the **clean agent answer** with no metadata appended. The activity summary travels separately.

```python
def _call_command_with_tools(
    command_str, messages, tools_info,
    handle_tool_call_kwargs, bookkeeping_kwargs,
    outer_turn, outer_turn_offset,
    verbose, max_output_tokens, max_rounds=20,
):
    """Run command provider with Swival tool-calling support.

    outer_turn / outer_turn_offset: the turn counter from run_agent_loop(),
    so that reports and events are keyed to the outer turn index.

    Returns (synthetic_message, "stop", activity_summary).
    activity_summary is a list of {"name": str, "succeeded": bool} dicts.
    """
    parts = shlex.split(command_str)
    transcript_messages = list(messages)
    consecutive_errors = {}
    tool_activity = []
    response_text = ""

    for round_num in range(max_rounds):
        transcript = _render_transcript(transcript_messages)
        proc = subprocess.run(parts, input=transcript, capture_output=True,
                              text=True, timeout=300)
        if proc.returncode != 0:
            error_text = proc.stderr.strip() or proc.stdout.strip() or f"exit code {proc.returncode}"
            raise AgentError(f"command provider failed: {error_text}")

        response_text = proc.stdout.strip()
        if not response_text:
            raise AgentError("command provider returned empty output")

        calls = _parse_swival_calls(response_text)
        if not calls:
            break  # final answer — response_text is the clean answer

        # Append assistant response to internal transcript
        transcript_messages.append({"role": "assistant", "content": response_text})

        round_interventions = []
        for call_id, name, args in calls:
            if "_parse_error" in args:
                tool_msg = {
                    "role": "tool", "tool_call_id": call_id,
                    "content": f"error: invalid JSON in tool arguments: {args['_parse_error']}",
                }
                tool_meta = {"name": name, "arguments": None,
                             "elapsed": 0.0, "succeeded": False}
            else:
                tc = _make_tool_call_obj(call_id, name, args)

                # Pre-call event (matches run_agent_loop behavior)
                _emit = bookkeeping_kwargs["_emit"]
                _emit(EVENT_TOOL_START, {"name": name, "turn": outer_turn})

                tool_msg, tool_meta = handle_tool_call(tc, **handle_tool_call_kwargs)

            # Post-call bookkeeping (events, report, guardrails)
            intv = _post_tool_bookkeeping(
                tool_msg, tool_meta,
                turn=outer_turn,
                turn_offset=outer_turn_offset,
                consecutive_errors=consecutive_errors,
                **bookkeeping_kwargs,
            )
            round_interventions.extend(intv)

            tool_activity.append({
                "name": name,
                "succeeded": tool_meta["succeeded"],
            })

            transcript_messages.append({
                "role": "tool", "tool_call_id": call_id,
                "name": name, "content": tool_msg["content"],
            })

        if round_interventions:
            transcript_messages.append({
                "role": "user",
                "content": "\n\n".join(round_interventions),
            })

    # Apply output token limit to the clean answer only
    if max_output_tokens and max_output_tokens > 0:
        from .tokens import truncate_to_tokens
        response_text = truncate_to_tokens(response_text, max_output_tokens)

    return _make_synthetic_message(response_text), "stop", tool_activity
```

### Activity Summary in Outer History

The activity summary must not appear in the user-facing answer (`msg.content`) or compete with `max_output_tokens`. Instead, it's injected into the `messages` list as a separate context message **after** the assistant message, visible only to the LLM on follow-up turns.

In `run_agent_loop()`, the call site changes from:

```python
msg, finish_reason = call_llm(...)
```

to (for command provider with tools only):

```python
msg, finish_reason, cmd_activity = call_llm(...)  # third element only for command+tools
```

After `messages.append(msg.model_dump(...))` (line 4101), if `cmd_activity` is non-empty, inject a context message:

```python
if cmd_activity:
    lines = [f"  - {a['name']}: {'ok' if a['succeeded'] else 'error'}"
             for a in cmd_activity]
    messages.append({
        "role": "user",
        "content": (
            "[Context for follow-up: external tool calls made during "
            "the previous response:\n" + "\n".join(lines) + "\n]"
        ),
    })
```

This context message:
- Is **not** in `msg.content`, so it doesn't print to stdout or get emitted as `EVENT_TEXT_CHUNK`
- Doesn't compete with `max_output_tokens`
- Is visible to the LLM on the next turn (persisted in `messages`)
- Is **droppable by compaction** — requires updating `is_pinned()` (agent.py:615). Currently, `is_pinned()` returns `True` for all user turns except those starting with `_IMAGE_SYNTHETIC_PREFIX`. Without a change, this context message would be pinned forever and never reclaimed. Fix: extend `is_pinned()` to also return `False` for messages starting with `"[Context for follow-up:"`

The prefix `"[Context for follow-up:"` must be added to `SYNTHETIC_USER_PREFIXES` (agent.py:84) so that `continue_here._find_last_user_task()` skips it when looking for the latest real user task. Without this, the continue/resume logic would mistake the tool-activity note for the current task and write it into `.swival/continue.md`.

`call_llm()` always returns `(msg, finish_reason, cmd_activity)` where `cmd_activity` defaults to `[]` for all non-command paths. This is a three-value return change that affects every consumer of `call_llm` / `call_llm_fn`:

**Call sites that must be updated:**

| Location | Current pattern | Change |
|---|---|---|
| `run_agent_loop()` (agent.py:3907) | `msg, finish_reason = call_llm(...)` | Unpack three values, inject context message when `cmd_activity` is non-empty |
| `run_agent_loop()` retry (agent.py:3997) | `msg, finish_reason = call_llm(...)` | Same |
| `_summarize_turns()` (agent.py:858) | `resp, _ = call_llm_fn(...)` | `resp, _, _ = call_llm_fn(...)` |
| `_summarize_and_drop()` (agent.py:898+) | passes `call_llm_fn` through | No change (wrapper) |
| `continue_here._enrich_with_llm()` (continue_here.py:273) | `resp, _ = call_llm_fn(...)` | `resp, _, _ = call_llm_fn(...)` |
| `reviewer.py` (reviewer.py:165) | `msg, finish_reason = call_llm(...)` | `msg, finish_reason, _ = call_llm(...)` |
| `_call_llm_for_secondary` (agent.py:3741) | wrapper → `return call_llm(...)` | No change (passes through three-tuple) |

The secondary consumers (`_summarize_turns`, `_enrich_with_llm`, `reviewer`) never use the command provider, so `cmd_activity` will always be `[]` for them. But they still need to unpack the third value to avoid a `ValueError`.

## Parsing

The regex must handle attributes in either order (`id` before `name` or `name` before `id`) and tolerate extra whitespace. Rather than a single rigid pattern, use a two-step approach: match the tag boundaries, then extract attributes from the opening tag.

```python
# Match the full block: opening tag (with any attributes) → body → closing tag
_SWIVAL_BLOCK_RE = re.compile(
    r'<swival:call\s([^>]+)>\s*(\{.*?\})\s*</swival:call>',
    re.DOTALL,
)

# Extract individual attributes from the opening tag's attribute string
_ATTR_RE = re.compile(r'(\w+)="([^"]*)"')


def _parse_swival_calls(text):
    """Extract (call_id, tool_name, args_dict) tuples from agent output.

    Attribute order in the opening tag does not matter. Unknown attributes
    are ignored. Both id and name are required; blocks missing either are
    skipped.

    Malformed JSON in a block produces an entry with {"_parse_error": "..."}
    so the caller can feed an error result back to the agent.
    """
    results = []
    for m in _SWIVAL_BLOCK_RE.finditer(text):
        attr_str, args_json = m.group(1), m.group(2)
        attrs = dict(_ATTR_RE.findall(attr_str))

        call_id = attrs.get("id")
        name = attrs.get("name")
        if not call_id or not name:
            continue  # missing required attributes, skip

        try:
            args = json.loads(args_json)
        except json.JSONDecodeError as e:
            results.append((call_id, name, {"_parse_error": str(e)}))
            continue
        results.append((call_id, name, args))
    return results
```

This handles `<swival:call name="foo" id="c1">`, `<swival:call id="c1" name="foo">`, extra spacing, and even unexpected attributes like `<swival:call id="c1" name="foo" priority="high">` (unknown attrs are ignored).

## What Changes

| Component | Change |
|---|---|
| `agent.py` | Extract `_post_tool_bookkeeping()` from `run_agent_loop()` (post-call only; `EVENT_TOOL_START` stays at call site) |
| `agent.py` | `run_agent_loop()` calls the extracted helper (refactor, no behavior change) |
| `agent.py` | New `_call_command_with_tools()` with inner loop |
| `agent.py` | `_render_transcript()` handles `swival_result` messages |
| `agent.py` | `_COMMAND_PROVIDER_SYSTEM_PROMPT` gets conditional tool catalog |
| `agent.py` | `call_llm()` returns three-tuple `(msg, finish_reason, cmd_activity)`, `cmd_activity=[]` for all non-command paths |
| `agent.py` | `SYNTHETIC_USER_PREFIXES` — add `"[Context for follow-up:"` prefix |
| `agent.py` | `is_pinned()` — return `False` for `"[Context for follow-up:"` prefix (so compaction can drop it) |
| `agent.py` | `run_agent_loop()` unpacks three-tuple at both call sites, injects context message when `cmd_activity` is non-empty |
| `agent.py` | `_summarize_turns()` — unpack third value (`resp, _, _ = ...`) |
| `continue_here.py` | `_enrich_with_llm()` — unpack third value (`resp, _, _ = ...`) |
| `reviewer.py` | `call_llm()` call — unpack third value (`msg, finish_reason, _ = ...`) |

## What Doesn't Change

- `handle_tool_call()` — called as-is
- `dispatch()` — called by `handle_tool_call()` as normal
- `_SyntheticMessage` — no changes (no new slots, content stays clean)
- `_call_llm_for_secondary` wrapper — passes through three-tuple transparently
- Tool definitions — same schemas, rendered as text for the prompt
- Compaction, safety, sandboxing

## v1 Limitations (Acknowledged)

1. **No full inner history in outer messages.** Follow-up turns see a terse activity summary (context message), not the full tool results. Sufficient for continuity, not for replay.
2. **No think/todo/snapshot.** The command provider can't use Swival's persistence-across-compaction tools.
3. **No file I/O tools.** The agent must use its own. If the agent has none (raw LLM wrapper), it can't do file operations through Swival in v1.
4. **Agent compliance is best-effort.** The system prompt instructs the agent to use the XML convention, but some agents may ignore it. Failure mode is graceful: no blocks parsed → plain text answer.

## Implementation Order

1. Extract `_post_tool_bookkeeping()` from `run_agent_loop()` — post-call only, `EVENT_TOOL_START` stays at call site — refactor + tests to verify no behavior change
2. `_parse_swival_calls(text)` — order-insensitive attribute parser + tests (both attribute orders, extra attrs, missing attrs, malformed JSON)
3. `_render_swival_tool_catalog(tool_schemas)` — text catalog from tool defs
4. `_call_command_with_tools()` — inner loop with bookkeeping, outer turn plumbing, clean three-tuple return
5. `_render_transcript()` — handle `swival_result` messages
6. System prompt — inject catalog when exposable tools exist
7. `call_llm()` — return three-tuple `(msg, finish_reason, [])` for all paths; route to `_call_command_with_tools()` when applicable
8. Update all secondary `call_llm` / `call_llm_fn` consumers to unpack three values: `_summarize_turns()`, `_enrich_with_llm()`, `reviewer.py`
9. Add `"[Context for follow-up:"` to `SYNTHETIC_USER_PREFIXES` and update `is_pinned()` to unpin it
10. `run_agent_loop()` — unpack three-tuple at both call sites, inject context message when `cmd_activity` is non-empty
11. Tests — mock commands emitting `<swival:call>` blocks (both attr orders), verify bookkeeping against outer turn, verify context message in `messages` but not in returned answer string, verify continue_here skips the synthetic context message, verify `is_pinned()` returns `False` for `[Context for follow-up:` messages (i.e. compaction can actually drop them)
