# Phase D Agent Prompt: QueueProcessor Extraction

## Context

You are continuing the llm_service.py decomposition (Phase 4.5 of the monorepo restructure).

Phases A-C are COMPLETE:
- kollabor/llm/background_task_manager.py (Phase A)
- kollabor/llm/system_prompt_builder.py (Phase B)
- kollabor/llm/streaming_handler.py (Phase B)
- kollabor/llm/native_tools_handler.py (Phase B)
- kollabor/llm/session_manager.py (Phase C)

Current state: llm_service.py is 2031 lines. Target: ~1200 lines.

Read the full spec: docs/specs/llm-service-refactor-spec.md
Read Phase B reference: kollabor/llm/streaming_handler.py (good example of the pattern)

## Your Task: Phase D - QueueProcessor

This is the HARDEST extraction because:
1. _process_message_batch and _continue_conversation are ~300 lines of near-identical code that must be DEDUPED
2. QueueProcessor needs access to many LLMService fields (shared state problem)
3. The code touches many subsystems (API, tools, events, display, logging)

### Step 1: Create ConversationState dataclass

File: kollabor/llm/conversation_state.py

```python
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional
from kollabor_events.data_models import ConversationMessage

@dataclass
class ConversationState:
    """Shared mutable state between LLMService and QueueProcessor.

    Both LLMService and QueueProcessor hold a reference to the SAME instance.
    Changes made by either side are immediately visible to the other.
    """
    conversation_history: List[ConversationMessage] = field(default_factory=list)
    session_stats: Dict[str, Any] = field(default_factory=dict)
    stats: Dict[str, Any] = field(default_factory=dict)
    current_parent_uuid: Optional[str] = None
    pending_tools: List[Dict[str, Any]] = field(default_factory=list)
    question_gate_active: bool = False
    question_gate_enabled: bool = True
    current_processing_tokens: int = 0
    processing_start_time: Optional[float] = None
    native_tool_calling_enabled: bool = True
```

### Step 2: Create QueueProcessor

File: kollabor/llm/queue_processor.py

Methods to extract from llm_service.py (find by name):
- _enqueue_with_overflow_strategy (line ~462)
- _process_queue (line ~1167)
- _process_message_batch (line ~1225)
- _continue_conversation (line ~1523)

Instance vars to move:
- processing_queue (asyncio.Queue)
- max_queue_size (int)
- dropped_messages (int)
- is_processing (bool)
- turn_completed (bool)
- cancel_processing (bool)
- cancellation_message_shown (bool)

#### CRITICAL: DEDUP _process_message_batch and _continue_conversation

These two methods are ~300 lines each and share ~90% of their logic.
The ONLY difference:
- _process_message_batch: adds user messages to conversation_history FIRST, then does the LLM turn
- _continue_conversation: skips the user message step, goes straight to the LLM turn

Extract the shared logic into `_execute_llm_turn()`:

```python
async def _execute_llm_turn(self, user_message: str = None):
    """Execute a single LLM turn (shared between process_message_batch and continue_conversation).

    Args:
        user_message: If provided, add as user message before calling LLM.
                     If None, this is a continuation turn (no new user message).
    """
    # If user_message provided, add to conversation history
    if user_message:
        self._add_message_fn(
            ConversationMessage(role="user", content=user_message)
        )

    # === SHARED LOGIC (both methods do this identically) ===
    # 1. Start thinking animation + token estimation
    # 2. Call LLM API
    # 3. Update token stats from API response
    # 4. Check for native tool calls -> execute if present -> return (continue loop)
    # 5. Parse XML response
    # 6. Update turn_completed, stats
    # 7. Brief "Receiving..." display
    # 8. Question gate check
    # 9. Execute XML tools (if not question gate)
    # 10. Emit LLM_RESPONSE event, check force_continue
    # 11. Display response via message_display_service
    # 12. Log assistant message
    # 13. Add to conversation history
    # 14. Log and batch tool results
```

Then _process_message_batch becomes:
```python
async def _process_message_batch(self, messages):
    combined = "\n".join(messages)
    await self._execute_llm_turn(user_message=combined)
```

And _continue_conversation becomes:
```python
async def _continue_conversation(self):
    await self._execute_llm_turn(user_message=None)
```

#### Constructor

```python
class QueueProcessor:
    def __init__(
        self,
        state: ConversationState,
        task_config,
        api_service,
        tool_executor,
        response_parser,
        message_display_service,
        renderer,
        config,
        event_bus,
        conversation_logger,
        streaming_handler,
        native_tools_handler,
        task_manager,
        queue_metrics: dict,
        add_message_fn,  # callback to LLMService._add_conversation_message
    ):
```

Note: `add_message_fn` is a callback because _add_conversation_message stays on LLMService
(it touches both conversation_manager and conversation_history).

#### Key references for _execute_llm_turn

The deduped method needs access to:
- self.state (ConversationState) for conversation_history, stats, session_stats, current_parent_uuid, pending_tools, question_gate_active/enabled, current_processing_tokens, processing_start_time, native_tool_calling_enabled
- self._call_llm_fn -> calls streaming_handler.call_llm(...)
- self._execute_native_tools_fn -> calls native_tools_handler.execute_tool_calls(tool_executor)
- self.api_service -> get_last_token_usage, has_pending_tool_calls, get_last_tool_calls, format_tool_result, model
- self.tool_executor -> execute_all_tools, format_result_for_conversation
- self.response_parser -> parse_response, get_all_tools
- self.message_display_service -> display_complete_response, display_cancellation_message, display_error_message
- self.renderer -> update_thinking, clear_active_area
- self.config -> get("kollabor.llm.processing_delay"), get("kollabor.llm.thinking_delay")
- self.event_bus -> emit_with_hooks
- self.conversation_logger -> log_assistant_message, log_system_message
- self._add_message_fn -> callback
- get_token_io_state() -> imported from kollabor.io.status.core_widgets

For the streaming_handler call, make a helper:
```python
async def _call_llm(self):
    return await self._streaming_handler.call_llm(
        conversation_history=self.state.conversation_history,
        max_history=self._max_history,
        native_tools=self._native_tools_handler.tools,
        mcp_discovery_complete=self._native_tools_handler.discovery_complete,
        is_cancelled_fn=lambda: self.cancel_processing,
    )
```

### Step 3: Wire into llm_service.py

In LLMService.__init__, AFTER creating all other sub-components:
```python
# Shared conversation state (referenced by both LLMService and QueueProcessor)
self._conversation_state = ConversationState(
    conversation_history=self.conversation_history,
    session_stats=self.session_stats,
    stats=self.stats,
    current_parent_uuid=self.current_parent_uuid,
    pending_tools=self.pending_tools,
    question_gate_active=self.question_gate_active,
    question_gate_enabled=self.question_gate_enabled,
    current_processing_tokens=self.current_processing_tokens,
    processing_start_time=self.processing_start_time,
    native_tool_calling_enabled=self.native_tool_calling_enabled,
)
```

WAIT - this won't work for primitives (bool, int, str) because Python doesn't share
primitive references. Only mutable containers (list, dict) are shared by reference.

BETTER APPROACH: Don't use ConversationState. Instead, have QueueProcessor access
LLMService fields via a reference to the LLMService instance itself, OR pass the
mutable containers (conversation_history, session_stats, stats, _queue_metrics) by
reference and use properties/getters for the primitives.

SIMPLEST APPROACH that matches the established pattern:
- Pass mutable containers by reference (conversation_history, session_stats, stats, pending_tools, _queue_metrics)
- For primitives (turn_completed, cancel_processing, is_processing, etc.), KEEP them on QueueProcessor
- LLMService reads them via self._queue_processor.is_processing, self._queue_processor.cancel_processing, etc.
- The fields that LLMService currently exposes (cancel_processing, is_processing, etc.) become properties forwarding to QueueProcessor

```python
self._queue_processor = QueueProcessor(
    conversation_history=self.conversation_history,
    session_stats=self.session_stats,
    stats=self.stats,
    pending_tools=self.pending_tools,
    queue_metrics=self._queue_metrics,
    task_config=self.task_config,
    api_service=self.api_service,
    tool_executor=self.tool_executor,
    response_parser=self.response_parser,
    message_display_service=self.message_display_service,
    renderer=self.renderer,
    config=self.config,
    event_bus=self.event_bus,
    conversation_logger=self.conversation_logger,
    streaming_handler=self._streaming,
    native_tools_handler=self._native_tools,
    task_manager=self._task_manager,
    add_message_fn=self._add_conversation_message,
    max_history=self.max_history,
    question_gate_enabled=self.question_gate_enabled,
)
```

Forwarding properties on LLMService:
```python
@property
def processing_queue(self):
    return self._queue_processor.processing_queue

@property
def is_processing(self):
    return self._queue_processor.is_processing

@property
def cancel_processing(self):
    return self._queue_processor.cancel_processing

@cancel_processing.setter
def cancel_processing(self, value):
    self._queue_processor.cancel_processing = value

@property
def turn_completed(self):
    return self._queue_processor.turn_completed

@turn_completed.setter
def turn_completed(self, value):
    self._queue_processor.turn_completed = value

@property
def cancellation_message_shown(self):
    return self._queue_processor.cancellation_message_shown

@cancellation_message_shown.setter
def cancellation_message_shown(self, value):
    self._queue_processor.cancellation_message_shown = value

@property
def dropped_messages(self):
    return self._queue_processor.dropped_messages

@dropped_messages.setter
def dropped_messages(self, value):
    self._queue_processor.dropped_messages = value

@property
def max_queue_size(self):
    return self._queue_processor.max_queue_size

@property
def current_processing_tokens(self):
    return self._queue_processor.current_processing_tokens

@current_processing_tokens.setter
def current_processing_tokens(self, value):
    self._queue_processor.current_processing_tokens = value

@property
def processing_start_time(self):
    return self._queue_processor.processing_start_time

@processing_start_time.setter
def processing_start_time(self, value):
    self._queue_processor.processing_start_time = value

# pending_tools, question_gate_active are shared lists/bools that need forwarding too
@property
def pending_tools(self):
    return self._queue_processor.pending_tools

@pending_tools.setter
def pending_tools(self, value):
    self._queue_processor.pending_tools = value

@property
def question_gate_active(self):
    return self._queue_processor.question_gate_active

@question_gate_active.setter
def question_gate_active(self, value):
    self._queue_processor.question_gate_active = value
```

Delegation:
```python
async def _enqueue_with_overflow_strategy(self, message):
    await self._queue_processor.enqueue(message)

async def _process_queue(self):
    await self._queue_processor.process_queue()

async def _process_message_batch(self, messages):
    await self._queue_processor.process_message_batch(messages)

async def _continue_conversation(self):
    await self._queue_processor.continue_conversation()
```

### Differences between _process_message_batch and _continue_conversation

CONFIRMED identical sections (diff analysis):
1. Token estimation + thinking animation start (identical)
2. _call_llm() call (identical)
3. Token usage update from API (identical - except _process_message_batch stores usage_stats in log)
4. Native tool call check + execution (identical except log message string)
5. Response parsing (identical)
6. Turn completion + stats update (near-identical - _process_message_batch updates stats, _continue does too but at different position)
7. "Receiving..." display (identical)
8. Question gate check (identical except log message string)
9. LLM_RESPONSE event emission (identical)
10. force_continue check (identical except log message string)
11. Display response (identical)
12. Log assistant message (near-identical - _process_message_batch includes token usage_stats, _continue only includes thinking_duration)
13. Add to conversation history (identical)
14. Log tool results + batch (identical)
15. CancelledError handling (identical except log message string)
16. Exception handling (_process_message_batch has more, _continue has less)

The ONLY real difference:
- _process_message_batch: first adds combined user message to history, then does the turn
- _continue_conversation: skips user message, goes straight to turn

Minor differences (handle with parameters):
- Log messages say "continue" vs "batch" (use is_continuation bool for logging)
- _process_message_batch usage_stats includes input/output tokens, _continue only thinking_duration
- _process_message_batch catches exceptions and marks turn_completed=True, _continue doesn't

## Execution Steps

1. Read llm_service.py current state (methods at lines ~462, ~1167, ~1225, ~1523)
2. Create kollabor/llm/conversation_state.py (SKIP - use direct field passing instead)
3. Create kollabor/llm/queue_processor.py with DEDUPED _execute_llm_turn
4. Wire QueueProcessor into llm_service.py:
   a. Add import
   b. Remove queue-related instance vars from __init__ (move to QueueProcessor)
   c. Create QueueProcessor instance after all deps exist
   d. Add forwarding properties for fields accessed externally
   e. Replace method bodies with delegation
   f. Remove extracted method bodies
5. Verify tests

## Verification

After wiring:
```bash
uv run pytest tests/unit/llm/test_task_management.py -x -q  # must stay 15/15
uv run python -c "from kollabor.llm.llm_service import LLMService; print('ok')"
uv run pytest tests/unit/llm/ -q --ignore=tests/unit/llm/test_integrated_session_ids.py
```

Expected: 28 failed, 49 passed (same baseline - 0 new failures).

## Key Gotchas from Previous Phases

- Init ordering matters: create sub-component AFTER all its deps exist in __init__
- Tests patch kollabor.llm.llm_service.Path - keep that import
- Use property forwarding for fields with many internal references
- callback pattern for _add_conversation_message (stays on LLMService)
- Mutable containers (list, dict) share by reference; primitives need properties
