
---
title: LLM Integration Documentation
description: LLM and AI integration architecture
category: llm-integration
status: stable
last_updated: 2025-01-15
version: 1.0.0
tags:
  - level: advanced
  - component: llm
  - status: stable
---

# LLM Integration Documentation

Complete documentation for LLM and AI integration in Kollabor CLI.

## Overview

The LLM Integration layer provides comprehensive AI capabilities, including API communication, tool execution, conversation management, agent orchestration, and MCP integration.

## Documents

### LLM Service Architecture
**[LLM Service Architecture](llm-service.md)** - Complete LLM service orchestration

**Description**: Comprehensive LLM service architecture covering API communication, tool execution, conversation management, profile management, agent orchestration, and MCP integration.

**Topics**:
- LLMService - Main orchestration
- APICommunicationService - HTTP and streaming
- ToolExecutor - Native and MCP tools
- ConversationManager - History and context
- ProfileManager - LLM profiles
- AgentManager - Agent orchestration
- MCPIntegration - MCP servers
- ResponseParser and ResponseProcessor
- MessageDisplayService

**Level**: Advanced  
**Status**: Stable

**Key Components**:
- `core/llm/llm_service.py` - Main service
- `core/llm/api_communication_service.py` - API communication
- `core/llm/tool_executor.py` - Tool execution
- `core/llm/conversation_manager.py` - Conversation history
- `core/llm/profile_manager.py` - Profile management
- `core/llm/agent_manager.py` - Agent orchestration
- `core/llm/mcp_integration.py` - MCP integration

**Related**:
- [LLM Message Flow](../reference/llm-message-flow.md)
- [Tool Execution System](./tool-execution.md)
- [MCP Integration](./mcp-integration.md)

---

## System Architecture

```mermaid
graph TB
    subgraph "LLM Integration Layer"
        LS[LLMService<br/>Orchestrator]
        ACS[APICommunicationService<br/>HTTP/Streaming]
        TE[ToolExecutor<br/>Native/MCP Tools]
        CM[ConversationManager<br/>History/Context]
        PM[ProfileManager<br/>LLM Profiles]
        AM[AgentManager<br/>Agent Orchestration]
        MI[MCPIntegration<br/>MCP Servers]
        RP[ResponseParser<br/>Response Parsing]
        RProc[ResponseProcessor<br/>Tool Calls]
        MDS[MessageDisplayService<br/>Display]
    end
    
    subgraph "External Systems"
        LLM[LLM APIs]
        MCP[MCP Servers]
        Agents[Subagents]
    end
    
    subgraph "Integration Points"
        EB[Event Bus]
        TR[Terminal Renderer]
        CL[Conversation Logger]
    end
    
    LS --> ACS
    LS --> TE
    LS --> CM
    LS --> PM
    LS --> AM
    LS --> MI
    
    ACS --> LLM
    TE --> MCP
    TE --> Agents
    
    CM --> CL
    PM --> ACS
    AM --> Agents
    
    LS --> RP
    RP --> RProc
    RProc --> TE
    
    TE --> MDS
    MDS --> TR
    
    LS --> EB
```

---

## Key Components

### LLMService
**Main orchestrator for all LLM interactions**

**Responsibilities**:
- Queue and process user requests
- Coordinate between specialized components
- Manage thinking animations
- Handle request/response cycles

**Documentation**: [LLM Service Architecture](llm-service.md#llmservice)

---

### APICommunicationService
**Handles HTTP communication with LLM APIs**

**Responsibilities**:
- HTTP request handling
- Streaming response processing
- Authentication and authorization
- API key management

**Documentation**: [LLM Service Architecture](llm-service.md#apicommunicationservice)

---

### ToolExecutor
**Executes LLM tool calls**

**Responsibilities**:
- Native tool execution
- MCP tool execution
- Permission gates and user approval
- Tool result formatting

**Documentation**: [LLM Service Architecture](llm-service.md#toolexecutor)

---

### ConversationManager
**Manages conversation history and context**

**Responsibilities**:
- Message history tracking
- Context window management
- Token estimation and trimming
- Metadata management

**Documentation**: [LLM Service Architecture](llm-service.md#conversationmanager)

---

### ProfileManager
**Manages LLM profiles and configuration**

**Responsibilities**:
- LLM profile management
- Profile switching
- Environment variable configuration
- API endpoint configuration

**Documentation**: [LLM Service Architecture](llm-service.md#profilemanager)

---

### AgentManager
**Manages subagent discovery and orchestration**

**Responsibilities**:
- Agent discovery
- Agent task orchestration
- Agent specialization
- Agent result aggregation

**Documentation**: [LLM Service Architecture](llm-service.md#agentmanager)

---

### MCPIntegration
**Manages MCP (Model Context Protocol) integration**

**Responsibilities**:
- MCP server management
- Tool discovery from MCP
- Tool execution on MCP servers
- Connection handling

**Documentation**: [LLM Service Architecture](llm-service.md#mcpintegration)

---

## Native Tools

Available native tools that can be called by LLM:

| Tool | Description | Parameters |
|------|-------------|-------------|
| `terminal` | Execute terminal commands | `command: str` |
| `file_read` | Read file contents | `filepath: str` |
| `file_write` | Write to file | `filepath: str, content: str` |
| `grep` | Search for patterns | `pattern: str, path: str` |
| `file_search` | Search files | `query: str, path: str` |

**Documentation**: [LLM Service Architecture](llm-service.md#native-tool-handlers)

---

## Supported LLM Providers

### Default Profile
- **API**: Local LLM server
- **Model**: qwen/qwen3-4b
- **Features**: Full tool support, streaming

### Claude Profile
- **API**: Anthropic Claude API
- **Model**: claude-3-sonnet-20240229
- **Features**: Full tool support, 200K context window

### GPT-4 Profile
- **API**: OpenAI GPT-4 API
- **Model**: gpt-4
- **Features**: Full tool support, 8K context window

**Documentation**: [LLM Service Architecture](llm-service.md#profile-manager)

---

## Tool Execution Flow

```mermaid
sequenceDiagram
    participant LLM as LLM Service
    participant Parser as Response Parser
    participant Executor as Tool Executor
    participant User as User
    participant Native as Native Tools
    participant MCP as MCP Servers
    
    LLM->>Parser: Parse LLM response
    Parser-->>LLM: Extracted tools
    
    LLM->>Executor: Execute tools
    
    loop Each tool
        Executor->>Executor: Check permission
        alt Permission required
            Executor->>User: Request approval
            User-->>Executor: Approve/Deny
        end
        
        alt Native tool
            Executor->>Native: Execute
            Native-->>Executor: Result
        else MCP tool
            Executor->>MCP: Execute
            MCP-->>Executor: Result
        end
        
        Executor-->>LLM: Tool result
    end
    
    LLM->>LLM: Display results
```

---

## Configuration

### LLM Service
```json
{
  "llm": {
    "max_queue_size": 10,
    "overflow_strategy": "discard_oldest",
    "system_prompt": "You are a helpful AI assistant."
  }
}
```

### API Communication
```json
{
  "core": {
    "llm": {
      "api_url": "http://localhost:1234/v1/chat/completions",
      "api_key": null,
      "model": "qwen/qwen3-4b",
      "temperature": 0.7,
      "max_tokens": 4000
    }
  }
}
```

### Tool Execution
```json
{
  "llm": {
    "tool_permission_required": false,
    "permission_timeout": 60
  }
}
```

### Conversation Management
```json
{
  "llm": {
    "max_history": 90,
    "context_window": 100000
  }
}
```

---

## Environment Variables

Supported environment variables for LLM configuration:

| Variable | Description | Example |
|----------|-------------|---------|
| `KOLLABOR_CLAUDE_TOKEN` | Claude API key | `sk-ant-...` |
| `ANTHROPIC_API_KEY` | Anthropic API key | `sk-ant-...` |
| `OPENAI_API_KEY` | OpenAI API key | `sk-...` |
| `LLM_API_URL` | Custom API endpoint | `http://localhost:1234/v1/chat/completions` |
| `LLM_API_KEY` | Custom API key | `your-api-key` |
| `LLM_MODEL` | Model name | `qwen/qwen3-4b` |
| `LLM_TEMPERATURE` | Sampling temperature | `0.7` |
| `LLM_MAX_TOKENS` | Max response tokens | `4000` |

**Documentation**: [APICommunicationService](llm-service.md#environment-variable-handling)

---

## Tool Call Format

LLM can call tools using XML-like syntax:

```
<tool_name>({"param1": "value1", "param2": "value2"})</tool_name>
```

**Example**:
```
Let me check the file.

<file_read({"filepath": "README.md"})</file_read>

Now let me search for TODOs.

<grep({"pattern": "TODO", "path": "."})</grep>
```

**Documentation**: [ResponseParser](llm-service.md#response-parser)

---

## MCP Integration

### What is MCP?
**Model Context Protocol (MCP)** - A protocol for connecting AI models to external tools and data sources.

### Features
- Tool discovery from MCP servers
- Standardized tool execution
- Resource access
- Secure connection handling

### Supported MCP Servers
Any MCP-compliant server can be integrated:
- Filesystem tools
- Database access
- Git operations
- External APIs

**Documentation**: [MCPIntegration](llm-service.md#mcp-integration)

---

## Performance Optimization

### Streaming Responses
- Enable streaming for real-time response
- Reduces perceived latency
- Updates thinking display with content

### Context Window Management
- Automatic trimming when exceeds limit
- Keeps recent messages
- Prioritizes system and user messages

### Tool Execution
- Async tool execution
- Parallel tool execution when possible
- Result caching

### Conversation History
- Configurable max history (default: 90 messages)
- Efficient storage and retrieval
- Token estimation for trimming

---

## Troubleshooting

### API Connection Issues
**Symptoms**: "API error", "Connection refused"

**Solutions**:
1. Check API URL is correct
2. Verify API key is set
3. Test network connectivity
4. Check API service is running

**Documentation**: [LLM Service Architecture](llm-service.md#issue-api-connection-errors)

---

### Tool Execution Failures
**Symptoms**: Tool returns error or "Tool not found"

**Solutions**:
1. Verify tool name matches
2. Check tool parameters are correct
3. Review tool execution logs
4. Check MCP server connection

**Documentation**: [LLM Service Architecture](llm-service.md#issue-tool-execution-fails)

---

### Context Window Issues
**Symptoms**: "Conversation too long", trimmed messages

**Solutions**:
1. Increase context window in config
2. Reduce max_history
3. Clear old messages
4. Use summary tools

**Documentation**: [LLM Service Architecture](llm-service.md#issue-context-window-exceeded)

---

## Related Documentation

### Message Flow
- [LLM Message Flow](../reference/llm-message-flow.md) - Complete message display flow
- [Resume Message Flow](../reference/resume-message-flow.md) - Resume flow

### Core Systems
- [Input Handling System](../core-systems/input-handling.md) - Input processing
- [Terminal Rendering System](../core-systems/terminal-rendering.md) - Display system

### Guides
- [Performance Guide](../guides/performance.md) - Optimization techniques
- [Security Guide](../guides/security.md) - Security best practices

---

## Glossary

**LLM**: Large Language Model - AI model for text generation

**Streaming**: Real-time response generation, displayed as it arrives

**Tool Call**: LLM requests execution of a specific tool/function

**MCP**: Model Context Protocol - Protocol for AI tool integration

**Context Window**: Maximum amount of text LLM can process

**Profile**: Configuration for a specific LLM provider

**Agent**: Specialized AI assistant for specific tasks

**Permission Gate**: User approval mechanism for tool execution

---

**Last Updated**: 2025-01-15  
**Status**: Stable  
**Version**: 1.0.0
