# CODEBUDDY.md

This file provides guidance to CodeBuddy Code when working with code in this repository.

## Project Overview

XGSE (Extreme General Skill Engine) is an implementation of the Anthropic Agent Skills Protocol. It provides a modular skill framework that supports dynamic skill discovery, progressive context loading, planning, and task execution. The system implements a multi-level progressive context loading mechanism to efficiently manage skill discovery and execution.

## Architecture

### Core Components

**Multi-Level Progressive Context Loading:**
1. **Level 1 (Metadata)**: Load all skill names and descriptions
2. **Level 2 (Retrieval)**: Retrieve and load SKILL.md when relevant with the query
3. **Level 3 (Resources)**: Load additional files (references, scripts, resources) only when referenced in SKILL.md
4. **Level 4 (Analysis|Planning|Execution)**: Analyze the loaded skill context, plan execution steps, and run necessary scripts

**Engine Layer** (`xgse/engine/`):
- `skill_engine.py`: Core XGASkillEngine class that orchestrates skill loading, retrieval, planning, and execution
- `loader.py`: SkillLoader for loading skills from directories (single skill, root directory, or multiple skills)
- `retrieve.py`: Retriever for finding relevant skills using keyword or semantic search
- `schema.py`: Data structures (SkillSchema, SkillContext, SkillFile, ExecutionResult) and SkillSchemaParser
- `prompts.py`: Prompt templates for skill planning, task generation, and implementation
- `spec.py`: Spec class for storing and dumping execution plans

**Sandbox Layer** (`xgse/sandbox/`):
- `sandbox_base.py`: Abstract base class SkillSandbox defining the sandbox interface
- `sandbox_factory.py`: Factory function create_skill_sandbox for creating sandbox instances
- `e2b_sandbox.py`: E2BSkillSandbox implementation using E2B code interpreter

**Utils Layer** (`xgse/utils/`):
- `llm_client.py`: LLMClient wrapping LiteLLM for LLM API calls with retry logic, streaming, and Langfuse integration
- `setup_env.py`: Logging setup and Langfuse initialization
- `utils.py`: Utility functions for file operations, package installation, text extraction

### Skill Structure

Skills follow the Anthropic Skills Protocol with:
- **SKILL.md**: Main skill file with YAML frontmatter (name, description, version, author, tags) and detailed instructions
- **scripts/**: Python scripts that can be executed by the skill
- **references/**: Markdown documentation files (.md)
- **resources/**: Other files (data files, templates, etc.)
- **LICENSE.txt**: Optional license file

The SkillSchemaParser parses SKILL.md YAML frontmatter and categorizes files by extension into scripts, references, and resources.

### Execution Flow

1. **Skill Discovery**: Skills are loaded from directories using SkillLoader
2. **Skill Retrieval**: Retriever finds relevant skills using keyword or semantic search
3. **Skill Planning**: LLM analyzes SKILL.md, references, and scripts to create an execution plan
4. **Task Generation**: LLM generates specific tasks and identifies required scripts/references/resources
5. **Implementation**: LLM generates code blocks or script invocations based on tasks
6. **Execution**: Code blocks are executed either in sandbox (E2B) or local environment
7. **Result Return**: Execution results are aggregated and returned

### Sandbox vs Local Execution

The engine supports two execution modes:
- **Sandbox (use_sandbox=True)**: Secure execution in E2B sandbox with automatic dependency installation, file upload, and isolation
- **Local (use_sandbox=False)**: Direct execution in local environment (requires manual confirmation, recommended only for trusted skills)

## Development Commands

### Environment Setup

```bash
# Create virtual environment
python -m venv .venv
source .venv/bin/activate

# Install pip in .venv (required for uv-managed environments)
python -m ensurepip --upgrade

# Install dependencies with uv
uv sync
```

### Environment Configuration

Create `.env` file with the following variables:

```bash
# Logging
LOG_LEVEL=INFO
LOG_FILE=log/xgse.log
LOG_ENABLE=True

# LLM Configuration
LLM_MODEL=openai/qwen3-235b-a22b
LLM_API_BASE=https://dashscope.aliyuncs.com/compatible-mode/v1
LLM_API_KEY=your_api_key_here
LLM_MAX_TOKENS=16384
LLM_TEMPERATURE=0
LLM_MAX_RETRIES=1
LLM_STREAM=False

# Sandbox Configuration (optional)
SANDBOX_TYPE=e2b
SANDBOX_WORK_DIR=/skill_workspace
SANDBOX_TIMEOUT=300

# Langfuse (optional)
LANGFUSE_PUBLIC_KEY=your_public_key
LANGFUSE_SECRET_KEY=your_secret_key
LANGFUSE_HOST=https://cloud.langfuse.com
```

### Running Skills

```bash
# Run the example skill runner
example-run-skill

# Or run directly
python examples/skill/run_skill.py
```

### Building the Package

```bash
# Build the package
uv build

# The build output is in dist/
```

### Testing

Currently no automated test suite exists. Manual testing is done via the example runner in `examples/skill/run_skill.py`.

## Key Design Patterns

**Skill Loading**: Skills are copied to a working directory during initialization to provide isolation. The loader can handle single skills, a root directory containing multiple skills, or a list of specific skill directories.

**Retrieval Strategy**: The retriever supports both keyword-based (simple text matching with weights) and semantic-based (TF-IDF-like approach) retrieval. Semantic search is more sophisticated but currently uses a simple algorithm that can be extended with embeddings.

**Prompt Engineering**: The system uses structured prompts with XML-like tags (`<QUERY>`, `<PLAN>`, `<SCRIPTS>`, etc.) to guide LLM responses through the planning and execution phases.

**Progressive Context**: Context is loaded progressively - only SKILL.md is loaded initially, then references/scripts/resources are loaded only when the LLM determines they're needed. This minimizes token usage while maintaining comprehensive capabilities.

**Execution Result Handling**: The ExecutionResult dataclass captures success status, output, and messages. Multiple code blocks can be executed sequentially, with execution stopping on failure.

**LLM Integration**: The LLMClient wraps LiteLLM and provides:
- Automatic retry with exponential backoff
- Streaming support
- Langfuse integration for observability
- Model-specific parameter handling (Claude, OpenAI, Bedrock, etc.)
- Cache control for Anthropic models

## Important Implementation Details

**Skill Execution Analysis**: When executing scripts, the LLM is used to analyze the script content and parameters to extract:
- The correct shell command to run
- Required Python packages from import statements
- Proper parameter formatting

This happens in `skill_engine.py:_analyse_code_block()` and uses an LLM call to ensure correct command construction.

**File Path Handling**: All file paths in sandbox execution are converted to be relative to the sandbox work directory. In local execution, paths are relative to the host work directory.

**Spec Dumping**: After each skill run, the execution plan, tasks, and implementation are dumped to `{work_dir}/.spec/` as plan.md, tasks.md, and implementation.md for debugging and reproducibility.

**Dependency Management**: Required packages are automatically installed in the sandbox before execution. In local mode, packages are installed directly using pip.

**Error Handling**: The engine has comprehensive error handling at multiple levels - skill loading, LLM calls, script execution, and sandbox operations all have try-catch blocks with detailed logging.

## Skill Development

To create a new skill:

1. Create a directory with your skill name
2. Add `SKILL.md` with YAML frontmatter:
   ```yaml
   ---
   name: Your Skill Name
   description: A brief description of what your skill does (max 1024 chars)
   version: 0.1.0
   author: Your Name
   tags: [tag1, tag2]
   ---
   ```

3. Add detailed instructions in SKILL.md after the frontmatter
4. Add executable scripts to `scripts/` directory
5. Add reference documentation to `references/` directory (optional)
6. Add resource files to `resources/` directory (optional)

The skill loader will automatically categorize files by extension when loading the skill.

## Common Issues

**Pip installation in .venv**: If using `uv` for dependency management, you may need to manually install pip in the .venv before running skills locally:
```bash
source .venv/bin/activate
python -m ensurepip --upgrade
```

**E2B Sandbox API Key**: To use sandbox execution, set `E2B_API_KEY` in your environment or `.env` file.

**LLM API Key**: Set `LLM_API_KEY` for the model you're using (e.g., OpenAI, Anthropic, Alibaba Cloud).

**Path Resolution**: Skills are copied to the work directory during initialization, so paths within the skill should be relative to the skill root, not the project root.

## Project Structure

```
xgse/
├── xgse/
│   ├── __init__.py
│   ├── engine/
│   │   ├── skill_engine.py      # Core engine orchestration
│   │   ├── loader.py            # Skill loading from directories
│   │   ├── retrieve.py          # Skill retrieval and matching
│   │   ├── schema.py            # Data structures and parser
│   │   ├── prompts.py           # Prompt templates
│   │   └── spec.py             # Execution spec storage
│   ├── sandbox/
│   │   ├── sandbox_base.py      # Abstract sandbox interface
│   │   ├── sandbox_factory.py   # Sandbox factory
│   │   └── e2b_sandbox.py      # E2B implementation
│   └── utils/
│       ├── llm_client.py        # LLM API client
│       ├── setup_env.py         # Environment setup
│       └── utils.py            # Utility functions
├── examples/skill/              # Example skill runner
│   ├── run_skill.py
│   ├── skills/                  # Example skills
│   └── example_data/           # Test data
├── skill_workspace/             # Working directory (excluded from build)
├── pyproject.toml              # Project configuration
├── README.md                   # Project documentation
└── .env                       # Environment variables (not in version control)
```
