# Harbor RL

## Installation

```bash
uv pip install 'tinker-cookbook[modal] @ git+https://github.com/thinking-machines-lab/tinker-cookbook.git@nightly'
```

RL training on Harbor formatted tasks (e.g., Terminal Bench 2.0) with sandboxed code execution. An agent gets a bash tool inside a sandboxed container, attempts a task, and receives reward based on test results.

## HarborTask
Harbor offers a standardized format for SWE/Terminal-Bench style task.
Adhering to this allows seperation between task creation layer and evaluation/training harness layer.
We can download the harbor datasets through `uvx harbor datasets download terminal-bench@2.0`.
By default, the task will land in `~/.cache/harbor/tasks/` with the structure
```
~/.cache/harbor/tasks/
  └── <shortuuid(task_id)>/       # deterministic hash for deduplication
      └── <task_name>/            # human-readable task directory
          ├── environment/
          │   └── Dockerfile
          ├── tests/
          │   └── test.sh
          ├── instruction.md
          ├── task.toml
          └── solution/
```
To use harbor tasks for training or evaluation, we designed the following interface

```python
@dataclass(frozen=True)
class HarborTask:
    task_name: str
    instruction: str
    task_dir: Path      # must contain environment/Dockerfile and tests/test.sh
    config: dict[str, Any] = field(default_factory=dict)
```

You can load your downloaded tasks (e.g., 89 Terminal-Bench tasks) via `load_harbor_tasks()` in `launch_terminal_bench.py`:

```python
from tinker_cookbook.recipes.harbor_rl.launch_terminal_bench import load_harbor_tasks

tasks = load_harbor_tasks()  # reads from ~/.cache/harbor/tasks/ by default
print(f"Loaded {len(tasks)} tasks")
print(tasks[0].task_name, tasks[0].task_dir)
```
The training environment is implemented against this interface.
You can customize your own task as long as they conforms to the interface above.

## Sandbox Protocol and custom backends

### The Protocol

`tinker_cookbook.sandbox.sandbox_interface` defines `SandboxInterface`:

```python
@runtime_checkable
class SandboxInterface(Protocol):
    async def run_command(self, command: str, workdir: str | None = None, timeout: int = 60, max_output_bytes: int | None = None) -> SandboxResult: ...
    async def read_file(self, path: str, max_bytes: int | None = None, timeout: int = 60) -> SandboxResult: ...
    async def write_file(self, path: str, content: str | bytes, executable: bool = False, timeout: int = 60) -> SandboxResult: ...
    async def send_heartbeat(self) -> None: ...
    async def cleanup(self) -> None: ...
```

`ModalSandbox` implements this interface.

### SandboxFactory and injection

`harbor_env.py` defines a factory type and default:

```python
SandboxFactory = Callable[[modal.Image, int], Awaitable[SandboxInterface]]

async def default_sandbox_factory(image: modal.Image, timeout: int) -> SandboxInterface:
    return await ModalSandbox.create(image=image, timeout=timeout)
```

`cli_main()` accepts an optional `sandbox_factory` parameter. When `None`, it falls back to `default_sandbox_factory` (Modal). The factory flows through: `cli_main` -> `HarborDatasetBuilder` -> `HarborEnvGroupBuilder.make_envs()`.

## Running

First, download the Terminal-Bench tasks:

```bash
uvx harbor datasets download terminal-bench@2.0
```

Then launch training:

```bash
uv run python tinker_cookbook/recipes/harbor_rl/scripts/train_terminal_bench.py
```

## Evaluation

Evaluate a Tinker endpoint on Harbor datasets without training.

Download datasets:
```bash
uvx harbor datasets download terminal-bench@2.0 -o ~/.cache/harbor/tasks/terminal-bench-2.0
uvx harbor datasets download swebench-verified@1.0 -o ~/.cache/harbor/tasks/swebench-verified-1.0
```

Run evaluation:
```bash
uv run python tinker_cookbook/recipes/harbor_rl/scripts/eval_terminal_bench.py
```

Key parameters in `EvalConfig`: `max_turns`, `max_tokens`, `temperature`.
`run_eval()` also accepts `sandbox_factory` for custom sandbox backends and `output_path` to control where results are written (default: `tinker_cookbook/recipes/harbor_rl/scripts/results/<timestamp>/`).

We evaluated SWE-Bench-Verified-1.0 and Terminal-Bench-2.0 at 32K context length and naive agent harness with no advanced features like context compatification that summarizes the tool calling history.

### Results: Kimi-K2-Thinking (32K context, no compaction)

| Benchmark | Total | PASS | FAIL | ERROR | Pass Rate |
|-----------|-------|------|------|-------|-----------|
| SWE-Bench Verified 1.0 | 500 | 46 (9.2%) | 52 (10.4%) | 402 (80.4%) | 9.2% |
| Terminal-Bench 2.0 | 89 | 18 (20.2%) | 36 (40.4%) | 35 (39.3%) | 20.2% |

**Config**: `max_turns=200, max_tokens=8192, temperature=0.1, sandbox_timeout=3600s`

All ERRORs are context window overflow (`prompt_tokens + max_tokens > 32768`).
These occur when the conversation history exceeds ~24.5K tokens, leaving insufficient room for the 8192 `max_tokens` generation budget.
