# ToolOrchestra -- NVIDIA Orchestrator Framework

## META
- source: arXiv:2511.21689 (Nov 26, 2025)
- repo: https://github.com/NVlabs/ToolOrchestra (Apache 2.0)
- model: https://huggingface.co/nvidia/Orchestrator-8B
- dataset: https://huggingface.co/datasets/nvidia/ToolScale
- relevance: CORE -- direct inspiration for LLMHosts Conductor Brain
- last_verified: 2026-02-13

## SUMMARY

ToolOrchestra trains a small (8B param) orchestrator model to coordinate diverse tools and LLMs via end-to-end RL. The orchestrator decides WHEN and WHICH tools/models to invoke, optimizing for correctness, cost-efficiency, and user preference alignment simultaneously. Key result: 8B orchestrator outperforms GPT-5 on HLE (37.1% vs 35.1%) at 30% cost.

## ARCHITECTURE

### System Hierarchy
```
Orchestrator-8B (Coordinator / Brain)
├── Basic Tools
│   ├── Web Search (Tavily API)
│   ├── Code Interpreter (Python sandbox)
│   ├── Local Search (Faiss + Qwen3-Embedding-8B)
│   └── Domain Functions (get_flight_status, etc.)
├── Specialized LLMs
│   ├── GPT-5 (code writer role)
│   ├── GPT-5-mini (code writer role)
│   ├── Qwen2.5-Coder-32B-Instruct (code writer)
│   ├── Qwen2.5-Math-72B (math)
│   └── Qwen2.5-Math-7B (math)
└── Generalist LLMs
    ├── GPT-5
    ├── GPT-5-mini
    ├── Llama-3.3-70B-Instruct
    └── Qwen3-32B
```

### Multi-Turn Reasoning Loop (MDP Formulation)
1. **Reasoning Phase**: Orchestrator analyzes state, plans next action (chain-of-thought)
2. **Tool Call Phase**: Selects tool from available set, specifies parameters
3. **Observation Phase**: Tool executes, output appended to context
4. Repeat up to 50 turns or until termination signal

### Formal MDP Definition
- State: interaction history h_k = (u, o_0, a_0, o_1, ..., a_{k-1}, o_k)
- Action: tool invocation a_k selected by policy pi_theta(a_k | h_k)
- Observations: tool responses o_{k+1}
- Costs: each action has monetary cost c_i and latency l_i
- User preferences: vector P specifying desired tool usage patterns

## REWARD DESIGN (Critical for LLMHosts)

### Three-Objective Reward System
1. **Outcome Reward**: r_outcome(tau) in {0,1} -- binary task success
2. **Efficiency Rewards**:
   - r_compute(tau) = -$(tau) -- negative monetary cost
   - r_latency(tau) = -Clock(tau) -- negative wall-clock time
3. **Preference Reward**: Dot product of normalized trajectory metrics with user preference vector P

### Combined Reward Formula
```
R(tau) = M_normalized^tau · P   (if task solved)
R(tau) = 0                       (otherwise)
```
Where:
- M^tau = [tool_call_counts..., r_outcome, r_compute, r_latency]
- P = [tool_preferences..., p_outcome, p_compute, p_latency]
- Each element normalized min-max within rollout batch

### User Preference Vector Examples
- P[anthropic_opus] = 1.0 → strongly prefer Opus
- P[outcome] = 1.0, P[compute] = 0.0 → maximize accuracy, ignore cost
- P[compute] = 1.0, P[latency] = 1.0 → minimize cost and latency

## TRAINING APPROACH

### Algorithm: GRPO (Group Relative Policy Optimization)
- Backbone: Qwen3-8B
- SFT data: GeneralThought-430K dataset + synthetic ToolScale data
- Learning rate: 1e-6
- Max input sequence: 24,000 tokens
- Max generation: 8,000 tokens
- Batch size: 16, rollout batch: 8
- Max turns per rollout: 50
- Hardware: 16x NVIDIA H100 GPUs

### Training Stability Techniques
1. **Homogeneity filtering**: Skip batches where reward std < 0.1
2. **Format consistency filtering**: Reject outputs not matching tool call format
3. **Invalid output filtering**: Reject outputs without valid answers

### ToolScale Data Synthesis Pipeline
1. Select domain D
2. LLM generates database schema + tool APIs for domain
3. LLM proposes diverse intents for domain
4. Convert intents to specific tasks with golden actions
5. Additional LLM complicates tasks (adds constraints)
6. Filter: remove if golden calls error, LLM can't solve in pass@8, or solvable without actions
- Domains: 10 (airline, bank, basketball, etc.)

## CUSTOMIZATION POINTS (For LLMHosts Fork)

### Tool Configuration
- Edit `tools.json` to add/remove/update tools
- Update `tool_config` at line 27 in eval scripts
- Implement handlers in `call_tool()` function

### LLM Backend Integration
- Modify `get_llm_response()` in `LLM_CALL.py`
- Supports vLLM and OpenAI-compatible endpoints

### Prompt Modification
- HLE prompts: lines 455-458 in `eval_hle.py`
- FRAMES prompts: lines 506-509 in `eval_frames.py`

### Tool Configuration Randomization (for training)
- Randomize tool subsets per training instance
- Vary pricing schedules across instances
- Prevents overfitting to fixed toolkit

## PERFORMANCE BENCHMARKS

| Benchmark | Orchestrator-8B | GPT-5 | Cost Ratio |
|-----------|----------------|-------|------------|
| HLE | 37.1% | 35.1% | 2.5x cheaper |
| tau2-Bench | 80.2% | 77.7% | ~30% cost |
| FRAMES | 76.3% | 74.0% | ~30% cost |
| GAIA | #1 ranking | -- | -- |

## KEY INSIGHTS FOR LLMHOST

1. **Small orchestrators work**: 8B model can outperform frontier models by delegating intelligently
2. **Cost-quality tradeoff is learnable**: RL trains the model to balance these automatically
3. **User preferences are trainable**: Preference vectors let users control routing without manual rules
4. **Generalization to unseen tools**: Orchestrator adapts to new tools not seen during training
5. **Balanced tool usage**: RL-trained orchestrator avoids bias toward any single model (unlike prompted models)
6. **Selective delegation**: Only invoke expensive models when necessary, use cheaper tools for simpler subtasks

## REPO STRUCTURE
```
ToolOrchestra/
├── LLM_CALL.py              # LLM integration layer
├── prepare_sft_data.py       # SFT data preparation
├── requirements.txt          # Dependencies
├── data/                     # Domain data
├── data_synthesis/           # Synthetic data pipeline
│   └── run.ipynb            # Data generation notebook
├── evaluation/               # Benchmark evaluation
│   ├── run_hle.py           # HLE runner
│   ├── eval_hle.py          # HLE evaluator
│   ├── run_frames.py        # FRAMES runner
│   ├── eval_frames.py       # FRAMES evaluator
│   └── tau2-bench/          # tau2-Bench evaluation
└── training/                 # RL training
    └── resume_h100.py       # Training entry point
```

## DEPENDENCIES (from requirements.txt)
- Python 3.12
- vLLM 0.9.2
- flash-attn, flashinfer
- transformers, datasets
- faiss-gpu, pyserini
- tavily-python
- wandb
- uvicorn, fastapi
