# Agent Orchestration Strategy -- LLMHosts.com

## META
- **created**: 2026-02-14
- **references**: implementation-blueprint.md, test-strategy.md, github-strategy.md
- **research**: AOrchestra (agent tuple model), COMPASS (context management), AgentOrchestra (hierarchical delegation)
- **purpose**: Define how to break work into optimal units for single-pass quality with AI coding agents (Cursor subagents, Copilot coding agent, Claude)

---

## 1. CORE PRINCIPLE: CONTEXT-FOCUSED UNITS OF WORK

The quality of AI agent output is directly proportional to:
- **Focus**: One clear objective per agent invocation
- **Context density**: Only relevant information in the context window, nothing extraneous
- **Completeness**: Everything the agent needs is present (no guessing, no assuming)
- **Verification**: Clear acceptance criteria the agent can self-check

Each unit of work (task) must fit entirely within a single agent context window (~128K tokens for most models). This means:
- Task description + relevant docs + code context < 80K tokens (leave room for reasoning)
- One PR per task (atomic, reviewable, revertible)
- Task output is self-contained and testable

### Anti-Patterns to Avoid
- ❌ Dumping entire codebase into context (wastes tokens, reduces focus)
- ❌ Vague task descriptions ("improve router" vs "implement kNN router Tier 1")
- ❌ Missing dependencies (agent guesses interfaces instead of reading them)
- ❌ No acceptance criteria (agent doesn't know when done)
- ❌ Multi-feature PRs (hard to review, hard to revert, violates single-responsibility)

---

## 2. TASK TAXONOMY (Size categories)

### XS Task (< 1 hour, < 10K context)
- **Scope**: Single file change, well-defined
- **Examples**: 
  - Add a config field
  - Fix a typo
  - Update a dependency version
  - Add a test case
- **Agent**: fast model subagent
- **Context needed**: Just the file + acceptance criteria
- **PR size**: 1-50 lines changed

### S Task (1-4 hours, 10-30K context)
- **Scope**: Single component, clear spec
- **Examples**: 
  - Implement a utility function
  - Add an API endpoint
  - Create a React component
  - Write unit tests for a module
- **Agent**: standard subagent
- **Context needed**: Component spec + related interfaces + test fixtures
- **PR size**: 50-300 lines changed
- **From blueprint**: Most individual steps map to S tasks

### M Task (4-8 hours, 30-60K context)
- **Scope**: Multi-file feature, integration required
- **Examples**: 
  - Implement router tier
  - Build cache layer
  - Create CLI command with tests
- **Agent**: capable model subagent
- **Context needed**: Feature spec + API contracts + related components + test strategy
- **PR size**: 300-1000 lines changed
- **Decomposition**: Decompose into 2-3 S tasks if possible
- **From blueprint**: Steps 7 (proxy), 13 (vCache), 14 (kNN router) are M tasks

### L Task (1-2 days, 60-80K context)
- **Scope**: Cross-cutting feature, multiple components
- **Examples**: 
  - Full proxy pipeline
  - Dashboard page + API + WebSocket
  - Tunnel integration
- **MUST decompose**: Split into 3-5 M or S tasks
- **Orchestrator coordinates**: Subagents execute individual pieces
- **From blueprint**: Steps 8 (serve command), 16 (web dashboard) are L tasks → decompose

### XL Task (2-5 days)
- **Scope**: Epic-level work
- **ALWAYS decompose**: Never assign to a single agent
- **Break into**: L → M → S chain
- **From blueprint**: Entire Phase 1 is XL → break into 8 steps → each step is S/M

---

## 3. TASK SPECIFICATION FORMAT

Every task assigned to an agent must include:

```markdown
## Task: [ID] [Title]
### Objective
One sentence: what this task produces.

### Context Files (read these)
- path/to/file1.py -- why this is relevant
- path/to/file2.md -- why this is relevant

### Spec
Detailed requirements, interfaces, data shapes.

### Acceptance Criteria
- [ ] Criterion 1 (testable)
- [ ] Criterion 2 (testable)
- [ ] All existing tests still pass

### Tests to Write
- test_function_a: tests X
- test_function_b: tests Y

### Out of Scope
What NOT to touch or change.

### Dependencies
- Requires: [task IDs that must complete first]
- Blocks: [task IDs that depend on this]
```

### Example Task Spec

```markdown
## Task: T-011 vCache Exact Hash (Tier 0)
### Objective
Implement SHA-256 exact match cache tier for identical query lookups.

### Context Files (read these)
- docs/architecture/implementation-blueprint.md -- Step 11 spec
- docs/architecture/system-architecture.md -- Cache architecture overview
- llmhosts/cache/__init__.py -- Cache module structure
- tests/test_cache_hash.py -- Test file to create

### Spec
- Create `llmhosts/cache/normalizer.py`: normalize queries (lowercase, strip whitespace, remove system prompt)
- Create `llmhosts/cache/hash_cache.py`: SHA-256 hash lookup, hit tracking
- Create `llmhosts/cache/manager.py`: orchestrates cache tier checks (stub for now, will add Tier 1/2 later)
- Cache key: SHA-256 of normalized user message (not system prompt)
- Store in SQLite `cache_entries` table: hash_key, response, hit_count, last_hit_at
- Respect `config.cache.max_size_mb` limit

### Acceptance Criteria
- [ ] Identical queries return cached response in <5ms
- [ ] Different queries never match (0% false positives)
- [ ] Hit count and last_hit_at tracked per entry
- [ ] Cache respects max_size_mb config (LRU eviction)
- [ ] `llmhosts cache stats` shows hash tier stats
- [ ] All existing tests still pass

### Tests to Write
- test_normalize_query: lowercase, whitespace, system prompt removal
- test_hash_cache_hit: identical query returns cached response
- test_hash_cache_miss: different query misses
- test_hash_cache_eviction: LRU eviction when max_size_mb exceeded
- test_hash_cache_stats: hit count tracking

### Out of Scope
- Entity namespace (Tier 1) -- separate task
- Semantic matching (Tier 2) -- separate task
- Cache warming or preloading

### Dependencies
- Requires: T-003 (database), T-002 (config)
- Blocks: T-012 (entity namespace), T-013 (vCache semantic)
```

---

## 4. ORCHESTRATION PATTERN

### Coordinator-Implementor-Verifier (CIV) Pattern

```
[Orchestrator Agent]
    |
    ├── Read: implementation-blueprint.md, PRD, test-strategy.md
    ├── Decompose: Break current epic into S/M tasks
    ├── For each task:
    │   ├── [Implementor Agent] -- focused subagent with task spec
    │   │   ├── Read context files
    │   │   ├── Write tests first (TDD)
    │   │   ├── Implement code
    │   │   ├── Run tests locally
    │   │   └── Return: files changed, tests passed/failed
    │   ├── [Verifier Agent] -- separate subagent
    │   │   ├── Read changed files + test results
    │   │   ├── Check acceptance criteria
    │   │   ├── Run linter
    │   │   ├── Check for regressions
    │   │   └── Return: pass/fail + issues
    │   └── If fail: Orchestrator creates fix task, assigns new Implementor
    └── Commit + PR when all tasks in epic pass
```

### Why CIV?
- **Implementor has narrow focus** (high quality): Only sees task spec + relevant files, not entire codebase
- **Verifier has fresh eyes** (catches errors): Different agent checks work, catches what implementor is blind to
- **Orchestrator maintains big picture** (no drift): Ensures tasks align with blueprint, no scope creep
- **Matches AOrchestra research**: Specialized sub-agents > generalist agents

### When to Use CIV
- **M and L tasks**: Always use CIV (implementor + verifier)
- **S tasks**: Optional CIV (can self-verify if simple)
- **XS tasks**: No CIV needed (too small)

### Alternative: Self-Verification (for S tasks)
For simple S tasks, the implementor can self-verify:
1. Write tests first (TDD)
2. Implement code
3. Run tests + linter
4. Check acceptance criteria
5. Return: pass/fail + any issues

---

## 5. CONTEXT MANAGEMENT (COMPASS-inspired)

### For each agent invocation, context is structured as:

1. **SYSTEM**: Agent role and constraints (from AGENTS.md)
2. **SPEC**: Task specification (from task spec format above)
3. **CODE**: Only files directly relevant to this task (not entire codebase)
4. **TESTS**: Existing test patterns to follow (from test strategy + conftest.py)
5. **PROGRESS**: What's been done so far in this epic (brief summary, not full history)

### Context Budget:
- System prompt: ~2K tokens
- Task spec: ~3K tokens  
- Code files: ~20-50K tokens (3-8 files typically)
- Test patterns: ~5K tokens
- Progress summary: ~2K tokens
- **Total: ~32-62K tokens**, leaving 50%+ for reasoning

### Anti-patterns:
- ❌ **DO NOT** dump the entire codebase into context
- ❌ **DO NOT** include files the agent won't modify
- ❌ **DO NOT** include historical discussion or debate docs
- ✅ **DO include** all interfaces the agent's code must conform to
- ✅ **DO include** test fixtures and patterns

### Example Context for Task T-011 (vCache Hash)

**SYSTEM** (from AGENTS.md):
- Docker only, Python 3.12+, FastAPI, Pydantic v2
- TDD: tests written first
- Follow test-strategy.md patterns

**SPEC** (task spec above)

**CODE** (only these files):
- `llmhosts/cache/__init__.py` -- module structure
- `llmhosts/cache/normalizer.py` -- to create
- `llmhosts/cache/hash_cache.py` -- to create
- `llmhosts/cache/manager.py` -- to create
- `llmhosts/config.py` -- to read cache.max_size_mb
- `llmhosts/db/models.py` -- to read cache_entries table schema

**TESTS**:
- `tests/conftest.py` -- pytest fixtures
- `tests/unit/test_cache.py` -- existing cache test patterns (if any)
- `tests/test_cache_hash.py` -- to create

**PROGRESS**:
- T-003 (database): ✅ Complete, cache_entries table exists
- T-002 (config): ✅ Complete, cache.max_size_mb config available
- T-011 (this task): In progress

**Total context**: ~45K tokens (well within 80K budget)

---

## 6. IMPLEMENTATION BLUEPRINT TO TASK MAPPING

The implementation-blueprint.md has 22 steps. Map them to GitHub issues:

### Phase 1 - Foundation (Steps 1-8): 
- **Each step = 1 GitHub issue = 1 agent task** (S or M sized)
- **Sequential**: step N depends on step N-1
- **Agent receives**: step spec + previous step's output files

**Mapping**:
- Step 1 (skeleton): **S task** → Issue #1
- Step 2 (config): **S task** → Issue #2
- Step 3 (database): **M task** → Issue #3
- Step 4 (discovery): **M task** → Issue #4
- Step 5 (BYOK): **M task** → Issue #5
- Step 6 (LiteLLM): **M task** → Issue #6
- Step 7 (proxy): **M task** → Issue #7
- Step 8 (serve): **L task** → Issue #8 (decompose into 3 S tasks: startup sequence, TUI integration, web integration)

### Phase 2 - Intelligence (Steps 9-14):
- **Steps can partially parallelize** (router tiers are independent)
- **Each tier = 1 issue = 1 agent task** (M sized)
- **Agent receives**: router interface + test fixtures + tier spec

**Mapping**:
- Step 9 (rule router): **M task** → Issue #9
- Step 10 (cost tracking): **M task** → Issue #10 (can parallelize with Step 9)
- Step 11 (hash cache): **S task** → Issue #11
- Step 12 (namespace cache): **S task** → Issue #12 (depends on #11)
- Step 13 (vCache semantic): **M task** → Issue #13 (depends on #12, requires [smart])
- Step 14 (kNN router): **M task** → Issue #14 (can parallelize with #13, requires [smart])

### Phase 3 - Interface (Steps 15-19):
- **Dashboard components can parallelize**
- **Each page = 1 issue = 1 agent task** (M sized)
- **Agent receives**: wireframe + component inventory + API spec for that page

**Mapping**:
- Step 15 (TUI): **M task** → Issue #15
- Step 16 (web dashboard): **L task** → Issue #16 (decompose into: React SPA setup, SavingsTicker component, RequestLog component, CacheStats component, BackendStatus component, WebSocket integration)
- Step 17 (doctor): **M task** → Issue #17
- Step 18 (cache CLI): **S task** → Issue #18
- Step 19 (suggest): **S task** → Issue #19

### Phase 4 - Integration (Steps 20-22):
- **Sequential**: Docker -> testing -> release
- **Each step = 1 issue = 1 agent task** (M sized)
- **Full context needed** for integration testing

**Mapping**:
- Step 20 (CI/CD): **M task** → Issue #20
- Step 21 (docs): **M task** → Issue #21
- Step 22 (release): **M task** → Issue #22

### Issue Template

Each GitHub issue should follow this template:

```markdown
## Task: [Step X] [Title]

### Objective
[From implementation-blueprint.md]

### Context Files
- docs/architecture/implementation-blueprint.md -- Step X
- [other relevant files]

### Spec
[From implementation-blueprint.md Step X]

### Acceptance Criteria
[From implementation-blueprint.md Step X]

### Tests to Write
[From test-strategy.md + blueprint]

### Dependencies
- Requires: #[previous-step-issue]
- Blocks: #[next-step-issue]

### Labels
- size: S/M/L
- phase: phase-1-core
- component: comp-[router|cache|proxy|cli|dashboard]
- agent-ready: true
```

---

## 7. SPRINT STRUCTURE (2-week sprints)

Based on implementation-blueprint.md effort estimates:

### Sprint 1: Foundation Core (Steps 1-4)
- **Duration**: 2 weeks
- **Tasks**: Skeleton, config, database, discovery
- **Goal**: Core infrastructure ready
- **Deliverable**: `llmhosts --version` works, config system functional, database initialized, discovery detects Ollama

### Sprint 2: Proxy Core (Steps 5-8)
- **Duration**: 2 weeks
- **Tasks**: BYOK, LiteLLM, FastAPI proxy, serve command
- **Goal**: Working proxy that routes requests
- **Deliverable**: `llmhosts serve` starts proxy, routes to Ollama + cloud APIs

### Sprint 3: Intelligence (Steps 9-12)
- **Duration**: 2 weeks
- **Tasks**: Rule router, cost tracking, hash cache, namespace cache
- **Goal**: Basic routing and caching operational
- **Deliverable**: Router makes decisions, cache reduces redundant calls

### Sprint 4: Smart Intelligence (Steps 13-14)
- **Duration**: 2 weeks
- **Tasks**: vCache semantic, kNN router ([smart] tier)
- **Goal**: Intelligent routing and semantic caching
- **Deliverable**: Semantic cache hits, kNN routing decisions

### Sprint 5: Interfaces (Steps 15-19)
- **Duration**: 2 weeks
- **Tasks**: TUI, web dashboard, doctor, cache CLI, suggest
- **Goal**: User-facing interfaces complete
- **Deliverable**: Dashboard shows stats, doctor verifies setup, cache CLI manages cache

### Sprint 6: Ship (Steps 20-22)
- **Duration**: 2 weeks
- **Tasks**: CI/CD, docs, release
- **Goal**: Production-ready release
- **Deliverable**: v0.1.0 on PyPI + Docker Hub, GitHub release, announcement

**Total**: 12 weeks (6 sprints) for Phase 1 complete

---

## 8. QUALITY GATES

### Before any task is marked done:
1. ✅ **All new code has tests** (TDD: tests written first)
2. ✅ **All tests pass** (unit + integration for that component)
3. ✅ **Linter clean** (`ruff check` + `ruff format`)
4. ✅ **Type checked** (`mypy --strict`)
5. ✅ **No regressions** (full unit suite passes)
6. ✅ **Acceptance criteria met** (from task spec)
7. ✅ **PR passes CI pipeline** (GitHub Actions)

### Before any sprint is done:
1. ✅ **All sprint tasks pass quality gates**
2. ✅ **Spec tests pass** for new features (tests/spec/)
3. ✅ **Contract tests pass** (schemathesis against openapi.yaml)
4. ✅ **Demo**: feature works end-to-end
5. ✅ **Documentation updated** if interfaces changed

### Before release:
1. ✅ **Full test suite passes** (all layers: unit + integration + e2e)
2. ✅ **Performance benchmarks met** (from PRD non-functional requirements)
3. ✅ **Security scan clean** (bandit + safety + pip-audit)
4. ✅ **CHANGELOG updated** (Keep-a-Changelog format)
5. ✅ **Version bumped** (semver: MAJOR.MINOR.PATCH)

### Quality Gate Automation

**GitHub Actions CI** enforces:
- Lint + type check on every push
- Unit tests on every PR
- Integration tests on PR to main
- Contract tests on PR to main
- Security scan on PR to main
- E2E tests on release branches

**Manual gates** (human review):
- Acceptance criteria verification
- Architecture alignment check
- Performance benchmark validation
- Documentation completeness

---

## 9. TASK EXECUTION WORKFLOW

### Step-by-Step Process

1. **Orchestrator reads blueprint**
   - Identifies next step(s) to implement
   - Checks dependencies (previous steps complete?)
   - Creates GitHub issue with task spec

2. **Orchestrator assigns to Implementor**
   - Provides task spec + context files list
   - Implementor reads context files
   - Implementor writes tests first (TDD)

3. **Implementor implements**
   - Writes code to pass tests
   - Runs tests locally
   - Checks linter + type checker
   - Creates PR with task ID in title

4. **Verifier reviews** (for M/L tasks)
   - Reads PR diff
   - Checks acceptance criteria
   - Runs full test suite
   - Verifies no regressions
   - Approves or requests changes

5. **Orchestrator merges**
   - PR merged → issue closed
   - Next task unblocked
   - Progress tracked in GitHub Projects

### Example Execution: Task T-011 (vCache Hash)

```
[Orchestrator]
  → Creates Issue #11: "Step 11: vCache Exact Hash (Tier 0)"
  → Assigns to Implementor Agent

[Implementor Agent]
  → Reads: blueprint Step 11, cache module structure, db models, config
  → Writes: tests/test_cache_hash.py (5 test functions)
  → Implements: normalizer.py, hash_cache.py, manager.py
  → Runs: pytest tests/test_cache_hash.py ✅
  → Runs: ruff check ✅, mypy ✅
  → Creates: PR #42 "feat(cache): implement exact hash cache tier (#11)"

[Verifier Agent]
  → Reads: PR #42 diff
  → Checks: acceptance criteria (all ✅)
  → Runs: pytest tests/ ✅ (no regressions)
  → Runs: ruff check ✅
  → Approves: PR #42

[Orchestrator]
  → Merges: PR #42
  → Closes: Issue #11
  → Unblocks: Issue #12 (namespace cache)
```

---

## 10. ERROR HANDLING AND RETRY

### When Implementor Fails

1. **Tests fail**: Verifier catches, creates fix task, assigns new Implementor
2. **Linter errors**: Verifier catches, Implementor fixes in same PR
3. **Type errors**: Verifier catches, Implementor fixes in same PR
4. **Regression**: Verifier catches, Implementor fixes in same PR
5. **Acceptance criteria not met**: Verifier rejects, Orchestrator creates follow-up task

### When Verifier Fails

1. **False positive**: Human reviewer overrides verifier
2. **Verifier bug**: Fix verifier logic, re-run verification

### When Task is Too Large

1. **Implementor reports**: "This task is L-sized, needs decomposition"
2. **Orchestrator**: Splits into 2-3 S/M tasks
3. **Re-assigns**: Smaller tasks to Implementors

### Maximum Retries

- **Per task**: Maximum 2 retries before human intervention
- **Per PR**: Maximum 3 review cycles before escalation
- **Escalation**: Human developer takes over if agent fails repeatedly

---

## 11. METRICS AND MONITORING

### Task-Level Metrics
- **Time to complete**: XS <1h, S <4h, M <8h, L <2d
- **First-pass success rate**: Target 80%+ (agent completes without retry)
- **Test coverage**: Target 90%+ for new code
- **PR size**: XS <50 lines, S <300 lines, M <1000 lines

### Sprint-Level Metrics
- **Velocity**: Tasks completed per sprint
- **Quality**: % tasks passing quality gates on first try
- **Blockers**: Dependencies causing delays

### Project-Level Metrics
- **Blueprint progress**: Steps completed / total steps
- **Test coverage trend**: Coverage % over time
- **CI pass rate**: % PRs passing CI on first try

### Dashboard
GitHub Projects board shows:
- Backlog | Ready | In Progress | In Review | Done
- Filter by: Phase, Component, Priority, Size
- Burndown chart: Tasks remaining vs sprint progress

---

## 12. BEST PRACTICES

### For Orchestrator
- ✅ Break large tasks into smaller ones (S/M preferred)
- ✅ Provide complete context (all interfaces, all dependencies)
- ✅ Clear acceptance criteria (testable, measurable)
- ✅ Check dependencies before assigning
- ❌ Don't assign tasks with missing dependencies
- ❌ Don't create vague tasks ("improve X")

### For Implementor
- ✅ Read all context files before coding
- ✅ Write tests first (TDD)
- ✅ Run tests + linter before creating PR
- ✅ Keep PRs focused (one task = one PR)
- ❌ Don't modify files outside task scope
- ❌ Don't skip tests

### For Verifier
- ✅ Check all acceptance criteria
- ✅ Run full test suite (not just new tests)
- ✅ Verify no regressions
- ✅ Check code style + types
- ❌ Don't approve without verification
- ❌ Don't skip quality gates

---

## 13. EVOLUTION AND ADAPTATION

### Review and Refine
- **Weekly**: Review task completion rates, adjust task sizes if needed
- **Sprint retrospective**: What worked? What didn't? Adjust CIV pattern if needed
- **Monthly**: Review context management strategy, optimize token usage

### Learning from Failures
- **Failed tasks**: Analyze why (missing context? unclear spec? too large?)
- **Retry patterns**: Identify common failure modes, update task specs
- **Quality trends**: If first-pass success drops, investigate root cause

### Scaling Up
- **Parallelization**: As more agents available, parallelize independent tasks
- **Specialization**: Create specialized agents (router expert, cache expert, dashboard expert)
- **Automation**: Automate more quality gates (auto-verify simple S tasks)

---

**Last Updated**: 2026-02-14
**Next Review**: After Sprint 1 completion
