# Browser Hybrid — Eval Strategy

## Overview

Browser-hybrid automates browsers via Chrome DevTools Protocol. Unlike typical LLM applications, most failures are **deterministic** — either the browser connected, or it didn't; either we found the element, or we didn't. This means most evals can be **code-based** rather than LLM-as-judge.

## Eval Categories for Browser-Hybrid

### 1. Code-Based Evals (Deterministic)

These have expected outputs that can be asserted programmatically.

| Category | What it tests | Example |
|----------|---------------|---------|
| **Connection** | Can connect to Chrome | `assert browser.is_connected()` |
| **Navigation** | URL changes correctly | `assert "example.com" in tab.url` |
| **Element Finding** | Find by selector/text | `assert browser.find_by_text("Submit")` |
| **JavaScript Execution** | Return values | `assert browser.evaluate("1+1") == 2` |
| **Screenshot** | Valid PNG created | `assert is_png(screenshot_path)` |
| **Accessibility Tree** | Tree structure | `assert tree.role == "WebArea"` |
| **Error Handling** | Proper exceptions | `assert_raises(ConnectionError, ...)` |

### 2. Subjective Evals (LLM-as-Judge)

These require judgment — multiple acceptable outcomes exist.

| Category | What it tests | Criteria |
|----------|---------------|----------|
| **Element Ranking** | Which element to click when multiple match | Correct semantic match |
| **Snapshot Readability** | Is the snapshot human-readable? | Has structure, not overwhelming |
| **Error Recovery** | Did we recover gracefully? | Clear error message + context |
| **Accessibility Tree Completeness** | Is the tree complete? | All visible elements present |

---

## Golden Datasets

### Connection Evals

```yaml
# evals/connection.yaml
suite: connection
cases:
  - name: connect_localhost
    input:
      host: localhost
      port: 9222
    expected: connected

  - name: connect_invalid_port
    input:
      host: localhost
      port: 9999
    expected:
      error: ConnectionError
      message_contains: "Failed to connect"

  - name: version_retrieval
    input:
      operation: get_version
    expected:
      browser_contains: "Chrome"
```

### Navigation Evals

```yaml
# evals/navigation.yaml
suite: navigation
cases:
  - name: navigate_example
    input:
      url: "https://example.com"
      wait: 1.0
    expected:
      url_contains: "example.com"
      title: "Example Domain"

  - name: navigate_invalid_url
    input:
      url: "not-a-url"
    expected:
      url_pattern: "chrome-error://*"  # Chrome shows error page

  - name: navigate_redirect
    input:
      url: "https://httpbin.org/redirect/2"
    expected:
      final_url_contains: "httpbin.org"
```

### Element Finding Evals

```yaml
# evals/elements.yaml
suite: elements
cases:
  - name: find_by_text_exact
    input:
      page: "https://example.com"
      find_text: "More information"
    expected:
      found: true
      element_role: "link"

  - name: find_by_text_not_found
    input:
      page: "https://example.com"
      find_text: "XyzzyPlughBazoo"
    expected:
      found: false

  - name: find_by_role_links
    input:
      page: "https://example.com"
      find_role: "link"
    expected:
      min_count: 1
```

### JavaScript Evaluation Evals

```yaml
# evals/javascript.yaml
suite: javascript
cases:
  - name: evaluate_arithmetic
    input:
      script: "1 + 1"
    expected: 2

  - name: evaluate_dom_query
    input:
      page: "https://example.com"
      script: "document.title"
    expected: "Example Domain"

  - name: evaluate_object
    input:
      page: "https://example.com"
      script: "window.location.hostname"
    expected: "example.com"
```

### Accessibility Tree Evals

```yaml
# evals/accessibility.yaml
suite: accessibility
cases:
  - name: tree_has_root
    input:
      page: "https://example.com"
    expected:
      root_role: "WebArea"

  - name: tree_find_heading
    input:
      page: "https://example.com"
      find_role: "heading"
    expected:
      min_count: 1

  - name: tree_find_paragraphs
    input:
      page: "https://example.com"
      find_role: "paragraph"
    expected:
      min_count: 1
```

### Error Handling Evals

```yaml
# evals/errors.yaml
suite: errors
cases:
  - name: connection_error_context
    input:
      host: "invalid-host"
      port: 9222
    expected:
      error: ConnectionError
      context_has:
        - host
        - port

  - name: timeout_error_context
    input:
      url: "https://example.com"
      timeout: 0.001  # Intentionally short
    expected:
      error: TimeoutError
      context_has:
        - timeout
        - operation
```

---

## Eval Runner

### CLI Interface

```bash
# Run all evals
browser-hybrid eval

# Run specific suite
browser-hybrid eval --suite connection
browser-hybrid eval --suite navigation

# Run with verbose output
browser-hybrid eval -v

# Generate report
browser-hybrid eval --report html
```

### Python API

```python
from browser_hybrid.eval import EvalRunner

runner = EvalRunner()
results = runner.run()
print(results.summary())

# Output:
# === Eval Results ===
# Connection: 3/3 PASSED
# Navigation: 2/3 PASSED (1 FAILED)
# Elements: 2/2 PASSED
# JavaScript: 3/3 PASSED
# Accessibility: 3/3 PASSED
# Errors: 2/2 PASSED
#
# Total: 15/16 PASSED (93.75%)
```

---

## LLM-as-Judge Evals

### Element Ranking Eval

When `find_by_text("Submit")` returns multiple candidates, how do we choose?

**Approach**: Create golden dataset with ambiguous cases, then use LLM to judge if the correct element was chosen.

```yaml
# evals/subjective/element_ranking.yaml
suite: element_ranking
cases:
  - name: multiple_submit_buttons
    input:
      page: "form_with_multiple_submits.html"
      find_text: "Submit"
    human_judgment:
      correct_element: "form[id='main'] button[type='submit']"
      critique: "The main form's submit button is the semantic target"
    expected:
      ranking: PASS  # LLM evaluates if we picked the right one
```

### Evaluating the Judge

The LLM judge must be validated against human judgments:

```python
# Evaluate judge accuracy
judge_results = []
for case in gold_dataset:
    predicted = llm_judge(case)
    actual = case.human_judgment
    judge_results.append({
        "predicted": predicted,
        "actual": actual,
    })

# Calculate metrics
tp = sum(1 for r in judge_results if r["predicted"] == "PASS" and r["actual"] == "PASS")
tn = sum(1 for r in judge_results if r["predicted"] == "FAIL" and r["actual"] == "FAIL")
fp = sum(1 for r in judge_results if r["predicted"] == "PASS" and r["actual"] == "FAIL")
fn = sum(1 for r in judge_results if r["predicted"] == "FAIL" and r["actual"] == "PASS")

tpr = tp / (tp + fn)  # True Positive Rate
tnr = tn / (tn + fp)  # True Negative Rate

print(f"Judge accuracy: {(tp + tn) / len(judge_results):.1%}")
print(f"True Positive Rate: {tpr:.1%}")
print(f"True Negative Rate: {tnr:.1%}")
```

**Target**: TPR and TNR should both be ≥80% for the judge to be trusted.

---

## CI/CD Integration

### Pipeline Stages

```yaml
# .github/workflows/eval.yaml
name: Evals

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 6 * * *'  # Daily eval run

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Chrome
        uses: browser-actions/setup-chrome@v1

      - name: Install browser-hybrid
        run: pip install -e ".[dev]"

      - name: Run code-based evals
        run: browser-hybrid eval --suite all --report junit

      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: eval-results/
```

### Regression Detection

```python
# Eval regression detection
class EvalRunner:
    def check_regression(self, current_results, baseline_results):
        """Check if results regressed from baseline."""
        for suite in baseline_results:
            if current_results[suite].pass_rate < baseline_results[suite].pass_rate:
                return RegressionError(
                    f"{suite}: {current_results[suite].pass_rate:.1%} < "
                    f"{baseline_results[suite].pass_rate:.1%}"
                )
```

---

## Error Analysis Workflow

### Step 1: Collect Traces

```python
# Run agent on test cases, collect traces
traces = []
for case in eval_cases:
    with trace_collector() as trace:
        result = browser_hybrid.run(case.input)
    traces.append({
        "case": case,
        "result": result,
        "trace": trace,
    })
```

### Step 2: Open Coding

Review traces, annotate failures:

```
case: find_by_text_multiple
trace_id: tr_2024_01_15_001
notes:
  - "Found 3 elements matching 'Submit', chose first one"
  - "First element was footer submit, should have chosen main form submit"
  - "Need to rank by semantic importance, not just order"
```

### Step 3: Axial Coding

Group notes into themes:

```
Theme: Element Ranking
- Count: 12
- Examples: footer buttons, nav links, duplicate content

Theme: Timeout Handling
- Count: 5
- Examples: slow pages, network issues

Theme: Accessibility Tree Gaps
- Count: 3
- Examples: dynamic content, iframes
```

### Step 4: Prioritize

```python
# Pivot table
import pandas as pd

themes = pd.DataFrame([
    {"theme": "Element Ranking", "count": 12},
    {"theme": "Timeout Handling", "count": 5},
    {"theme": "Accessibility Tree Gaps", "count": 3},
])

themes.sort_values("count", ascending=False)
# → Focus on Element Ranking first
```

---

## Implementation Roadmap

### Phase 1: Code-Based Evals (Week 1)

- [ ] Create `evals/` directory structure
- [ ] Implement `EvalRunner` class
- [ ] Add YAML parsing for test cases
- [ ] Create connection, navigation, elements, javascript suites
- [ ] Add `browser-hybrid eval` CLI command

### Phase 2: Integration (Week 2)

- [ ] Add GitHub Actions workflow for evals
- [ ] Implement baseline comparison for regression detection
- [ ] Add eval results reporting (text, JSON, HTML)
- [ ] Document eval creation process

### Phase 3: LLM-as-Judge (Week 3)

- [ ] Create subjective eval golden dataset
- [ ] Implement LLM judge interface
- [ ] Validate judge against human judgments
- [ ] Calculate TPR/TNR metrics

### Phase 4: Error Analysis (Week 4)

- [ ] Add trace collection during evals
- [ ] Create custom trace viewer
- [ ] Document open coding process
- [ ] Establish feedback loop to improve evals

---

## Eval Maintenance

### Continuous Improvement Flywheel

```
┌─────────────────────────────────────────┐
│                                         │
│   Run Evals ──► Analyze Failures        │
│       ▲              │                  │
│       │              ▼                  │
│   Improve        Open Coding            │
│   Code              │                  │
│       ▲              ▼                  │
│       │      Axial Coding               │
│       │              │                  │
│       └──────────────┘                  │
│              (New Eval Cases)           │
│                                         │
└─────────────────────────────────────────┘
```

### Adding New Eval Cases

1. **Observe failure** in production or testing
2. **Create test case** capturing the scenario
3. **Add to YAML** with expected behavior
4. **Run eval** to verify it catches the regression
5. **Document** in CHANGELOG.md

---

## Success Metrics

| Metric | Target | Measurement |
|--------|--------|--------------|
| Code-based eval pass rate | ≥95% | `browser-hybrid eval --suite all` |
| LLM judge accuracy | ≥80% TPR/TNR | Validate against human judgments |
| Regression detection | 100% | All regressions caught in CI |
| Eval coverage | All core features | Every public method tested |