# Implementation Plan: --exit-on-first-fail and --no-results-on-fail Flags

## Overview
Add two new CLI flags to control error handling behavior during benchmark execution:
- `--exit-on-first-fail`: Stop execution on first failed test and exit with non-zero status
- `--no-results-on-fail`: Prevent saving/printing any results when error is experienced (implicitly enables `--exit-on-first-fail`)

## Changes Required

### 1. config.py (src/llama_benchy/config.py)

#### Add to BenchmarkConfig dataclass (around line 29):
```python
exit_on_first_fail: bool = Field(False, description="Stop execution on first failed test and exit with non-zero status")
no_results_on_fail: bool = Field(False, description="Prevent saving/printing any results when error is experienced, turns on --exit-on-first-fail as well")
```

#### Add CLI arguments (after line 57):
```python
parser.add_argument("--exit-on-first-fail", action="store_true", help="Stop execution on first failed test and exit with non-zero status")
parser.add_argument("--no-results-on-fail", action="store_true", help="Prevent saving/printing any results when error is experienced, turns on --exit-on-first-fail as well")
```

#### Update constructor call (in from_args method, after line 83):
```python
exit_on_first_fail=args.exit_on_first_fail,
no_results_on_fail=args.no_results_on_fail
```

### 2. runner.py (src/llama_benchy/runner.py)

#### Modify run_suite method to track errors and handle early exit:

Add error tracking at the start of run_suite (after line 28):
```python
error_occurred = False
error_message = None
```

#### Modify the batch execution loop (around line 136-137):
After collecting batch_results, check for errors:
```python
# Check for errors in batch results
has_error = any(getattr(res, 'error', None) for res in batch_results)
if has_error:
    error_occurred = True
    error_message = next((res.error for res in batch_results if getattr(res, 'error', None)), "Unknown error")
    print(f"Error encountered: {error_message}")
    
    if self.config.exit_on_first_fail:
        print("Exiting due to --exit-on-first-fail flag")
        if self.config.no_results_on_fail:
            print("No results will be saved due to --no-results-on-fail flag")
        raise SystemExit(1)
```

#### Modify results saving (around line 166):
```python
# Only save results if no error occurred or if --no-results-on-fail is not set
if not (error_occurred and self.config.no_results_on_fail):
    self.results.save_report(self.config.save_result, self.config.result_format, max(self.config.concurrency_levels) if self.config.concurrency_levels else 1)

# Exit with error code if error occurred
if error_occurred:
    raise SystemExit(1)
```

#### Update exception handler (around line 168-181):
```python
except (asyncio.CancelledError, KeyboardInterrupt, SystemExit) as e:
    # Don't save results if --no-results-on-fail and error occurred
    if self.results.runs and not (error_occurred and self.config.no_results_on_fail):
        print("\n[Interrupted] Saving partial results...")
        if self.results.metadata is None:
            self.results.metadata = BenchmarkMetadata(
                version=__version__,
                timestamp=datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%SZ"),
                latency_mode=self.config.latency_mode,
                latency_ms=latency * 1000,
                model=self.config.model,
                prefix_caching_enabled=self.config.enable_prefix_caching,
                max_concurrency=max_concurrency
            )
        self.results.save_report(self.config.save_result, self.config.result_format, max_concurrency)
    raise
```

### 3. __main__.py (src/llama_benchy/__main__.py)

No changes needed - SystemExit exceptions will be handled naturally by Python's event loop.

## Testing

After implementation, test with:
```bash
# Test --exit-on-first-fail
llama-benchy --base-url <invalid-url> --model test --exit-on-first-fail
# Should exit immediately with non-zero status

# Test --no-results-on-fail
llama-benchy --base-url <invalid-url> --model test --no-results-on-fail
# Should exit with non-zero status and not save any results
```

## Notes
- `--no-results-on-fail` implicitly enables `--exit-on-first-fail` behavior
- Error detection happens after each batch of concurrent requests
- Partial results are still saved on interruption unless `--no-results-on-fail` is set
