# Benchmarks

This folder contains the YAML benchmark suite and validation runners.

## Structure

- `circuits/` — YAML netlists with embedded `benchmark` metadata.
- `benchmarks.yaml` — scenario matrix and benchmark list.
- `benchmark_runner.py` — executes benchmarks and produces results artifacts.
- `validation_matrix.py` — runs all solver/integrator combinations.
- `benchmark_ngspice.py` — Pulsim vs external SPICE parity runner (`ngspice` or `ltspice` backends).
- `stress_suite.py` — tiered stress validation runner (tiers A/B/C + pass criteria).
- `stress_catalog.yaml` — stress tier definitions, mapped cases, and acceptance criteria.

## Running

```bash
# Use local build bindings (repository workflow)
export PYTHONPATH=build/python

python3 benchmarks/benchmark_runner.py --output-dir benchmarks/out
python3 benchmarks/validation_matrix.py --output-dir benchmarks/matrix
python3 benchmarks/stress_suite.py --output-dir benchmarks/stress_out

# Compare Pulsim vs ngspice (manifest mode)
python3 benchmarks/benchmark_ngspice.py \
  --backend ngspice \
  --output-dir benchmarks/ngspice_out

# Compare Pulsim vs LTspice (explicit executable path is required)
python3 benchmarks/benchmark_ngspice.py \
  --backend ltspice \
  --ltspice-exe "/Applications/LTspice.app/Contents/MacOS/LTspice" \
  --output-dir benchmarks/ltspice_out

# Compare one YAML vs one .cir directly
python3 benchmarks/benchmark_ngspice.py \
  --backend ngspice \
  --pulsim-netlist benchmarks/circuits/rc_step.yaml \
  --spice-netlist benchmarks/ngspice/rc_step.cir \
  --output-dir benchmarks/ngspice_single
```

Benchmark runners are Python-first and execute through `pulsim` runtime bindings
(`YamlParser` + `Simulator`).
When a benchmark netlist omits `simulation.adaptive_timestep`, runners default to
fixed-step mode (`adaptive_timestep: false`) for deterministic comparisons.

Generate missing reference baselines:

```bash
python3 benchmarks/benchmark_runner.py --generate-baselines
```

## Output Artifacts

Each run produces:

- `results.csv` — per-scenario metrics.
- `results.json` — full structured results and metadata.
- `summary.json` — pass/fail summary.

Telemetry fields are sourced from structured simulation result objects and included in `results.json`.
Analytical `max_error` thresholds in `circuits/*.yaml` are calibrated for the current
Python-first runtime defaults (fixed-step unless explicitly overridden).

`benchmark_ngspice.py` also emits:

- `parity_results.csv` — per benchmark/scenario parity results.
- `parity_results.json` — machine-readable parity payload (`schema_version`, backend metadata, per-observable metrics).
- `parity_summary.json` — pass/fail totals and grouped failure reasons.

For `--backend ngspice`, legacy filenames (`ngspice_results.*`, `ngspice_summary.json`) are also written for compatibility.

`stress_suite.py` emits:

- `stress_results.csv` — per scenario stress execution rows with telemetry columns.
- `stress_results.json` — tier criteria + per-tier evaluation + scenario records.
- `stress_summary.json` — overall pass/fail and per-tier status.

## Adding Benchmarks

1. Create a YAML netlist in `circuits/` with a `benchmark` block.
2. Add it to `benchmarks.yaml` and assign scenarios.
3. If you want parity for this benchmark:
   - add `ngspice_netlist: ngspice/<file>.cir` and/or `ltspice_netlist: ltspice/<file>.cir` in `benchmarks.yaml`.
4. Optional backend-specific mapping:
   - `ngspice_observables`: `{ column: "V(out)", ngspice_vector: "v(out)" }`
   - `ltspice_observables`: `{ column: "V(out)", ltspice_vector: "V(out)" }`
5. Metric thresholds can be configured under `benchmark.expectations.metrics`:
   - `max_error`, `rms_error`, `phase_error_deg`, `steady_state_max_error`, `steady_state_rms_error`
6. If using `reference` validation, add a baseline CSV under `baselines/`.
7. Optional validation window controls:
   - `benchmark.validation.ignore_initial_samples`: ignore N leading samples.
   - `benchmark.validation.start_time`: compare only from a minimum time.
