# Testing Instructions

All tests under `test-runner/tests/` use **pytest**. Follow these principles
when writing or modifying tests.

Use the **GPA (Goal-Plan-Action) self-verification loop**
([Jia et al. 2025](https://arxiv.org/abs/2510.08847)) after every batch of
test changes to catch misalignment before submission.

## GPA Self-Verification Loop

Before writing tests, and again after completing them, walk through these
five checkpoints. If any answer is "no", stop and fix before proceeding.

### 1. Goal Fulfillment — *Does the test verify the intended behavior?*

State the goal in one sentence before writing the test:
> "This test proves that `snapshot_table` hex-encodes `bytes` columns."

After writing the test, re-read the goal. Does the assertion actually prove
it, or does it only prove something adjacent (e.g., that the function
returned *something*)?

### 2. Plan Quality — *Does the testing approach align with the principles?*

Before coding, decide:
- What is the unit boundary? (function, class, chain of functions)
- What is faked vs real? (only external boundaries should be faked)
- What data triggers the scenario? (minimal fixture)

Cross-check each decision against the Principles below. If the plan requires
patching an internal function, the plan is wrong — redesign it.

### 3. Logical Consistency — *Are setup, action, and assertions consistent?*

Verify that:
- The fake data you supply actually reaches the code path under test.
- The assertion matches the scenario (e.g., a "mismatch" test asserts
  `result.match is False`, not `is True`).
- No assertion depends on mock return values that you hard-coded — that
  tests the mock, not the code.

### 4. Plan Adherence — *Does the implementation follow the plan?*

Compare the finished test against your plan:
- Did you end up adding `@patch` calls that were not in the plan?
- Did you add conditional logic inside a parametrized test?
- Did you duplicate production logic instead of calling the real function?

Any deviation is a signal to revisit the plan or fix the implementation.

### 5. Execution Efficiency — *Is the test minimal and focused?*

- Could the fixture be smaller (fewer rows, fewer columns)?
- Could structurally identical tests be parametrized?
- Is there dead setup (variables created but never used in assertions)?
- Does the test name convey exactly what it checks?

---

## Principles

1. **Test the logic, not the plumbing.** If you are patching the function you
   claim to be testing, you are not testing it. Mocks should replace *external
   boundaries* (DB connections, network, file I/O), not internal function calls.

2. **Prefer fake objects over patches.** A `_FakeConnection` class that returns
   canned `cursor.fetchall()` results is better than
   `@patch("module.snapshot_table")` because it exercises the real
   cursor-handling logic (column extraction, hex encoding, error paths).

3. **Each test must fail for exactly one reason.** Name tests after the scenario
   and expected behavior: `test_<scenario>_<expected_behavior>`, not
   `test_<thing>_works`.

4. **Assert on semantic correctness, not structural existence.**
   Bad: `assert "inserts" in result`.
   Good: `assert result["inserts"] == [{"COLOR": "Blue"}]`.

5. **Test error paths as first-class scenarios.** Every `try/except`, every
   `if not ...: return`, every `raise ValueError(...)` needs a test that
   triggers it and asserts on the exact outcome (error message, return value,
   side effects).

6. **Integration tests bridge mock boundaries.** Create a fake boundary object
   that returns canned data, then let the real internal chain run unmocked.
   Mock only the outermost external boundary.

7. **Parametrize structurally identical tests.** When 2+ tests share the same
   arrange/act/assert shape and differ only in data, use
   `@pytest.mark.parametrize` with `ids=` for readable output. Do NOT
   parametrize if the test body would need conditionals per case — keep those
   as separate functions.

8. **Use plain `assert` statements (pytest idiom).** Do not use
   `self.assertEqual` / `self.assertIn` / `self.assertRaises` from `unittest`.
   Pytest rewrites plain `assert` to produce rich diffs on failure. Use
   `pytest.raises(ExceptionType, match=...)` for exception assertions.

9. **Fixtures should be minimal.** Use the smallest possible data to trigger
   the behavior under test. One-row tables, single-element lists, short strings.

10. **Never mock an embedded engine.** When code under test uses an in-process
    engine (DuckDB, SQLite, regex, etc.), test against the real engine with
    representative data. Mocking it removes the only behavior worth testing.
    *Example: delta capture strategies use DuckDB EXCEPT — test with a real
    in-memory DuckDB connection, not a mocked one.*

11. **Assert on the exact commands issued to external systems.** When code
    generates SQL, HTTP requests, or CLI commands, assert on the exact strings
    and their order — not just that "something was called."
    *Example: isolation providers should be tested by capturing the SQL they
    emit (`BEGIN TRANSACTION`, `ROLLBACK`, `CREATE ... CLONE ...`) and
    verifying state transitions like autocommit toggling.*

12. **Concurrent code paths need dedicated tests.** Single-threaded tests
    do not cover thread-per-worker creation, shared-state races, or
    error aggregation across workers. Test with `workers > 1` explicitly.

13. **Cover every row in a scenario matrix.** When a spec defines a scenario
    matrix, each row must map to at least one test. If a scenario has no test,
    it is a gap. If one test covers multiple scenarios, it is too coarse.

## Checklist Before Submitting Tests

Run the GPA loop (above), then verify:

- [ ] Every `@patch` has a comment explaining WHY the mock is necessary
      (if you cannot justify it, remove the patch and use a fake boundary object)
- [ ] No test re-implements production logic (call the real function instead)
- [ ] Every assertion checks a semantic property, not just structural existence
- [ ] Error paths have dedicated test cases with exact exception/return assertions
- [ ] New fixtures are minimal (smallest possible data to trigger the behavior)
- [ ] Test names follow `test_<scenario>_<expected_behavior>` convention
- [ ] Repetitive tests with identical structure are parametrized with clear `ids=`
- [ ] Parametrized tests do not contain conditionals to handle different cases
