---
type: skill
domain: development
---

# ERROR_RECOVERY
> **ROLE:** Rollback and state recovery specialist
> **TRIGGER:** When code fails, data corrupts, or operations need undo

## Core Directive
Ensure every destructive or stateful operation can be safely undone. Never leave the system in a half-broken state.

## Step-by-Step Protocol
1. **Identify blast radius:** List all files/state that will change. Check `git status` first.
2. **Create checkpoint:** `git stash` or snapshot current state before risky operations.
3. **Wrap in transaction:** Use try/finally for file ops, database transactions for DB ops, atomic writes for JSON.
4. **Implement rollback path:** For every write, define the undo. `os.replace()` is atomic on Windows; `shutil.move()` is NOT.
5. **Verify or revert:** After operation, validate the result. If invalid, execute rollback immediately.

## Metrics & Thresholds
- Every file write MUST use atomic operations (os.replace or filelock)
- Database operations: wrap in explicit transactions, rollback on any exception
- Max rollback time: <5 seconds for any single operation
- Never leave temp files behind — use try/finally cleanup

## Examples

### Good: Atomic JSON Update
```python
def atomic_update(filepath, updater):
    with filelock.FileLock(f"{filepath}.lock"):
        data = json.loads(filepath.read_text())
        data = updater(data)
        tmp = filepath.with_suffix(".tmp")
        tmp.write_text(json.dumps(data))
        os.replace(str(tmp), str(filepath))  # Atomic
```

### Bad: Non-Atomic Write
```python
# DANGER: If crash happens between truncate and write, data is LOST
with open(filepath, "w") as f:
    json.dump(data, f)  # Not atomic — file truncated before write completes
```

## Anti-Patterns
- Using `shutil.move()` for concurrent file operations (causes data loss)
- Catching exceptions silently without rolling back state
- Writing to files without locks in multi-agent environments

## Tools to Use
- `brain_query.py --query "atomic file operations"` — Check brain for known patterns
- `brain_utils.atomic_update_json()` — Built-in atomic JSON updater with filelock

## Verification
- [ ] Every file write uses atomic pattern (write-tmp-then-replace)
- [ ] Every database operation wrapped in transaction
- [ ] Rollback path tested (deliberately fail mid-operation, verify clean state)
- [ ] No temp files left behind after success or failure
