# Timeout Levers

Available timeout mechanisms for the nightly script, from most to least robust.
Asyncio cancellation is paradoxically the weakest layer in Python 3.12+.

## 1. Library-level timeouts (most reliable)

Use the library's own timeout parameter. Enforced at socket/protocol level
before asyncio ever needs to cancel anything.

**asyncssh** — native `connect_timeout` kwarg:
```python
# fragile in 3.12+:
await asyncio.wait_for(asyncssh.create_connection(...), timeout=timeout)

# better:
await asyncssh.create_connection(..., connect_timeout=timeout)
```

**aiohttp** (used in `node.send_action()` for CMC cards):
```python
timeout = aiohttp.ClientTimeout(connect=5, total=10)
async with aiohttp.ClientSession(timeout=timeout) as session:
    ...
```

## 2. `subprocess.run(timeout=...)` instead of `os.system()`

For `all_off()` in nightly.py:
```python
# hangs forever:
os.system(command)

# better:
subprocess.run(command, shell=True, timeout=300)
```
Raises `TimeoutExpired` and kills the child process. Completely independent of asyncio.

## 3. `signal.alarm()` as a hard backstop for the whole run

Works regardless of what any library does internally. SIGALRM is delivered by the
OS — no asyncio involvement whatsoever.
```python
import signal

def _hard_timeout(signum, frame):
    raise SystemExit("nightly: hard timeout reached")

signal.signal(signal.SIGALRM, _hard_timeout)
signal.alarm(3600)  # 1 hour absolute ceiling
nightly.run()
signal.alarm(0)     # cancel if we finish normally
```

## 4. Systemd service timeout

If run as a systemd timer, add to the `.service` unit:
```ini
TimeoutStartSec=3600
```
systemd will SIGTERM then SIGKILL after one hour. Survives even a fully wedged
event loop. Most robust backstop of all.

## 5. Thread-with-timeout for synchronous blocking calls (e.g. SidecarSyncClient)

Threads can't be killed in Python, but the main flow can be unblocked:
```python
from concurrent.futures import ThreadPoolExecutor, TimeoutError

with ThreadPoolExecutor(max_workers=1) as ex:
    future = ex.submit(sidecar_call, node.id)
    try:
        future.result(timeout=10)
    except TimeoutError:
        pass  # thread may linger, but main flow moves on
```

## Summary Table

| Problem                          | Fix                                      |
|----------------------------------|------------------------------------------|
| `asyncssh` connect hang          | `connect_timeout=` kwarg                 |
| `aiohttp` CMC call hang          | `aiohttp.ClientTimeout`                  |
| `SidecarSyncClient` hang         | thread executor with `future.result(timeout=)` |
| `os.system("rhubarbe bye")` hang | `subprocess.run(timeout=)`               |
| Everything else                  | `signal.alarm()` or systemd `TimeoutStartSec` |

## Key Principle

Asyncio cancellation is a **cooperation protocol** — it only works if every library
in the chain plays along. OS-level mechanisms (`signal.alarm`, subprocess timeout,
systemd) are not cooperative; they are enforced unilaterally, which is what you
want for a safety net.
