# Nightly Script Hang Analysis

Analysis of `r2lab-embedded/nightly/nightly.py` sporadic indefinite hang issues.

## Root Causes

### 1. `asyncio.wait_for` changed in Python 3.12 — core timeout fragility

Before 3.12, `wait_for` would cancel the inner task and return `TimeoutError` almost immediately.
**Since 3.12, `wait_for` waits for the inner task to completely finish its cancellation before raising `TimeoutError`.**

If `asyncssh.create_connection()` delays or mishandles `CancelledError` during cleanup
(half-open TCP, async internal state teardown), the `wait_for` in `SshProxy.connect()`
hangs permanently. The Scheduler's own timeout then also can't cancel the Job.

This is the most plausible explanation for the *sporadic* nature: only triggers when
a node's TCP state is pathological (SYN sent, no SYN-ACK or RST — a "black hole").

Relevant code: `rhubarbe/ssh.py:96-127` (`connect()`), `nightly.py` `global_send_action()`.

### 2. `SshProxy.wait_for()` has no total-time loop guard

The `timeout` parameter (`rhubarbe/ssh.py:163`) goes to the per-attempt
`asyncio.wait_for()` inside `connect()`, NOT to the `while True` loop.
The loop can only be stopped by the outer `asynciojobs.Scheduler` cancelling the Job.
If that cancellation doesn't land cleanly (see #1), the loop runs forever.

### 3. `SidecarSyncClient` in `mark_and_exclude` — synchronous, no timeout

```python
# nightly.py mark_and_exclude()
with SidecarSyncClient(SIDECAR_URL, **SSL_ARGS) as sidecar:
    sidecar.set_node_attribute(node.id, 'available', 'ko')
```

Runs synchronously outside any event loop. If sidecar is unreachable, blocks for
the OS TCP connect timeout (~2-3 min on Linux). Called once per failing node:
37 nodes × 2-3 min = 1+ hours of blocking.

### 4. `all_off()` uses `os.system()` — no timeout, no kill path

```python
os.system("rhubarbe bye fit01 fit02 ...")
```

`rhubarbe bye` itself calls `asyncio.run(device.off())` for PDUs and `os.system(ssh ...)`
for phones. If any SSH connection hangs (PDU, phone gateway), `rhubarbe bye` hangs,
and `os.system()` in nightly has no escape hatch.
**Most credible explanation for days-long hangs** — nothing external will ever kill it.

## Secondary Issues

### 5. Logic bug: `WONT_TURN_ON` never assigned in `global_send_action`

```python
reason = (
    Reason.WONT_TURN_OFF if mode == 'on'   # ← should be WONT_TURN_ON
    else Reason.WONT_TURN_OFF if mode == 'off'
    else Reason.WONT_RESET)
```

Corrupts failure report but doesn't cause hang.

### 6. `init_nodes()` not called before `global_send_action`

`global_send_action` reuses `self.nodes` and `self.bus` from the constructor, while all
subsequent phases call `init_nodes()`. Also uses `asyncio.Runner()` while other phases
use `asynciojobs.Scheduler.run()` — each creating a completely separate event loop.
Display never runs during `global_send_action` (queue fills up silently).

## Most Likely Cascade for a Days-Long Hang

1. A node's CMC/control network enters TCP black-hole state
2. `asyncssh.create_connection` hangs in cleanup when cancelled (Python 3.12+ semantics)
3. Scheduler timeout fires but also can't cancel that job → everything stops
4. **OR**: several nodes fail SSH checks → `mark_and_exclude` called repeatedly →
   sidecar momentarily unreachable → multiple minutes of synchronous blocking
5. **OR**: `all_off()` at the end hits a PDU or phone SSH that never responds → hung forever

## Architectural Root Cause

Timeouts are entirely delegated to asyncio machinery, which in Python 3.12+ became
stricter: it no longer returns from a timeout until the inner work is truly done.
Any library that doesn't cleanly handle `CancelledError` breaks this assumption.
