# DDR-0037: E27 Single In-Flight Request State Machine with Timeout-Based Release

Status: Accepted

Date: 2026-01-10

## Context

The Elk E27 panel has a limited internal packet pool shared across multiple client
sessions, the cloud connection, and internal operations. Vendor guidance indicates
the safest and preferred operating model is:

- send one request at a time per session
- send the next request immediately after the prior response is received

ADR-0111 establishes a strict single in-flight request policy to prevent response
queueing inside the panel and avoid packet pool exhaustion.

An implementation detail must be clarified: if a response never arrives, the client
must not deadlock. The failure mode must be handled without violating the single
in-flight rule.

Key distinction: pacing vs correctness

- Throttle/min-interval/burst is pacing: it limits bursts and smooths traffic.
- Reply-timeout is correctness/liveness: it prevents the queue from stalling when
  a response is lost or delayed indefinitely.

A throttle timer must not be used to "force release" and send the next request
without concluding the prior request; doing so violates the one request/one response
constraint and reintroduces packet pool risk.

## Decision

Implement an explicit, Kernel-owned state machine that enforces the single in-flight
request rule and uses a per-request reply timeout to guarantee forward progress.

### Definitions

- In-flight request: the single outbound request currently sent that expects a reply
  (has a `seq` and a pending future).
- Request completion: a terminal event for the in-flight request, occurring on:
  - matched reply received (success)
  - reply timeout fired (fail locally)
  - connection loss / explicit abort (fail locally)
- Throttle (min interval / burst): pacing logic that may delay sends but must not
  advance correctness.

### Normative rules

1. Single in-flight is absolute
   - MUST NOT transmit a second request that expects a reply while another request
     remains in-flight.

2. Throttle is pacing only
   - Throttle MAY delay a send after the in-flight gate is open.
   - Throttle MUST NOT be used to advance the request queue or bypass the in-flight
     rule.
   - Throttle MUST NEVER transmit the next queued request while a request remains
     in-flight.

3. Reply-timeout is liveness/correctness
   - Every request that expects a reply MUST have a reply-timeout.
   - If no matched reply arrives within the timeout, the request MUST be failed
     locally and MUST release the in-flight gate to allow the next queued request
     to be sent (still subject to throttle pacing).

4. Release is driven by completion only
   - The outbound request queue / send gate is released ONLY by request completion
     (success or failure).
   - Throttle timers do not release the queue; they only affect when a send may
     occur once the gate is released.

5. Late replies are inert
   - If a reply arrives after the request has completed (timed out/failed/aborted),
     it MUST be treated as unexpected/unsolicited:
       - MUST NOT release the in-flight gate
       - MUST NOT resurrect or modify failed request state

6. Exactly-once completion
   - Timeout, reply, and disconnect paths MUST release the in-flight gate exactly
     once per request. Double-release MUST be prevented (e.g., guard flag).

## State Machine Specification (Kernel-owned)

### States

A) IDLE
- No request expecting a reply is in-flight.
- Scheduler may attempt to send the next queued request (subject to throttle).

State variables (conceptual):
- active_seq = None
- active_timeout_handle = None
- active_released = False (unused in IDLE; used to guard exactly-once completion)

B) IN_FLIGHT
- Exactly one request expecting a reply has been sent and is awaiting completion.
- No other expects-reply request may be sent until completion.

State variables:
- active_seq = <seq>
- active_timeout_handle = armed
- active_released = False (completion guard)

### Events, guards, transitions, actions

1) ENQUEUE_REQUEST(req)
- If state is IDLE: kick scheduler to attempt sending.
- If state is IN_FLIGHT: keep queued; no send.

Transition: none
Action: kick_scheduler()

2) TRY_SEND_NEXT()
Guard conditions:
- state == IDLE
- queue not empty
- throttle permits send now (else schedule a future TRY_SEND_NEXT at earliest time)

On success:
- dequeue next request (respect priority lanes if present)
- assign seq (if applicable)
- create pending future for seq
- transmit request (transport send)
- arm reply-timeout for seq
- set active_seq = seq
- set active_released = False
- transition to IN_FLIGHT

Transition: IDLE -> IN_FLIGHT

3) REPLY_RECEIVED(msg)
Matched reply guard:
- state == IN_FLIGHT AND msg.seq == active_seq

Matched reply actions (in order):
- resolve pending future for active_seq
- cancel reply-timeout
- COMPLETE_ACTIVE(reason="reply")

Transition: IN_FLIGHT -> IDLE (via COMPLETE_ACTIVE)

Unmatched/late reply actions:
- log at debug/trace (include received seq and current active_seq)
- ignore; no state changes; no release

Transition: none

4) REPLY_TIMEOUT_FIRED(seq)
Guard:
- state == IN_FLIGHT AND seq == active_seq AND active_released == False

Actions (in order):
- fail pending future for seq with TimeoutError (or dedicated exception)
- COMPLETE_ACTIVE(reason="timeout")

Transition: IN_FLIGHT -> IDLE (via COMPLETE_ACTIVE)

If guard fails:
- stale timeout; ignore

5) CONNECTION_LOST(exc) / ABORT(exc)
If state == IN_FLIGHT and not released:
- cancel reply-timeout
- fail pending future for active_seq with ConnectionError/AbortError (wrap cause)
- COMPLETE_ACTIVE(reason="disconnect/abort")

Transition: IN_FLIGHT -> IDLE (via COMPLETE_ACTIVE)

Queued requests handling:
- policy decision outside this DDR (fail immediately vs retain for reconnect),
  but MUST NOT violate single in-flight semantics.

### Completion primitive (exactly-once release)

COMPLETE_ACTIVE(reason)
Guard:
- if active_released is True: return (no-op)
- else set active_released = True and proceed

Actions:
- clear active_seq
- clear active_timeout_handle
- set state = IDLE
- kick_scheduler() to attempt sending next queued request (still subject to throttle)

Invariants enforced:
- exactly one completion per request
- throttle never advances while IN_FLIGHT
- late replies cannot release because active_seq cleared and/or pending removed

## Consequences

- Each expects-reply request must have an associated reply-timeout.
- Timeout, reply, and disconnect paths must release the in-flight gate exactly once.
- Late responses are safely ignored and do not affect pacing.
- Throttle remains secondary safety (pacing), not a correctness mechanism.
- Implementation should remain small, localized, and backwards compatible.

## Validation / Test Matrix (must pass)

1. Normal:
   - enqueue A -> send A -> receive reply A -> send B (subject to throttle)

2. Timeout:
   - enqueue A -> send A -> no reply -> timeout fires -> A fails -> send B

3. Race:
   - reply and timeout occur near-simultaneously -> only one completion -> no double-release

4. Late reply:
   - A times out -> later reply A arrives -> logged and ignored -> does not unblock

5. Disconnect:
   - send A -> connection lost -> A fails -> gate released (no deadlock)

## References

- ADR-0111: E27 Request Concurrency and Backpressure (Single In-Flight Policy)
- ADR-0111 Addendum: Single In-Flight Request with Timeout-Based Release
- ADR-0112: E27 Bootstrap and Configuration Download Sequencing
- Vendor guidance on E27 packet pool limitations
