# Uvicorn request limits and how they relate to asgi-runway

Understanding where requests can queue or be dropped is essential for
configuring autoscaling correctly. There are three distinct layers between
a client and your route handler, each with its own limit.

---

## The full request path

```
Client
  │
  │  TCP SYN
  ▼
┌─────────────────────────────────────────┐
│  OS TCP backlog                         │  ← Layer 1
│  Default: 2048  (--backlog)             │
│  Full? → connection silently dropped    │
└─────────────────────────────────────────┘
  │  connection accepted by uvicorn
  ▼
┌─────────────────────────────────────────┐
│  uvicorn --limit-concurrency            │  ← Layer 2
│  Default: unlimited                     │
│  Full? → HTTP 503 returned immediately  │
└─────────────────────────────────────────┘
  │  request enters ASGI app
  ▼
┌─────────────────────────────────────────┐
│  RunwayMiddleware                       │  ← asgi-runway measures here
│  REQUESTS_IN_FLIGHT.inc()               │
│                                         │
│  ┌───────────────────────────────────┐  │
│  │  thread pool  (sync routes only)  │  │  ← Layer 3
│  │  Size: min(32, cpu_count + 4)     │  │
│  │  Full? → request waits in asyncio │  │
│  └───────────────────────────────────┘  │
│                                         │
│  REQUESTS_IN_FLIGHT.dec()               │
└─────────────────────────────────────────┘
  │
  ▼
Response sent to client
```

`asgi-runway_requests_in_flight` only counts what is inside the middleware zone.
Requests dropped at Layer 1 or rejected at Layer 2 are invisible to it.

---

## Layer 1 — TCP backlog

The OS queues incoming TCP connections before uvicorn has a chance to accept
them. This is the kernel-level socket backlog.

**Default in uvicorn:** 2048 (the `--backlog` flag).

**What happens when it fills:** The OS silently drops new SYN packets. The
client experiences a connection timeout, not a TCP reset or HTTP error. There
is no log entry on the server side — the request simply never arrives.

**OS cap:** Linux further limits the backlog with `net.core.somaxconn`, which
defaults to **128** on many distributions. If your `--backlog` is 2048 but
`somaxconn` is 128, the effective backlog is 128.

Check and raise it:

```bash
# Check current value
sysctl net.core.somaxconn

# Raise it for the current session
sysctl -w net.core.somaxconn=4096

# Persist across reboots
echo "net.core.somaxconn=4096" >> /etc/sysctl.conf
```

In Kubernetes, set this as a privileged init container or via a node-level
`DaemonSet` if you are running high-throughput workloads.

**When does this matter:** Only if your pod is receiving thousands of new
connections per second faster than uvicorn can accept them. For most services
the default is fine. For high-connection-rate APIs (many short-lived clients),
raise `somaxconn` and `--backlog` together.

---

## Layer 2 — `--limit-concurrency`

Once a connection is accepted by uvicorn, this flag acts as a second gate.

**Default:** unlimited (no gate).

**What happens when it fills:** Uvicorn returns **HTTP 503 Service
Unavailable** immediately, before the request reaches your ASGI middleware or
any route handler. The response is instant and clean.

**How to set it:**

```bash
uvicorn app:app --limit-concurrency 60
```

This is the right place to put a hard ceiling. It provides controlled
degradation — clients get a retryable 503 rather than waiting indefinitely or
hitting a silent TCP timeout.

---

## Layer 3 — thread pool (sync routes only)

If you use `def` (synchronous) route handlers, FastAPI runs them via
`run_in_threadpool`, which uses `anyio`'s default thread pool.

**Default pool size:** `min(32, os.cpu_count() + 4)`

| Pod CPUs | Default thread pool |
|---|---|
| 1 | 5 |
| 2 | 6 |
| 4 | 8 |
| 8 | 12 |

Once all threads are occupied, additional requests queue inside the asyncio
event loop, waiting for a thread to become free. These waiting requests **are
counted** by `asgi-runway_requests_in_flight` — they have entered the middleware
and incremented the gauge, they are just not yet executing on a thread.

This is why `asgi-runway_requests_in_flight` is a reliable saturation signal for
CPU-bound workloads: it rises as requests pile up waiting for threads, well
before any connection is dropped.

**Async routes (`async def`) are not affected by this limit** — they run
directly on the event loop and do not consume a thread. For async I/O-bound
routes, there is no artificial concurrency ceiling; the practical limit is
memory and event loop overhead.

---

## Recommended production configuration

Use `--limit-concurrency` as a circuit breaker set at roughly 3× your
autoscaling threshold. The three numbers form a deliberate ladder:

```
autoscaling threshold  →  triggers scale-up  (e.g. 20)
limit-concurrency      →  hard cap, 503       (e.g. 60)
TCP backlog            →  OS buffer            (e.g. 2048, rarely hits)
```

Example:

```bash
uvicorn app:app \
  --backlog 2048 \
  --limit-concurrency 60
```

With KEDA configured to scale at `asgi-runway_requests_in_flight > 20`:

- In-flight reaches 20 → KEDA triggers scale-up
- New pod takes ~30s to become ready
- During those 30s, load continues to climb
- At 60 in-flight → uvicorn starts returning 503 (controlled degradation)
- New pod becomes ready → in-flight drops → 503s stop

Without `--limit-concurrency`, the pod would accept all connections and queue
them internally. Memory grows, latency climbs, and the pod may OOM before the
new instance is ready.

---

## What asgi-runway sees vs. what it does not

| Event | Counted by asgi-runway? |
|---|---|
| Request waiting in TCP backlog | No |
| Request rejected by `--limit-concurrency` (503) | No |
| Request inside middleware, waiting for thread pool | **Yes** |
| Request actively executing in route handler | **Yes** |
| Request where handler raised an exception | **Yes** (until `finally` runs) |

The implication: `asgi-runway_requests_in_flight = 0` is a reliable signal that
the pod is truly idle. A non-zero value always means work is happening or
queued inside the process.
