# Operations Runbook

This guide covers operational procedures for the unified proxy: restart procedures, recovery from failures, DNS fallback activation, and troubleshooting common issues.

## Proxy Restart Procedure

The unified proxy supports graceful shutdown and restart without data loss.

### Graceful Restart (Recommended)

```bash
# From the host machine
docker compose -f docker-compose.credential-isolation.yml restart unified-proxy
```

**What happens:**
1. Docker sends SIGTERM to mitmproxy process
2. Proxy stops accepting new connections
3. In-flight requests complete (up to 10s grace period)
4. Readiness marker (`/var/run/proxy/ready`) is removed
5. Container restarts with fresh state

### Full Stop and Start

```bash
# Stop the proxy (and dependent containers)
docker compose -f docker-compose.credential-isolation.yml stop unified-proxy

# Start the proxy
docker compose -f docker-compose.credential-isolation.yml start unified-proxy
```

### Force Restart (Emergency)

Use only when graceful restart fails:

```bash
# Force kill and restart
docker compose -f docker-compose.credential-isolation.yml kill unified-proxy
docker compose -f docker-compose.credential-isolation.yml up -d unified-proxy
```

**Caution:** Force kill terminates in-flight requests immediately.

### Restart Verification

After restart, verify the proxy is healthy:

```bash
# Check container status
docker compose -f docker-compose.credential-isolation.yml ps unified-proxy

# Check readiness marker (inside proxy container)
docker exec unified-proxy test -f /var/run/proxy/ready && echo "Ready" || echo "Not ready"

# Check mitmproxy is listening
docker exec unified-proxy ss -tlnp | grep 8080

# Test credential injection (from sandbox)
curl -v https://api.anthropic.com/v1/messages 2>&1 | head -20
```

## SQLite Recovery Procedure

The container registry uses SQLite with WAL mode. Recovery is rarely needed but documented here.

### Database Location

```
/var/lib/unified-proxy/registry.db      # Main database
/var/lib/unified-proxy/registry.db-wal  # Write-ahead log
/var/lib/unified-proxy/registry.db-shm  # Shared memory file
```

### Check Database Integrity

```bash
docker exec unified-proxy sqlite3 /var/lib/unified-proxy/registry.db "PRAGMA integrity_check;"
```

Expected output: `ok`

### Recovery from Corruption

**Symptom:** Container registration fails with "database disk image is malformed"

**Step 1: Stop the proxy**
```bash
docker compose -f docker-compose.credential-isolation.yml stop unified-proxy
```

**Step 2: Backup corrupted database (optional)**
```bash
docker cp unified-proxy:/var/lib/unified-proxy/registry.db ./registry.db.corrupt
```

**Step 3: Remove database files**
```bash
docker exec unified-proxy rm -f /var/lib/unified-proxy/registry.db*
```

**Step 4: Restart proxy**
```bash
docker compose -f docker-compose.credential-isolation.yml start unified-proxy
```

The proxy will create a fresh database on startup. Containers will re-register automatically on their next request.

### WAL Checkpoint (Performance)

If the WAL file grows large, force a checkpoint:

```bash
docker exec unified-proxy sqlite3 /var/lib/unified-proxy/registry.db "PRAGMA wal_checkpoint(TRUNCATE);"
```

### Export Container Registrations

```bash
docker exec unified-proxy sqlite3 -header -csv /var/lib/unified-proxy/registry.db \
  "SELECT container_id, ip_address, datetime(registered_at, 'unixepoch') as registered, ttl_seconds FROM containers;"
```

## DNS Fallback Activation

The proxy supports two DNS modes: DNS-enabled (with filtering) and DNS-disabled (fallback).

### When to Activate Fallback

- mitmproxy DNS mode fails go/no-go criteria (latency > 50ms p99)
- DNS filtering causes legitimate requests to fail
- Emergency bypass for debugging

### Check Current DNS Mode

```bash
# Check if DNS is enabled
docker exec unified-proxy printenv PROXY_ENABLE_DNS
```

Output: `true` (DNS enabled) or empty/false (DNS disabled)

### Activate DNS Fallback

**Step 1: Edit docker-compose override**

Create or edit `docker-compose.override.yml`:

```yaml
services:
  unified-proxy:
    environment:
      - PROXY_ENABLE_DNS=false
```

**Step 2: Restart proxy**
```bash
docker compose -f docker-compose.credential-isolation.yml up -d unified-proxy
```

**Step 3: Verify DNS mode disabled**
```bash
docker logs unified-proxy 2>&1 | grep -i "dns"
# Should show: "DNS filtering enabled" is absent
```

### Re-enable DNS Filtering

Remove the `PROXY_ENABLE_DNS=false` override and restart:

```bash
# Remove override or set to true
docker compose -f docker-compose.credential-isolation.yml up -d unified-proxy

# Verify DNS mode enabled
docker logs unified-proxy 2>&1 | grep "DNS filtering enabled"
```

### DNS Fallback Implications

| Feature | DNS Enabled | DNS Disabled (Fallback) |
|---------|-------------|-------------------------|
| Domain allowlist filtering | DNS + Firewall | Firewall only |
| NXDOMAIN for blocked domains | Yes | No (upstream DNS resolves) |
| Defense in depth | Full | Reduced |
| Latency overhead | +5-20ms | None |

**Security note:** DNS fallback reduces defense in depth. Use only when necessary.

### DNS Filtering (Enabled by Default)

DNS filtering is **enabled by default** for full defense-in-depth security. When enabled:

1. **NXDOMAIN for blocked domains** - Prevents DNS-level reconnaissance
2. **DNS bypass protection** - Containers cannot query external DNS servers (iptables rules block port 53 to non-proxy destinations)
3. **Complete domain isolation** - Domain names cannot leak to external resolvers

To disable DNS filtering for debugging:

```yaml
# docker-compose.override.yml
services:
  unified-proxy:
    environment:
      - PROXY_ENABLE_DNS=false
```

Or set the environment variable when starting the sandbox:

```bash
PROXY_ENABLE_DNS=false cast new myproject
```

**Note:** Disabling DNS filtering reduces defense in depth. Use only when necessary for debugging.

## Troubleshooting

### Proxy Won't Start

**Symptom:** Container exits immediately or healthcheck fails

**Check 1: View startup logs**
```bash
docker logs unified-proxy 2>&1 | head -50
```

**Check 2: Validate required files**
```bash
docker exec unified-proxy ls -la /opt/proxy/addons/
docker exec unified-proxy ls -la /etc/proxy/certs/
```

**Check 3: Verify CA certificate**
```bash
docker exec unified-proxy cat /etc/proxy/certs/mitmproxy-ca.pem | head -3
```

Expected: `-----BEGIN CERTIFICATE-----`

**Resolution:** If addons or certificates missing, check volume mounts in docker-compose.

### Containers Not Registered

**Symptom:** Requests fail with "unknown container" errors

**Check registrations:**
```bash
PROXY_CONTAINER="sandbox-<sandbox-name>-unified-proxy-1"
docker exec "$PROXY_CONTAINER" curl -s --unix-socket /var/run/proxy/internal.sock \
  http://localhost/internal/containers | jq .
```

**Manual registration (emergency):**
```bash
PROXY_CONTAINER="sandbox-<sandbox-name>-unified-proxy-1"
docker exec "$PROXY_CONTAINER" curl -X POST --unix-socket /var/run/proxy/internal.sock \
  -H "Content-Type: application/json" \
  -d '{"container_id":"sandbox-123","ip_address":"172.17.0.2"}' \
  http://localhost/internal/containers
```

### High Latency

**Symptom:** Requests take > 5 seconds

**Check 1: Circuit breaker state**
```bash
curl -s http://unified-proxy:8080/internal/metrics | grep circuit_breaker
```

If state = 2 (OPEN), upstream is failing. See [Circuit Breaker Investigation](observability.md#circuit-breaker-investigation).

**Check 2: Rate limit exhaustion**
```bash
curl -s http://unified-proxy:8080/internal/metrics | grep rate_limit_remaining
```

If remaining < 5, container is rate limited. Wait for refill.

**Check 3: DNS resolution time**
```bash
curl -s http://unified-proxy:8080/internal/metrics | grep dns_duration
```

If p95 > 100ms, consider DNS fallback mode.

### Certificate Errors

**Symptom:** `SSL certificate problem: unable to get local issuer certificate`

**Check 1: Verify CA mounted in sandbox**
```bash
docker exec <sandbox-container> ls -la /certs/
```

**Check 2: Verify environment variables**
```bash
docker exec <sandbox-container> printenv | grep -E "(NODE_EXTRA|SSL_CERT|REQUESTS_CA)"
```

**Check 3: Verify system trust store**
```bash
docker exec <sandbox-container> ls /etc/ssl/certs/ | grep mitmproxy
```

**Resolution:** If CA not found, check `mitm-certs` volume mount.

### Git Operations Failing

**Symptom:** `git clone` or `git push` fails

**Check 1: Verify proxy reachability**
```bash
curl -v http://unified-proxy:8080/
```

**Check 2: Check credential injection logs**
```bash
docker logs unified-proxy 2>&1 | grep -i "credential\|inject" | tail -20
```

**Check 3: Verify GitHub token**
```bash
docker exec unified-proxy printenv GITHUB_TOKEN | head -c 10
# Should show first 10 chars of token (ghp_... or gho_...)
```

### Branch Isolation Errors

**Symptom:** Git commands fail with "branch isolation: access denied" or "missing branch identity"

**Check 1: Verify sandbox has branch identity**
```bash
PROXY_CONTAINER="sandbox-<sandbox-name>-unified-proxy-1"
docker exec "$PROXY_CONTAINER" curl -s --unix-socket /var/run/proxy/internal.sock \
  http://localhost/internal/containers | jq '.[].metadata'
```

Look for `sandbox_branch` and `from_branch` fields in the metadata. If missing, the sandbox was created before branch isolation support (v0.10).

**Check 2: Review audit logs for denied operations**
```bash
docker logs unified-proxy 2>&1 | grep -E "git_audit.*denied" | tail -20
```

**Resolution:**
- **Legacy sandbox (no branch identity):** Recreate the sandbox with `cast destroy && cast new` to get branch isolation metadata
- **Branch not in allowed set:** The sandbox can only access its own branch, well-known branches (main, master, develop, production, release/*, hotfix/*), and tags
- **SHA not reachable:** SHA arguments must be ancestors of allowed branches

### Git Fetch Hangs

**Symptom:** `git fetch` or `git pull` appears to hang indefinitely

`git_operations.py` uses `fcntl.flock` to serialize concurrent fetch operations per bare repo, preventing corruption from parallel fetches. A hang may indicate a stale lock.

**Check 1: Identify the lock file**
```bash
docker exec unified-proxy ls -la /home/ubuntu/.sandboxes/repos/github.com/owner/repo.git/.fetch-lock
```

**Check 2: Check for stale processes**
```bash
docker exec unified-proxy ps aux | grep git
```

**Resolution:**
- If no git process is holding the lock, remove the stale lock file:
  ```bash
  docker exec unified-proxy rm -f /home/ubuntu/.sandboxes/repos/github.com/owner/repo.git/.fetch-lock
  ```
- If a git process is hung, kill it and retry:
  ```bash
  docker exec unified-proxy kill <pid>
  ```

### Request Blocked Unexpectedly

**Symptom:** Request returns 403 or connection refused

**Check 1: Policy engine logs**
```bash
docker logs unified-proxy 2>&1 | grep -i "policy\|blocked\|denied" | tail -20
```

**Check 2: DNS filter logs (if DNS enabled)**
```bash
docker logs unified-proxy 2>&1 | grep -i "dns.*blocked\|NXDOMAIN" | tail -20
```

**Check 3: Verify domain in allowlist**
```bash
docker exec unified-proxy cat /etc/unified-proxy/allowlist.yaml | grep <domain>
```

## Emergency Procedures

### Complete Proxy Reset

Use when proxy is in unrecoverable state:

```bash
# 1. Stop all credential isolation services
docker compose -f docker-compose.credential-isolation.yml down

# 2. Remove volumes (CAUTION: loses certificates)
docker volume rm $(docker volume ls -q | grep -E "mitm-certs|proxy-")

# 3. Recreate everything
docker compose -f docker-compose.credential-isolation.yml up -d
```

### Bypass Proxy (Emergency Only)

**CAUTION:** This bypasses credential isolation. Use only for debugging.

```bash
# Inside sandbox container, temporarily unset proxy
unset HTTP_PROXY HTTPS_PROXY
curl https://api.github.com  # Direct connection (no credential injection)
```

## HMAC Secret Rotation

Each sandbox has a per-sandbox HMAC secret used to authenticate git operations between the git wrapper (inside the sandbox) and the git API (in the proxy). Secrets are stored as files under `/run/secrets/sandbox-hmac/{sandbox_id}`.

### When to Rotate

- Suspected secret compromise
- Routine security hygiene (recommended: on sandbox recreation)
- After a security incident involving the proxy or sandbox containers

### Pre-Rotation Checklist

1. **Identify the sandbox** whose secret needs rotation:
   ```bash
   SANDBOX_ID="<your-sandbox-name>"
   ```

2. **Verify current secret exists**:
   ```bash
   docker exec unified-proxy ls -la /run/secrets/sandbox-hmac/${SANDBOX_ID}
   ```

3. **Ensure no git operations are in flight** from the target sandbox:
   ```bash
   docker logs unified-proxy 2>&1 | grep "${SANDBOX_ID}" | tail -5
   ```

### Rotation Procedure

**Step 1: Generate a new secret**

```bash
NEW_SECRET=$(openssl rand -hex 32)
```

**Step 2: Write the new secret to the proxy's secrets mount**

```bash
# Write to the secrets volume (proxy side)
echo -n "${NEW_SECRET}" | docker exec -i unified-proxy \
  tee /run/secrets/sandbox-hmac/${SANDBOX_ID} > /dev/null
```

**Step 3: Write the new secret to the sandbox's secrets mount**

```bash
# Write to the secrets volume (sandbox side)
echo -n "${NEW_SECRET}" | docker exec -i sandbox-${SANDBOX_ID} \
  tee /run/secrets/sandbox-hmac/${SANDBOX_ID} > /dev/null
```

**Step 4: Flush the proxy's secret cache**

The git API caches secrets in memory. Force a re-read by restarting the git API process, or call the rotation admin function:

```bash
# Option A: Restart the git API (brief interruption to git operations)
docker exec unified-proxy kill -HUP $(docker exec unified-proxy cat /var/run/proxy/mitmproxy.pid)

# Option B: Full proxy restart (if Option A is insufficient)
docker compose -f docker-compose.credential-isolation.yml restart unified-proxy
```

### Post-Rotation Verification

1. **Verify the sandbox can still perform git operations**:
   ```bash
   docker exec sandbox-${SANDBOX_ID} git -C /workspace status
   ```

2. **Check for authentication errors in proxy logs**:
   ```bash
   docker logs unified-proxy 2>&1 | grep -i "invalid.*signature\|auth.*fail" | tail -10
   ```

3. **Verify the nonce cache was cleared** (prevents stale nonce collisions):
   ```bash
   docker logs unified-proxy 2>&1 | grep -i "rotate\|revoke" | tail -5
   ```

### Bulk Rotation (All Sandboxes)

For rotating all sandbox secrets at once (e.g., after a security incident):

```bash
# List all active sandboxes
for secret_file in /run/secrets/sandbox-hmac/*; do
  SANDBOX_ID=$(basename "$secret_file")
  NEW_SECRET=$(openssl rand -hex 32)

  # Update proxy side
  echo -n "${NEW_SECRET}" | docker exec -i unified-proxy \
    tee /run/secrets/sandbox-hmac/${SANDBOX_ID} > /dev/null

  # Update sandbox side
  echo -n "${NEW_SECRET}" | docker exec -i sandbox-${SANDBOX_ID} \
    tee /run/secrets/sandbox-hmac/${SANDBOX_ID} > /dev/null 2>&1 || \
    echo "Warning: Could not update sandbox ${SANDBOX_ID} (may be stopped)"
done

# Restart proxy to flush all cached secrets
docker compose -f docker-compose.credential-isolation.yml restart unified-proxy
```

### Troubleshooting Rotation Issues

| Symptom | Cause | Resolution |
|---------|-------|------------|
| `git proxy authentication failed` after rotation | Secret mismatch between proxy and sandbox | Verify both sides have identical secret content |
| `Unknown sandbox or missing secret` | Secret file not written to proxy | Check `/run/secrets/sandbox-hmac/` contains the file |
| Rotation works but reverts on restart | Secrets volume not persisted | Ensure secrets are on a Docker volume, not tmpfs |

## See Also

- [Architecture](architecture.md) - System components
- [Observability](observability.md) - Metrics and debugging
- [Certificates](certificates.md) - CA management
- [ADR-005](adr/005-failure-modes.md) - Failure modes and recovery design
- [ADR-004](adr/004-dns-integration.md) - DNS integration decision
