# qRemoteX Operator Runbook

## Overview

qRemoteX is the remote executor agent for the qRaptor platform. It runs close to your data (in your VPC, on-prem, or on GPU clusters) and connects back to the qRaptor hosted runtime (qRunX) via the QRP (qRaptor Routing Protocol) over WebSocket.

---

## Installation

> **Gateway URL:** Your project's gateway URL is shown in the Studio UI setup wizard.
> Format: `wss://exec-<shortId>.qraptor.app/ws/executor`

### Docker (Recommended)

```bash
docker run -d \
  --name qremotex \
  -e QRAPTOR_API_KEY=<your-api-key> \
  -e QRAPTOR_GATEWAY_URL=wss://exec-<project-shortid>.qraptor.app/ws/executor \
  -e QRAPTOR_EXECUTOR_NAME=my-executor \
  -e QRAPTOR_MAX_CONCURRENCY=25 \
  qraptor/qremotex:latest
```

### pip

```bash
pip install qremotex

qremotex \
  --api-key <your-api-key> \
  --gateway-url wss://exec-<project-shortid>.qraptor.app/ws/executor \
  --name my-executor \
  --max-concurrency 25
```

### Binary

Download the appropriate binary from the releases page:

```bash
./qremotex \
  --api-key <your-api-key> \
  --gateway-url wss://exec-<project-shortid>.qraptor.app/ws/executor \
  --name my-executor \
  --max-concurrency 25
```

### Kubernetes (Helm)

```bash
helm repo add qraptor https://charts.qraptor.app
helm install qremotex qraptor/qremotex \
  --set apiKey=<your-api-key> \
  --set gatewayUrl=wss://exec-<project-shortid>.qraptor.app/ws/executor \
  --set executor.name=my-executor \
  --set executor.maxConcurrency=25
```

---

## Configuration Reference

| Environment Variable | CLI Flag | Default | Description |
|---------------------|----------|---------|-------------|
| `QRAPTOR_API_KEY` | `--api-key` | (required) | API key for authentication |
| `QRAPTOR_GATEWAY_URL` | `--gateway-url` | (required) | WebSocket URL for gateway connection |
| `QRAPTOR_EXECUTOR_NAME` | `--name` | hostname | Human-readable executor name |
| `QRAPTOR_MAX_CONCURRENCY` | `--max-concurrency` | 25 | Max concurrent executions |
| `QRAPTOR_HEARTBEAT_INTERVAL` | `--heartbeat-interval` | 30 | Heartbeat interval (seconds) |
| `QRAPTOR_SCRIPT_TIMEOUT` | `--script-timeout` | 300 | Default script execution timeout (seconds) |
| `QRAPTOR_LOG_LEVEL` | `--log-level` | INFO | Log verbosity (DEBUG/INFO/WARNING/ERROR) |
| `QRAPTOR_LABELS` | `--labels` | {} | JSON labels for routing (e.g., `{"region":"us-east","gpu":"true"}`) |
| `QRAPTOR_TLS_CERT` | `--tls-cert` | (none) | Path to mTLS client certificate |
| `QRAPTOR_TLS_KEY` | `--tls-key` | (none) | Path to mTLS client key |
| `QRAPTOR_SANDBOX_MODE` | `--sandbox` | docker | Script sandbox: `docker`, `subprocess`, `none` |
| `QRAPTOR_CHUNK_SIZE` | `--chunk-size` | 65536 | Chunk size for large result streaming (bytes) |

---

## Troubleshooting

### Connection Failures

**Symptom:** Executor fails to connect or repeatedly disconnects.

1. **Verify API key is valid:** Check the API Keys page in the Studio UI.
2. **Verify gateway URL:** Must match `wss://exec-<shortId>.qraptor.app/ws/executor` (shown in Studio UI setup wizard).
3. **Check network:** Outbound WebSocket to port 443 must be allowed. Proxies may block WebSocket upgrade.
4. **Check gateway activation:** The executor gateway must be activated for the project.
5. **Check close codes:**
   - `4001` — Invalid API key
   - `4002` — Expired or revoked API key
   - `4003` — Plan executor limit exceeded
   - `4004` — Project mismatch (key from wrong project)

**Resolution:**
```bash
# Test connectivity
wscat -c wss://exec-<shortid>.qraptor.app/ws/executor

# Check executor logs for auth errors
docker logs qremotex --tail 50
```

### API Key Issues

**Symptom:** `Close code 4001` or `4002`.

1. Go to Studio UI → Project → Executors → API Keys
2. Verify the key is not expired or revoked
3. Rotate the key if compromised: Studio UI → API Keys → Rotate
4. Check that the key belongs to the correct project

### Timeout Tuning

**Symptom:** Long-running scripts fail with timeout errors.

- Per-script timeout: Set via QRP `timeout_seconds` (controlled by qRunX node config)
- Default timeout: `QRAPTOR_SCRIPT_TIMEOUT` environment variable
- Dual timeout: qRunX has its own outer timeout; ensure executor timeout < qRunX timeout
- For GPU workloads: Set `QRAPTOR_SCRIPT_TIMEOUT=3600` (1 hour)

### High Memory Usage

**Symptom:** Executor process uses excessive memory.

1. Check `QRAPTOR_MAX_CONCURRENCY` — lower it to reduce concurrent sandbox processes
2. For Docker sandbox: Check Docker container memory limits
3. For large results: Results >1 MB are chunked automatically; verify chunk assembly

---

## Log Interpretation

### Structured JSON Logs

qRemoteX emits structured JSON logs with these key fields:

| Field | Description |
|-------|-------------|
| `level` | Log level (DEBUG/INFO/WARNING/ERROR) |
| `msg` | Log message |
| `executor_id` | This executor's ID |
| `subscription_id` | Subscription context |
| `project_id` | Project context |
| `trace_id` | W3C trace ID for request correlation |
| `request_id` | QRP message request_id |
| `event` | Event type (connect, heartbeat, execute, result, error) |

### Key Log Messages

```
INFO  | event=connected | gateway=wss://... — Successfully connected to gateway
INFO  | event=heartbeat | seq=42 | latency_ms=12 — Heartbeat acknowledged
INFO  | event=execute_script | request_id=abc-123 — Script execution started
INFO  | event=execution_complete | request_id=abc-123 | duration_ms=1523 — Script completed
ERROR | event=execution_error | request_id=abc-123 | error=TimeoutError — Script timed out
WARN  | event=reconnecting | attempt=3 | backoff_ms=8000 — Reconnecting after disconnect
```

---

## Upgrade Procedures

### Docker

```bash
docker pull ghcr.io/qraptor/qremotex:latest
docker stop qremotex
docker rm qremotex
# Re-run with same env vars
docker run -d --name qremotex ...
```

### Kubernetes

```bash
helm upgrade qremotex qraptor/qremotex --reuse-values
```

### pip

```bash
pip install --upgrade qremotex
# Restart the process
```

### Rolling Upgrade (Zero Downtime)

1. Start new executor instance alongside existing one
2. Wait for new instance to register (status=online)
3. Drain old instance: Studio UI → Executor → Drain
4. Wait for in-flight tasks to complete
5. Shut down old instance

---

## Health Checks

### HTTP Health Endpoint

qRemoteX exposes a local health endpoint (disabled by default):

```bash
# Enable with:
QRAPTOR_HEALTH_PORT=8080

# Check:
curl http://localhost:8080/health
# → {"status":"healthy","connected":true,"active_tasks":3,"uptime_seconds":86400}
```

### Kubernetes Probes

```yaml
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 30
readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
```

---

## Security Considerations

- **API keys are secrets:** Store them in your secret manager (Vault, AWS Secrets Manager, K8s Secrets). Never commit API keys to version control.
- **Network isolation:** qRemoteX only needs outbound WebSocket to the gateway. No inbound ports required.
- **mTLS (optional):** For additional security, configure mTLS with `QRAPTOR_TLS_CERT` and `QRAPTOR_TLS_KEY`.
- **Sandbox mode:** Use `docker` sandbox for untrusted scripts. `subprocess` is lighter but less isolated. `none` disables sandboxing (not recommended for production).
- **Key rotation:** Rotate API keys periodically using the Studio UI or API. Grace period allows both old and new keys during transition.
