# Inference & Hardware Control

This guide covers running Neon in real-time: the inference server, the G1 hardware controller, and the closed-loop control pipeline.

---

## Inference Server

The `NeonInferenceServer` loads a trained model and provides fast action prediction:

```python
from neon.inference.server import NeonInferenceServer

server = NeonInferenceServer(
    model_path="cagataydev/neon-g1-v1",  # HuggingFace model ID or local path
    device="auto",                         # auto-detects CUDA > MPS > CPU
)

output = server.predict(
    image=camera_frame,                    # PIL Image
    instruction="Pick up the red cup",
    proprioception=joint_states,           # numpy array (action_dim,)
    smooth=True,                           # Temporal EMA smoothing
)
```

### Temporal Smoothing

The server applies **exponential moving average** between successive predictions to prevent discontinuities:

```python
# alpha=0.7 by default
new_actions = 0.7 * predicted + 0.3 * previous_actions
```

Higher alpha (0.9) = more responsive but jerkier. Lower alpha (0.5) = smoother but slower to react. Call `server.reset()` between episodes to clear the smoothing state.

### HTTP Server

Run the inference server as an HTTP endpoint:

```bash
python -m neon.inference.server --model cagataydev/neon-g1-v1 --port 8300
```

**Endpoints:**

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/predict` | Predict actions from image + instruction |
| `POST` | `/reset` | Reset episode state (clear smoothing) |
| `GET` | `/health` | Server status and model info |

**Request format:**

```json
{
    "instruction": "Pick up the red cup",
    "image_base64": "<base64-encoded JPEG>",
    "proprioception": [0.0, 0.3, 0.0, -0.75, ...]
}
```

**Response:**

```json
{
    "actions": [[0.12, -0.34, ...], ...],
    "raw_actions": [[0.17, -0.51, ...], ...]
}
```

---

## G1 Hardware Controller

The `G1Controller` interfaces with the physical Unitree G1 robot:

```python
from neon.inference.g1_controller import G1Controller, G1Config

config = G1Config(
    robot_ip="192.168.123.10",
    control_frequency=50.0,       # 50 Hz control loop
    mode="arms_only",
    max_joint_velocity=2.0,       # Safety: rad/s
    max_linear_velocity=0.5,      # Safety: m/s
    max_angular_velocity=0.5,     # Safety: rad/s
    enable_safety_checks=True,
)

controller = G1Controller(config)
controller.connect()
```

### Safety Limits

Every action command passes through safety checks before reaching the robot:

1. **Joint limits** — Clipped to each joint's physical range (from URDF)
2. **Velocity limits** — Maximum joint velocity to prevent violent motions
3. **Locomotion limits** — Maximum linear and angular velocity
4. **Emergency stop** — `controller.disconnect()` sends zero velocity

!!! danger "Always enable safety checks"
    The G1 is a 1.2m, 35kg humanoid. Uncontrolled joint movements can damage the robot or harm people. Never set `enable_safety_checks=False` in production.

---

## Closed-Loop Control

The full pipeline: observe → predict → act → repeat.

```python
# Full closed-loop control session
controller.run_control_loop(
    model=server,                 # Inference server or NeonVLA model
    instruction="Pick up the red cup",
    max_steps=200,                # ~4 seconds at 50 Hz
    action_chunk_index=0,         # Execute first step of each predicted chunk
)
```

### How It Works

```mermaid
sequenceDiagram
    participant Cam as Camera
    participant Ctrl as Controller
    participant Model as Neon Model
    participant Robot as G1 Robot

    loop Every 20ms (50 Hz)
        Cam->>Ctrl: Camera frame
        Ctrl->>Ctrl: Get joint states
        Ctrl->>Model: predict(image, instruction, joints)
        Model->>Ctrl: Action chunk (16 steps)
        Ctrl->>Ctrl: Apply safety limits
        Ctrl->>Robot: Send action[0] (first step)
    end
```

### Control Frequency

The control loop targets 50 Hz (20ms per iteration):

- **Model inference**: ~50ms on Jetson Orin (3B backbone, 4-bit)
- **Camera capture**: ~5ms
- **Communication**: ~2ms
- **Total**: ~57ms → actual frequency ~17 Hz

With action chunking, the robot has 16 steps to execute while the next prediction is computed. This effectively decouples inference latency from control frequency.

---

## Deployment Targets

| Device | GPU | Backbone | Latency | Notes |
|--------|-----|----------|---------|-------|
| **Jetson Orin** (Thor) | 32GB unified | 3B, 4-bit | ~50ms | On-robot, always-on |
| **RTX 4090** | 24GB | 7B, 4-bit | ~30ms | Desktop workstation |
| **A100** (EC2) | 40GB | 7B, 4-bit | ~20ms | Cloud, lowest latency |
| **MacBook M3** | MPS | 3B, 4-bit | ~200ms | Development only |

!!! tip "Run model on Thor, control loop on robot"
    For lowest latency, run the inference server directly on the Jetson Orin mounted on the G1. The controller communicates over localhost.
