# Relative Actions (Cosmos)

Cross-embodiment action representation ported from [Cosmos-Predict2.5](https://github.com/nvidia/cosmos-predict2).

---

## The Problem

Different robots have different kinematics. A "move forward 5cm" on a Franka arm requires completely different joint angles than on a SO-100 or G1. Absolute joint positions don't transfer across embodiments.

## The Solution: Gripper-Frame Relative Actions

Cosmos-Predict2.5 computes actions as **relative displacements in the gripper's local coordinate frame**:

```mermaid
graph TD
    subgraph "Previous State"
        P1["Gripper at (x₁, y₁, z₁)<br/>Rotation R₁"]
    end
    subgraph "Current State"
        P2["Gripper at (x₂, y₂, z₂)<br/>Rotation R₂"]
    end
    subgraph "Relative Action"
        RA["Δxyz = R₁ᵀ · (xyz₂ - xyz₁)<br/>Δrot = euler(R₁ᵀ · R₂)<br/>gripper_state"]
    end

    P1 --> RA
    P2 --> RA

    style RA fill:#e65100,color:#fff
```

**Key formula:**

```python
rel_xyz = prev_rotm.T @ (curr_xyz - prev_xyz)    # Position delta in gripper frame
rel_rot = rotm2euler(prev_rotm.T @ curr_rotm)     # Rotation delta
action  = [rel_xyz(3), rel_rot(3), gripper(1)]     # 7-DoF output
```

**Why this works:** The same physical movement (e.g., "push forward 2cm") produces the **same numbers** regardless of robot geometry, workspace origin, or coordinate convention. A 2cm push is `[0.02, 0, 0, 0, 0, 0, grip]` on any robot.

---

## Implementation

### Rotation Math

```python
from neon.data.relative_actions import euler2rotm, rotm2euler, rotm2quat

# Euler angles (ZYX convention) ↔ rotation matrix
R = euler2rotm([roll, pitch, yaw])    # (3,) → (3, 3)
angles = rotm2euler(R)                 # (3, 3) → (3,)
quat = rotm2quat(R)                   # (3, 3) → (4,) [w, x, y, z]
```

### State-to-Action Conversion

```python
from neon.data.relative_actions import states_to_relative_actions

# Convert end-effector state sequence to relative actions
actions = states_to_relative_actions(
    arm_states=ee_poses,          # (T, 6) [x, y, z, roll, pitch, yaw]
    gripper_states=gripper_widths, # (T,) [0=open, 1=closed]
    action_scaler=20.0,           # Cosmos default normalization
)
# → (T-1, 7) relative actions
```

### Bridge Dataset Compatible API

```python
from neon.data.relative_actions import get_action_sequence_from_states

# Works directly with Cosmos/Bridge data dicts
actions = get_action_sequence_from_states(
    data={"state": [...], "continuous_gripper_state": [...]},
    fps_downsample_ratio=2,  # 30fps source → 15fps target
    action_scaler=20.0,
)
```

---

## Action Scaler

Cosmos uses a **scale factor of 20.0** for action deltas. This normalizes the tiny EE displacements (~millimeters) to a range suitable for neural network training:

```
raw Δx = 0.002 meters (2mm)
scaled  = 0.002 × 20.0 = 0.04  (network-friendly)
```

Configurable via `DataSourceConfig(action_scaler=20.0)`.

---

## Data Soup Integration

The `cosmos_dreamgen` data source type automatically uses relative actions:

```python
DataSourceConfig(
    name="cosmos-bridge",
    type="cosmos_dreamgen",
    path="/data/bridge",
    weight=2.0,
    use_relative_actions=True,    # Enables Cosmos representation
    action_scaler=20.0,           # Cosmos default
    camera_ids=["head", "hand_0", "hand_1"],  # Multi-camera
)
```

Supports both local data (`videos/` + `annotations/*.json`) and HuggingFace datasets (`nvidia/GR1-100`, `NVIDIA/GR00T-Dreams-GR1`).
