# viser4d

viser4d is a small wrapper around `viser` that adds a time dimension. It records
scene operations across timesteps, supports timeline-synced audio playback, and
plays them back client-locally in each browser tab.

## Quickstart

```bash
pip install viser4d
```

```python
import numpy as np
import viser4d

server = viser4d.Viser4dServer(num_steps=10, fps=10)

with server.at(0):
    points = np.random.uniform(-1.0, 1.0, size=(200, 3))
    point_cloud = server.scene.add_point_cloud(
        "/points",
        points=points,
        colors=(255, 200, 0),
    )

for i in range(1, 10):
    with server.at(i):
        points = np.random.uniform(-1.0, 1.0, size=(200, 3))
        point_cloud.points = points

server.sleep_forever()
```

Open the viewer in your browser and use the built-in Playback controls to play,
scrub, and step through the client-local timeline.

## Timeline model

- The built-in browser controls (`Play`, `Pause`, `Prev`, `Next`, and the
  `Timestep` slider) are client-local. Different tabs can be on different
  timesteps at the same time.
- The `fps=` passed to `Viser4dServer(...)` defines the timeline step rate used
  for audio timing and `.viser` export, and also serves as the initial client
  playback speed.
- `server.on_timestep_change(...)` fires whenever any client commits a new
  discrete timestep and passes `(client, timestep)`. With multiple clients, it
  is an aggregate event stream and may repeat timesteps or arrive out of order.
- `server.play(...)` and `server.pause()` broadcast playback commands to the
  clients that are connected right now. They do not create a shared server
  clock.

## Streaming ingest

If data arrives incrementally, initialize components at `t=0` and then record
updates as each new frame arrives:

```python
import numpy as np
import viser4d

num_steps = 180
server = viser4d.Viser4dServer(num_steps=num_steps, fps=30)

def get_next_points() -> np.ndarray:
    # Replace with your real sensor/network/pipeline frame source.
    return np.random.normal(size=(400, 3)).astype(np.float32)

with server.at(0):
    point_cloud = server.scene.add_point_cloud(
        "/stream/points",
        points=get_next_points(),
    )

for t in range(1, num_steps):
    points = get_next_points()
    with server.at(t):
        point_cloud.points = points

server.sleep_forever()
```

## Timestep callbacks

If you have your own visualization logic and just want to use viser4d's timeline
infrastructure, you can register a callback that fires whenever any connected
client commits a new discrete timestep:

```python
import viser
import viser4d

server = viser4d.Viser4dServer(num_steps=100)

def on_timestep(client: viser.ClientHandle, t: int) -> None:
    update_video_frame(client.scene, t)
    update_client_overlays(client.scene, t)

server.on_timestep_change(on_timestep)
server.sleep_forever()
```

With multiple clients, this callback is aggregate: if two tabs both visit
timestep `3`, it will fire twice, once for each client.

## Server playback commands

`server.play(...)` starts each connected client from that client's own current
timestep. Passing `fps=...` to `server.play(...)` also updates the default
client playback speed for later `play()` calls and future clients.
`server.pause()` pauses each connected client wherever it currently is.
`server.set_fps(...)` updates the same default playback speed without starting
playback. Neither method changes the timeline step rate used for audio timing
or export; set that with `fps=` when you construct the server. New clients
always start paused at timestep `0`.

## Serialize `.viser` recordings

To serialize the full viser4d timeline, including audio, use `server.serialize()`:

```python
import viser4d

server = viser4d.Viser4dServer(num_steps=100)
# ... record timeline data ...
blob = server.serialize(start_timestep=0, end_timestep=None)
```

Write the returned bytes to disk yourself if needed.

## Streaming audio append

For audio that arrives incrementally, create a track once inside `at(t)` and
append chunks through the returned handle:

```python
import numpy as np
import viser4d

server = viser4d.Viser4dServer(num_steps=300, fps=30)

with server.at(0):
    audio = server.audio.add_track(
        "/stream/audio",
        data=np.zeros(1600, dtype=np.float32),
        sample_rate=16000,
    )

for _ in range(120):
    chunk = np.random.uniform(-0.05, 0.05, size=(1600,)).astype(np.float32)
    audio.append(chunk)
```

`AudioHandle.append(...)` extends the same track contiguously (same channel
count).

## How it works

Context determines behavior. `server.scene` is viser's normal scene API, but
its websocket target is swapped while you're inside an `at(t)` context:

```
Inside at(t):                          Outside at(t):
─────────────                          ──────────────
scene.add_frame(...)                   scene.add_frame(...)
       │                                      │
       ▼                                      ▼
    records to Timeline                    forwards to live viser scene
```

- **Inside `at(t)`**: Operations are recorded to a timeline, not executed.
- **Outside `at(t)`**: Operations forward directly to viser's live scene.
- **Client playback**: Each browser tab owns its own transport and playback state.
- **Timestep callbacks**: `on_timestep_change(...)` aggregates committed client steps and passes the source client.
- **Audio**: Add timeline-synced tracks with `server.audio.add_track(...)`.

See `examples/` for more.

## Quality checks

```bash
uvx ruff format .
uvx ruff check .
uvx ty check
pnpm run typecheck:runtime
pnpm run build:runtime
```

## Tests

```bash
pnpm run build:runtime
uv run --group dev pytest -q
```
