# Saving and loading weights and optimizer state

During training, you'll need to save checkpoints for two main purposes: *sampling* (to test your model) and *resuming training* (to continue from where you left off). The `TrainingClient` provides three methods to handle these cases:

1. `save_weights_for_sampler()`: saves a copy of the model weights that can be used for sampling.
2. `save_state()`: saves the weights and the optimizer state. You can fully resume training from this checkpoint.
3. `load_state()`: load the weights and the optimizer state. You can fully resume training from this checkpoint.

Note that (1) is faster and requires less storage space than (2).

Both `save_*` functions require a `name` parameter---a string that you can set to identify the checkpoint within the current training run. For example, you can name your checkpoints `"0000"`, `"0001"`, `"step_1000"`, etc.

The return value contains a `path` field, which is a fully-qualified path, which will look something like `tinker://<model_id>/<name>`. This path is persistent and can be loaded later by a new `ServiceClient` or `TrainingClient`.

### Example: Saving for sampling

```python
# Setup
import tinker
service_client = tinker.ServiceClient()
training_client = service_client.create_lora_training_client(
    base_model="meta-llama/Llama-3.2-1B", rank=32
)

# Save a checkpoint that you can use for sampling
sampling_path = training_client.save_weights_for_sampler(name="0000").result().path

# Create a sampling client with that checkpoint
sampling_client = service_client.create_sampling_client(model_path=sampling_path) #
```

**Shortcut:** Combine these steps with:

```python
sampling_client = training_client.save_weights_and_get_sampling_client()
```

### Example: Saving to resume training

Use `save_state()` and `load_state()` when you need to pause and continue training with full optimizer state preserved:

```python
# Save a checkpoint that you can resume from
resume_path = training_client.save_state(name="0010").result().path

# Load that checkpoint
training_client.load_state(resume_path)
```

### When to use `save_state()` and `load_state()`:


- Multi-step training pipelines (e.g. supervised learning followed by reinforcement learning)
- Adjusting hyperparameters or data mid-run
- Recovery from interruptions or failures
- Any scenario where you need to preserve exact optimizer state (momentum, learning rate schedules, etc.)

## Rolling checkpoints

The built-in training loops (SL, RL, DPO) support **rolling checkpoints**: cheap resume points saved at a finer interval than periodic checkpoints. Unlike periodic checkpoints which save both training state and sampler weights, rolling checkpoints only save training state (skipping the sampler-weight export), making them faster and cheaper. This is useful when `save_every` is set to a large value for storage reasons, but you still want frequent resume points in case of interruptions.

### How it works

- A resume-only checkpoint (training state without sampler weights) is saved every `rolling_save_every` steps.
- After each successful save, the previous rolling checkpoint is **deleted** to bound storage to at most one rolling checkpoint at any time.
- A short TTL (`rolling_ttl_seconds`, default 2 hours) acts as a safety net: if the process crashes before deleting the old checkpoint, the server cleans it up automatically.
- Rolling saves are skipped on steps where a periodic `save_every` checkpoint fires (the periodic checkpoint is strictly more valuable).
- In async training loops (SL, RL), rolling saves run in the background and overlap with the next training step.

### Configuration

Add these fields to your training config:

```python
config = sl_train.Config(
    # ... other fields ...
    save_every=20,           # periodic checkpoints (state + sampler, 7d TTL)
    rolling_save_every=1,    # rolling checkpoints every step (resume-only, auto-cleaned)
    rolling_ttl_seconds=7200,  # 2-hour TTL safety net (default)
)
```

Set `rolling_save_every=0` (the default) to disable rolling checkpoints.

### Resume behavior

No special handling is needed for resume. Rolling checkpoints are recorded in `checkpoints.jsonl` with a `state_path`, so `get_last_checkpoint()` picks them up automatically if they are the most recent checkpoint. The training loop resumes from whatever checkpoint is last in the file, whether it's a periodic or rolling checkpoint.
