# Training, inference, and export

This note provides a minimal, end-to-end path for training, inference, and exporting predictions.

Note: the in-repo trainer under `rtdetr_pose/` is scaffold-first, but supports a **production-style run contract**
(fixed artifact paths, full resume, safety guards, export + parity checks).

## TL;DR (copy-paste)

```bash
python3 -m pip install -r requirements-test.txt
bash tools/fetch_coco128.sh
python3 rtdetr_pose/tools/train_minimal.py \
  --dataset-root data/coco128 \
  --config rtdetr_pose/configs/base.json \
  --max-steps 50 \
  --run-dir runs/train_minimal_smoke
python3 tools/export_predictions.py \
  --adapter rtdetr_pose \
  --config rtdetr_pose/configs/base.json \
  --checkpoint runs/train_minimal_smoke/checkpoint.pt \
  --max-images 20 \
  --wrap \
  --output reports/predictions.json
python3 tools/eval_coco.py \
  --dataset data/coco128 \
  --predictions reports/predictions.json \
  --bbox-format cxcywh_norm \
  --max-images 20 \
  --dry-run
```

## Training (RT-DETR pose scaffold)

1) Install dependencies (CPU PyTorch for local dev):
- python3 -m pip install -r requirements-test.txt

2) Fetch the sample dataset (coco128):
- bash tools/fetch_coco128.sh

Optional: prepare full COCO (val2017/train2017) in YOLO-format (recommended path under `data/`, which is gitignored):

```bash
# Example COCO layout:
#   /data/coco/
#     annotations/instances_val2017.json
#     val2017/*.jpg
python3 tools/prepare_coco_yolo.py \
  --coco-root /data/coco \
  --split val2017 \
  --out data/coco-yolo
```

Notes:
- `tools/prepare_coco_yolo.py` writes `data/coco-yolo/dataset.json` with absolute paths so evaluators can find images without copying.
- To copy images into `data/coco-yolo/images/<split>/`, add `--copy-images` (slower, larger).

3) Run the minimal trainer:
- python3 rtdetr_pose/tools/train_minimal.py --dataset-root data/coco128 --config rtdetr_pose/configs/base.json --max-steps 50 --use-matcher

### Backbone swap (P3/P4/P5 contract)

Backbone is now configurable via `model.backbone.*` and projected to transformer `d_model` via `model.projector.d_model`.

Example config fragment:

```yaml
model:
  backbone:
    name: cspdarknet_s
    norm: bn
    args:
      width_mult: 0.5
      depth_mult: 0.5
  projector:
    d_model: 256
```

Other supported names: `resnet50`, `convnext_tiny`, `cspresnet`, `tiny_cnn`.

Contract details and extension guide: [backbones.md](backbones.md)

Common options:
- --device cuda:0
- --batch-size 4
- --num-queries 10
- --stage-off-steps 1000 --stage-k-steps 1000
- --cost-z 1.0 --cost-rot 1.0 --cost-t 1.0
- --cost-z-start-step 500 --cost-rot-start-step 1000 --cost-t-start-step 1500
- --checkpoint-out reports/rtdetr_pose_ckpt.pt
- --metrics-jsonl reports/train_metrics.jsonl
- --metrics-csv reports/train_metrics.csv

### Depth mode (none / sidecar / fuse_mid)

`rtdetr_pose/tools/train_minimal.py` supports optional depth integration while preserving the backbone swap boundary (`[P3,P4,P5]`):

- `--depth-mode none` (default): no depth path; baseline-compatible behavior.
- `--depth-mode sidecar`: ingest sidecar depth (`depth_path` / `depth`) and propagate per-image `depth_valid`.
- `--depth-mode fuse_mid`: apply lightweight depth fusion after projector (outside backbone), with optional `--depth-dropout` modality dropout.

Safety and unit handling:

- `--depth-unit` controls whether absolute-depth constraints are allowed (`unspecified|relative|metric`, default `unspecified`).
- In non-metric modes, absolute-depth matcher costs are disabled for safety (`cost_z`, `cost_t`).
- `--depth-scale` applies scaling to sidecar depth values before fusion/consumption.

Example (mixed sidecar depth ingestion):

```bash
python3 rtdetr_pose/tools/train_minimal.py \
  --dataset-root data/coco128 \
  --config rtdetr_pose/configs/base.json \
  --use-matcher \
  --depth-mode sidecar \
  --depth-unit metric \
  --depth-scale 1.0
```

### Config source-of-truth and key mapping

`rtdetr_pose/tools/train_minimal.py` reads YAML/JSON via `--config`, then applies explicit CLI flags on top.

- Priority: **CLI flags > config file > built-in defaults**
- Config keys use argparse destination names (`--weight-decay` -> `weight_decay`)
- Alias: `grad_accum` in config is accepted and mapped to `gradient_accumulation_steps`
- In strict run-contract mode (`run_contract` or `run_id` or `config_version` present), unknown config keys fail fast

### Optimizer / solver options (code-accurate)

Supported optimizer choices (`--optimizer`):
- `adamw` (default)
- `sgd`

Relevant parameters:

| Key (CLI / config) | Type | Default | Choices / Notes |
|---|---:|---:|---|
| `--optimizer` / `optimizer` | str | `adamw` | `adamw`, `sgd` |
| `--lr` / `lr` | float | `1e-4` | base LR |
| `--weight-decay` / `weight_decay` | float | `0.01` | base WD |
| `--momentum` / `momentum` | float | `0.9` | used by SGD |
| `--nesterov` / `nesterov` | bool | `false` | SGD only |
| `--use-param-groups` / `use_param_groups` | bool | `false` | split backbone/head groups |
| `--backbone-lr-mult` / `backbone_lr_mult` | float | `1.0` | group LR multiplier |
| `--head-lr-mult` / `head_lr_mult` | float | `1.0` | group LR multiplier |
| `--backbone-wd-mult` / `backbone_wd_mult` | float | `1.0` | group WD multiplier |
| `--head-wd-mult` / `head_wd_mult` | float | `1.0` | group WD multiplier |
| `--wd-exclude-bias` / `wd_exclude_bias` | bool | `true` | set bias WD=0 |
| `--wd-exclude-norm` / `wd_exclude_norm` | bool | `true` | set norm WD=0 |

### LR scheduler options (code-accurate)

Supported scheduler choices (`--scheduler`):
- `none` (default)
- `cosine`
- `onecycle`
- `multistep`

Relevant parameters:

| Key (CLI / config) | Type | Default | Choices / Notes |
|---|---:|---:|---|
| `--scheduler` / `scheduler` | str | `none` | `none`, `cosine`, `onecycle`, `multistep` |
| `--min-lr` / `min_lr` | float | `0.0` | cosine `eta_min` |
| `--scheduler-milestones` / `scheduler_milestones` | str/list | `""` | comma list for multistep |
| `--scheduler-gamma` / `scheduler_gamma` | float | `0.1` | multistep decay |
| `--lr-warmup-steps` / `lr_warmup_steps` | int | `0` | linear warmup steps |
| `--lr-warmup-init` / `lr_warmup_init` | float | `0.0` | LR at warmup step 0 |

Note: `linear` scheduler is **not** a supported value in current code.

### Production run contract (recommended)

For reproducible runs with fixed artifact paths, full resume, best/last checkpoints, and an ONNX parity gate:

```bash
yolozu train configs/examples/train_contract.yaml --run-id exp01

# Resume (from runs/exp01/checkpoints/last.pt)
yolozu train configs/examples/train_contract.yaml --run-id exp01 --resume
```

Contracted artifacts live under `runs/<run_id>/...`:
- `checkpoints/{last,best}.pt`
- `reports/{train_metrics,val_metrics}.jsonl`
- `reports/config_resolved.yaml`
- `reports/run_meta.json`
- `reports/onnx_parity.json`
- `exports/model.onnx` (+ meta JSON)

Full spec: [run_contract.md](run_contract.md)

### Optimizer options

Choose between SGD and AdamW optimizers with configurable learning rates and weight decay:

```bash
# AdamW (default)
python3 rtdetr_pose/tools/train_minimal.py \
  --dataset-root data/coco128 \
  --optimizer adamw \
  --lr 1e-4 \
  --weight-decay 0.01

# SGD with momentum and Nesterov
python3 rtdetr_pose/tools/train_minimal.py \
  --dataset-root data/coco128 \
  --optimizer sgd \
  --lr 0.1 \
  --momentum 0.9 \
  --nesterov \
  --weight-decay 1e-4

# Use parameter groups with different lr/wd for backbone vs head
python3 rtdetr_pose/tools/train_minimal.py \
  --dataset-root data/coco128 \
  --optimizer adamw \
  --lr 1e-4 \
  --use-param-groups \
  --backbone-lr-mult 0.1 \
  --head-lr-mult 1.0 \
  --backbone-wd-mult 0.5 \
  --head-wd-mult 1.0
```

### Learning rate scheduler options

Multiple scheduler types are supported with optional warmup:

```bash
# Cosine annealing with warmup
python3 rtdetr_pose/tools/train_minimal.py \
  --dataset-root data/coco128 \
  --scheduler cosine \
  --min-lr 1e-6 \
  --lr-warmup-steps 500 \
  --lr-warmup-init 1e-6

# OneCycleLR for super-convergence
python3 rtdetr_pose/tools/train_minimal.py \
  --dataset-root data/coco128 \
  --scheduler onecycle

# MultiStepLR with decay at specific steps
python3 rtdetr_pose/tools/train_minimal.py \
  --dataset-root data/coco128 \
  --scheduler multistep \
  --scheduler-milestones 1000,2000,3000 \
  --scheduler-gamma 0.1
```

### LoRA / QLoRA options (code-accurate)

LoRA is enabled when `--lora-r > 0`.

| Key (CLI / config) | Type | Default | Choices / Notes |
|---|---:|---:|---|
| `--lora-r` / `lora_r` | int | `0` | `>0` enables LoRA |
| `--lora-alpha` / `lora_alpha` | float/null | `null` | null means `alpha=r` |
| `--lora-dropout` / `lora_dropout` | float | `0.0` | LoRA input dropout |
| `--lora-target` / `lora_target` | str | `head` | `head`, `all_linear`, `all_conv1x1`, `all_linear_conv1x1` |
| `--lora-freeze-base` / `lora_freeze_base` | bool | `true` | train LoRA-only when true |
| `--lora-train-bias` / `lora_train_bias` | str | `none` | `none`, `all` |

TorchAO / QLoRA integration:

| Key (CLI / config) | Type | Default | Choices / Notes |
|---|---:|---:|---|
| `--torchao-quant` / `torchao_quant` | str | `none` | `none`, `int8wo`, `int4wo` |
| `--torchao-required` / `torchao_required` | bool | `false` | fail run if quant unavailable |
| `--qlora` / `qlora` | bool | `false` | requires `lora_r>0`; forces `torchao_quant=int4wo` (if none) and `lora_freeze_base=true` |

### Additional fine-grained training knobs (selected)

| Key (CLI / config) | Type | Default | Notes |
|---|---:|---:|---|
| `--gradient-accumulation-steps` / `gradient_accumulation_steps` | int | `1` | alias in config: `grad_accum` |
| `--clip-grad-norm` / `clip_grad_norm` | float | `0.0` | `>0` enables clipping |
| `--use-ema` / `use_ema` | bool | `false` | EMA on train weights |
| `--ema-decay` / `ema_decay` | float | `0.999` | EMA decay |
| `--ema-eval` / `ema_eval` | bool | `false` | use EMA weights at eval/export |
| `--amp` / `amp` | str | `none` | `none`, `fp16`, `bf16` (CUDA only) |
| `--use-amp` / `use_amp` | bool | `false` | back-compat alias for `amp=fp16` |
| `--task-aligner` / `task_aligner` | str | `none` | `none`, `uncertainty` |
| `--cost-z` / `cost_z` | float | `0.0` | matcher depth cost |
| `--cost-rot` / `cost_rot` | float | `0.0` | matcher rotation cost |
| `--cost-t` / `cost_t` | float | `0.0` | matcher translation cost |
| `--cost-z-start-step` / `cost_z_start_step` | int | `0` | staged matcher cost gate |
| `--cost-rot-start-step` / `cost_rot_start_step` | int | `0` | staged matcher cost gate |
| `--cost-t-start-step` / `cost_t_start_step` | int | `0` | staged matcher cost gate |
| `--stage-off-steps` / `stage_off_steps` | int | `0` | offsets-only stage |
| `--stage-k-steps` / `stage_k_steps` | int | `0` | k-head-only stage |

Continual-learning regularizers in the same trainer:

| Key (CLI / config) | Type | Default | Notes |
|---|---:|---:|---|
| `--self-distill-from` / `self_distill_from` | str/null | `null` | enable teacher distillation |
| `--self-distill-weight` / `self_distill_weight` | float | `1.0` | distill global weight |
| `--self-distill-temperature` / `self_distill_temperature` | float | `1.0` | logits temperature |
| `--self-distill-kl` / `self_distill_kl` | str | `reverse` | `forward`, `reverse`, `sym` |
| `--self-distill-keys` / `self_distill_keys` | str | `logits,bbox` | comma list |
| `--derpp` / `derpp` | bool | `false` | DER++ replay distillation |
| `--derpp-teacher-key` / `derpp_teacher_key` | str | `derpp_teacher_npz` | record key/path |
| `--derpp-weight` / `derpp_weight` | float | `1.0` | DER++ global weight |
| `--ewc` / `ewc` | bool | `false` | EWC regularizer |
| `--ewc-lambda` / `ewc_lambda` | float | `1.0` | EWC penalty weight |
| `--si` / `si` | bool | `false` | SI regularizer |
| `--si-c` / `si_c` | float | `1.0` | SI penalty weight |
| `--si-epsilon` / `si_epsilon` | float | `1e-3` | SI stabilization |

### Advanced training options

```bash
# Gradient clipping
python3 rtdetr_pose/tools/train_minimal.py \
  --dataset-root data/coco128 \
  --clip-grad-norm 1.0

# Gradient accumulation (effective batch size = batch_size * gradient_accumulation_steps)
python3 rtdetr_pose/tools/train_minimal.py \
  --dataset-root data/coco128 \
  --batch-size 2 \
  --gradient-accumulation-steps 4

# Automatic Mixed Precision (requires CUDA)
python3 rtdetr_pose/tools/train_minimal.py \
  --dataset-root data/coco128 \
  --device cuda:0 \
  --use-amp

# Exponential Moving Average (EMA) of model weights
python3 rtdetr_pose/tools/train_minimal.py \
  --dataset-root data/coco128 \
  --use-ema \
  --ema-decay 0.999 \
  --ema-eval  # Use EMA weights for evaluation/export

# Combined example: SGD + cosine scheduler + param groups + EMA
python3 rtdetr_pose/tools/train_minimal.py \
  --dataset-root data/coco128 \
  --optimizer sgd \
  --momentum 0.9 \
  --scheduler cosine \
  --min-lr 1e-6 \
  --lr-warmup-steps 500 \
  --use-param-groups \
  --backbone-lr-mult 0.1 \
  --use-ema \
  --ema-decay 0.999 \
  --clip-grad-norm 1.0
```

Plot loss curve (requires matplotlib):
- python3 tools/plot_metrics.py --jsonl reports/train_metrics.jsonl --out reports/train_loss.png

## Inference (adapter run)

Use the adapter tools to run inference and produce predictions JSON.

- python3 tools/export_predictions.py --adapter rtdetr_pose --config rtdetr_pose/configs/base.json --checkpoint /path/to.ckpt --max-images 50 --wrap --output reports/predictions.json

Optional TTA:
- python3 tools/export_predictions.py --adapter rtdetr_pose --tta --tta-seed 0 --tta-flip-prob 0.5 --wrap --output reports/predictions_tta.json

Note: TTA here is a lightweight **prediction-space transform** (a post-transform on the exported bboxes). It does not rerun the model on augmented inputs.

Optional TTT (test-time training, pre-prediction):
- Tent (recommended safe preset + guard rails):
	- python3 tools/export_predictions.py --adapter rtdetr_pose --ttt --ttt-preset safe --ttt-reset sample --wrap --output reports/predictions_ttt_safe.json
- MIM (recommended safe preset + guard rails):
	- python3 tools/export_predictions.py --adapter rtdetr_pose --ttt --ttt-preset mim_safe --ttt-reset sample --wrap --output reports/predictions_ttt_mim_safe.json
- Bounded adaptation-cost run (stream + batch/chunk knobs):
  - python3 tools/export_predictions.py --adapter rtdetr_pose --ttt --ttt-preset safe --ttt-reset stream --ttt-batch-size 4 --ttt-max-batches 8 --wrap --output reports/predictions_ttt_stream_b4_k8.json

Notes:
- TTT requires an adapter that supports `get_model()` + `build_loader()` and requires torch.
- TTT updates model parameters in-memory before calling `adapter.predict(records)`.
- `--ttt-batch-size` controls images per adaptation step; `--ttt-max-batches` caps adaptation batches for predictable runtime.
- Recommended comparison protocol and more examples: `docs/ttt_protocol.md`.

## Export predictions for evaluation

If you run inference externally (PyTorch/TensorRT/ONNX), export to the YOLOZU predictions schema.
Then validate and evaluate in this repo.

- python3 tools/validate_predictions.py reports/predictions.json
- python3 tools/eval_coco.py --dataset data/coco128 --predictions reports/predictions.json --bbox-format cxcywh_norm --max-images 50

## Scenario suite (local evaluation)

- `yolozu test configs/examples/test_setting.yaml --adapter precomputed --predictions reports/predictions.json --max-images 50`
- (source checkout) `python3 tools/run_scenarios.py --adapter precomputed --predictions reports/predictions.json --max-images 50`

## Notes
- When using GPU, install CUDA-enabled PyTorch and use --device cuda:0.
- Keep the predictions schema consistent with the adapter output: image path + detections list.

## YOLO26n smoke (RT-DETR scaffold)

This repo includes a tiny “it runs end-to-end” smoke command that:
- fetches `data/coco128` if missing
- runs a few steps of `rtdetr_pose/tools/train_minimal.py`
- exports `model.onnx`
- exports wrapped predictions JSON
- runs `tools/eval_suite.py --dry-run` to validate the full I/O chain

```bash
python3 tools/run_yolo26n_smoke_rtdetr_pose.py
```

Multi-bucket variant (n/s/m/l/x) + bucket configs:
- `python3 tools/run_yolo26_smoke_rtdetr_pose.py --buckets n,s,m,l,x`
- `docs/yolo26_rtdetr_pose_recipes.md`
