= Running pipelines with `sheets-run`

This tool lets you execute declarative pipelines (step chains) on spreadsheet data.
Pipelines describe *what to do*; profiles describe *where to read/write*.

== TL;DR

[source,bash]
----
python -m spreadsheet_handling.cli.run \
  --config scripts/spreadsheet_handling/examples/pipelines.yml \
  --profile pack_json_to_xlsx \
  --pipeline basic \
  -v
----

* `--pipeline` selects the step preset (use-case).
* `--profile` selects the I/O preset (channels + default paths).
* Add `--in-*` / `--out-*` flags to override paths/kinds without editing the YAML.

== Concepts

*Pipeline*::
A named list of steps (e.g. `validate → apply_fks → drop_helpers`). Defines the behavior.

*Profile*::
An I/O preset (e.g. `json_dir → xlsx`) with paths/options. Optionally binds a default pipeline.

== Minimal config (with profiles + pipelines)

[source,yaml]
----
io:
  profiles:
    pack_json_to_xlsx:
      pipeline: basic        # optional binding; used if --pipeline is omitted
      input:  { kind: json_dir, path: scripts/spreadsheet_handling/examples/roundtrip_start_json }
      output: { kind: xlsx,     path: scripts/spreadsheet_handling/tmp/world_out.xlsx }

    unpack_xlsx_to_json:
      input:  { kind: xlsx,     path: scripts/spreadsheet_handling/tmp/world.xlsx }
      output: { kind: json_dir, path: scripts/spreadsheet_handling/tmp/json_out }

pipelines:
  basic:
    - step: validate
      mode_duplicate_ids: warn
      mode_missing_fk:  warn
      defaults:
        id_field: id
        label_field: name
        detect_fk: true
        helper_prefix: "_"
    - step: apply_fks
      defaults:
        id_field: id
        label_field: name
        detect_fk: true
    - step: drop_helpers
      prefix: "_"
----

== Common invocations

Run a profile + pick a pipeline explicitly:
[source,bash]
----
python -m spreadsheet_handling.cli.run --config pipelines.yml --profile pack_json_to_xlsx --pipeline basic
----

Use the pipeline bound in the profile (no `--pipeline` needed):
[source,bash]
----
python -m spreadsheet_handling.cli.run --config pipelines.yml --profile pack_json_to_xlsx
----

Override I/O paths (keep YAML clean):
[source,bash]
----
python -m spreadsheet_handling.cli.run \
  --config pipelines.yml \
  --profile pack_json_to_xlsx \
  --pipeline basic \
  --in-path data/run1/in \
  --out-path build/run1/out.xlsx
----

Switch channels on the fly (no profiles at all):
[source,bash]
----
python -m spreadsheet_handling.cli.run \
  --pipeline basic \
  --in-kind csv_dir  --in-path data/in_csv \
  --out-kind xlsx    --out-path build/out.xlsx
----

== Custom pipelines from a standalone YAML

You can store just the pipeline steps in a separate YAML and pass it via `--pipeline-yaml`.
This *overrides* any `pipelines[...]`/`pipeline` in `--config`.

[pipeline.yml]
[source,yaml]
----
pipeline:
  - step: validate
    mode_duplicate_ids: warn
    mode_missing_fk:  fail
    defaults: { id_field: id, label_field: name, detect_fk: true, helper_prefix: "_" }
  - step: apply_fks
    defaults: { id_field: id, label_field: name, detect_fk: true }
  - step: drop_helpers
    prefix: "_"
----

[source,bash]
----
python -m spreadsheet_handling.cli.run \
  --config io.yml \
  --pipeline-yaml pipeline.yml \
  --profile pack_json_to_xlsx
----

== CLI reference

[source,bash]
----
python -m spreadsheet_handling.cli.run [options]

Options:
  --config FILE          # YAML with `io` and/or `pipelines` / `pipeline`
  --profile NAME         # selects `io.profiles[NAME]`
  --pipeline NAME        # selects `pipelines[NAME]` (overridden by --pipeline-yaml)
  --pipeline-yaml FILE   # standalone pipeline spec (wins over `pipelines[...]`)
  --in-kind KIND         # override input.kind  (e.g. json_dir|csv_dir|xlsx)
  --in-path PATH         # override input.path
  --out-kind KIND        # override output.kind (e.g. json_dir|csv_dir|xlsx)
  --out-path PATH        # override output.path
  -v / -vv               # increase verbosity
----

== Tips & patterns

* Keep *pipelines* reusable; store environment-specific paths in *profiles* or pass them via overrides.
* For `pack` vs. `unpack` style flows, define separate pipelines (behavior differs)
and point profiles to the default one via `pipeline: <name>`.
* Pipelines are pure `Frames -> Frames`; I/O is selected via backends in `io_backends.router`.

== Troubleshooting

*“Unknown profile ‘X’”*::
Check `io.profiles` in your config or pass `--in-kind/--in-path/--out-*` overrides.

*“I/O backend … not available”*::
The selected `kind` is not registered in `io_backends.router`. Add/import the backend and register it.

*Nothing happens*::
Run with `-vv` and verify that a pipeline was selected (via `--pipeline`, profile binding, or top-level `pipeline`).
