## hfschema

`hfschema` (CLI aliases: `hfschema` / `schemas-hf`) sits on top of
Hugging Face Datasets + HF Hub and focuses on one job:

- turn HF datasets or local tabular files into small, typed Python
  dataclasses and loader templates you can reuse across projects.

It deliberately does **not** own your raw data or training loops – it
just manages schemas, a bit of metadata, and thin convenience CLIs.

---

## Mental model

There are three layers to keep in mind:

- **HF datasets** (public or private): `owner/name[:config]`, loaded via
  `datasets.load_dataset`.
- **Local tabular files**: JSONL / Parquet / CSV that you want to treat
  *as if* they were HF datasets.
- **Generated schema modules**: small Python files under
  `schemas_hf/datasets/*.py` that define dataclasses mirroring a
  dataset’s fields.

Typical flows:

- For an existing HF dataset:
  - `hfschema download owner/name`  
    → calls `load_dataset_builder` and then generates schema modules.
  - `hfschema list`  
    → shows every dataset[:config] that currently has a schema.
  - `hfschema instantiate owner/name --dest ./schemas`  
    → copies one generated dataclass module into your own repo.
  - Optional: `hfschema init-loader ...` to scaffold a streaming loader.

- For local Parquet / JSONL / CSV that you want on HF:
  - `hfschema publish-hf uwunion/my-dataset --format parquet --path data/*.parquet`  
    → builds a script‑free HF dataset (private by default).
  - `hfschema sync --include uwunion/my-dataset`  
    or `hfschema download uwunion/my-dataset`  
    → generates schema modules for that dataset.

- For purely local schemas (no HF upload):
  - `hfschema make-local-schema local/thing -f jsonl -p raw/*.jsonl -o ./schemas_local`  
    → infers a dataclass from sample rows and writes a module you can
      import directly.

Where schema files go by default:

- `hfschema sync` and `hfschema download` write under
  `src/schemas_hf/datasets/` inside this package.
- You can override the output directory with `--output` when you want
  project‑specific schemas.

---

## Key commands (cheat sheet)

- Introspection:
  - `hfschema list` – list all dataset[:config] schemas present.
  - `hfschema info owner/name[:config]` – show HF metadata.

- Schema generation:
  - `hfschema sync` – scan caches, generate schemas for everything.
  - `hfschema sync --include owner/name[:config]` – only selected ids.
  - `hfschema download owner/name[:config]` – one‑shot
    “pull metadata + generate schemas” for specific ids.

- Publishing local data:
  - `hfschema publish-hf owner/name --format {parquet,csv,jsonl} --path ...`  
    → one split at a time, script‑free HF dataset.

- Local‑only:
  - `hfschema make-local-schema local/name -f jsonl -p 'raw/*.jsonl' -o ./schemas_local`

- Embeddings:
  - `hfschema embeddings add --space text_gemma --model google/gemma-2-9b-it`  
    → declare a logical embedding space.
  - `hfschema embeddings link dataset field --space text_gemma --field text`  
    → annotate that a given field should be embedded in that space.

---

## Images, audio, and other asset‑like columns

When you run `publish-hf` on Parquet sources, `hfschema` inspects the
schema for “asset‑like” columns such as:

- `image_path`, `audio_path`, `file_path`, `rel_path`

and emits a warning:

- by default, **only the table is uploaded** – HF sees the path strings,
  not the underlying JPEG / WAV / binary files.

For image columns specifically, you can turn path strings into a proper
`datasets.Image` feature so that the actual JPEGs get uploaded to HF:

- `--images` / `--no-images` on `publish-hf`:
  - `--images`: convert the first detected image path column into
    `Image` and upload the images.
  - `--no-images`: push metadata only, keep the path strings.
  - default: prompt if image paths are detected.

Privacy detail:

- When `--images` is enabled, `hfschema` **drops the original path
  column** before pushing to HF.
  - This avoids leaking absolute local paths such as
    `/home/you/workspace/.../img_align_celeba/000017.jpg`.
  - The HF dataset ends up with an `image` column plus your other
    metadata, but no `image_path` column.

For non‑image assets (audio, EEG, mocap, etc.), `hfschema` does **not**
try to invent custom feature types – see the next section for the
recommended pattern.

---

## Heavy assets (EEG, mocap, etc.): “reference dataset” pattern

For domains like EEG (`dmt-brains`), mocap (`cmu-mocap`), or large 3D
assets, shipping the raw binary blobs through HF Datasets often isn’t
worth the complexity or storage cost. Instead, the recommended pattern
is:

1. **Create a metadata table** with one row per logical unit you care
   about (e.g. per subject + condition for EEG).
2. Include *portable* references to the actual files:
   - `relative_path` or `file_path` within some external store.
   - optionally a pre‑built `remote_url` or `hf_path` if you know where
     you’ll host the binaries.
3. Use `hfschema publish-hf` on that table.
4. Let consumer code resolve the references however it wants
   (HTTP, S3, local mount, etc.).

### Example: `dmt-brains`

The `dmt-brains` materialized Parquet (from a separate pipeline) is a
subject‑level table with columns like:

- `subject_id`, `file_count`, `has_all_conditions`, …
- `dataset_name`, `modality`, `paradigm`, `drug`
- `conditions_json` – a JSON blob keyed by condition (`DMT`, `EC`,
  `EO`), each holding:
  - `filename`, `relative_path`, `file_format`, `file_size_mb`, …

This already contains everything needed to *locate* the EEG files
without embedding them in HF.

A convenient “reference dataset” is a flattened view with one row per
`(subject_id, condition)`:

- `subject_id`
- `condition` (e.g. `"DMT"`, `"EC"`, `"EO"`)
- `filename`, `relative_path`
- `file_format`, `file_size_mb`
- `dataset_name`, `modality`, `paradigm`, `drug`
- optionally `remote_url` or `hf_path`

This flattening step is intentionally **outside** of `hfschema` so we
don’t couple to any particular upstream (warp, local pipelines, etc.).
One simple helper script (in your own repo) might:

- read the original `dmt-brains` Parquet,
- `json.loads` the `conditions_json` column,
- emit one row per condition,
- write `dmt_brains_conditions.parquet`.

You then publish and schema‑ify it:

```bash
hfschema publish-hf uwunion/dmt-brains-conditions \
  --format parquet \
  --path /path/to/dmt_brains_conditions.parquet \
  --split train --private

hfschema sync -i uwunion/dmt-brains-conditions
```

From any machine:

```python
from datasets import load_dataset
from schemas_hf.datasets.uwunion__dmt_brains_conditions import DmtBrainsConditionsRecord

ds = load_dataset("uwunion/dmt-brains-conditions", split="train")
row = DmtBrainsConditionsRecord.from_example(ds[0])

# `row.relative_path` (and optionally `row.remote_url` / `row.hf_path`)
# tells your application where to fetch the EEG file.
```

You can choose where the raw EEG lives:

- **HF‑hosted binaries**: upload BDF files into the same HF dataset
  repo (e.g. under `files/`) and encode their HF paths or URLs in the
  metadata table.
- **External store**: keep EEG in S3 / GCS / OpenNeuro and encode
  stable HTTP(S) URLs in the metadata.

In both cases, `hfschema` only cares about the table; consumer code
decides how to turn `relative_path` / `remote_url` / `hf_path` into
bytes.

---

## Design principles

- **HF‑centric, script‑free datasets**:
  - Prefer Parquet/CSV/JSONL HF datasets over legacy script‑based ones.
  - `publish-hf` always produces script‑free datasets that work on
    modern `datasets` versions.

- **No hard dependency on upstream pipelines**:
  - `hfschema` does not know about `warpdatasets`, custom crawlers, or
    any particular cache layout.
  - Upstream code is responsible for materializing Parquet/JSONL/CSV in
    a shape that’s convenient to publish; `hfschema` just provides
    schema tooling on top.

- **Privacy‑aware defaults**:
  - Image upload mode drops the original path column.
  - Metadata‑only mode leaves paths untouched; if you plan to publish
    them, prefer relative paths or pre‑sanitized values.

If you find yourself repeating the same “flatten → publish‑hf → sync”
pattern for multiple datasets, that’s usually a sign to factor the
flattening into your own data pipeline – `hfschema` is meant to stay
thin and generic.

