---
name: zyplib-dataset
description: Use when an agent needs to build data loading pipelines, create DataLoaders, or set up train/val/test splits using `zyplib.dataset`. Trigger when the user asks to load EEG/signal files into training data, create DataLoaders from numpy arrays, split datasets at file-level or segment-level, set up K-fold cross-validation, or convert raw files into (X, y) pairs for model training.
---

# zyplib-dataset

Build the data pipeline from the agent's perspective: the user gives a training data requirement, and you must shape it into the correct `zyplib.dataset` API combination.

The source code of zyplib.dataset can be viewed in this way:
- First locate the installed/importable `zyplib` package root.

  ```bash
  python -c "import zyplib; from pathlib import Path; print(Path(zyplib.__file__).resolve().parent)"
  ```
- Then use Read/View tools on `<zyplib>/dataset/`.
  - Focus on `make_xy_pipeline.py` for Pipeline layer (file -> numpy).
  - Focus on `torch_data.py` for DataLoader layer (numpy -> DataLoader).
  - Focus on `split.py` for `random_split` utility.
  - Focus on `__init__.py` for public API surface.

## Understand the two-layer architecture

`zyplib.dataset` is organized in two layers:

```
Layer 1: XY Pipeline  (files -> numpy)
  fileseg_pipeline      read + segment + label (EEG typical)
  fileloader_pipeline   read + custom loader (flexible)
  train_test_pipeline   file-level split + pipeline
  kfold_pipeline        file-level K-fold + pipeline

Layer 2: DataLoader Factory  (numpy/Dataset -> DataLoader)
  make_basic_dataloader    from (X, y) tuples or Dataset objects
  make_split_dataloader    split one dataset into multiple loaders
  make_kfold_dataloader    segment-level K-fold loaders
```

Think in this order:

1. **Where does data come from?** -> Choose a Pipeline, or skip if (X, y) already exists.
2. **How to split?** -> File-level (recommended) or segment-level.
3. **How to feed the model?** -> `make_basic_dataloader`.

## Choose the right Pipeline

### fileseg_pipeline (alias: basic_pipeline)

Use when: each file contains a continuous signal that needs segmentation.

Typical EEG flow: read file -> preprocess -> segment into windows -> assign label.

```python
X, y = fileseg_pipeline(
    files,
    fn_read=read_mat,           # (file_path: str) -> ndarray [C, T]
    fn_segment=segment,         # (signal: [C, T]) -> [N, C, T]
    label=1,                    # int or (file_path: str) -> int
    fn_before_segment=preproc,  # optional: (signal: [C, T]) -> [C, T]
    fn_after_segment=None,      # optional: (signal: [N, C, T]) -> [N, C, T]
    fn_encode_label=None,       # optional: default is to_onehot
    random_shuffle=42,          # False | True | int (seed)
)
```

### fileloader_pipeline (alias: simple_pipeline)

Use when: each file needs custom loading logic that returns (X, y) directly.

```python
X, y = fileloader_pipeline(
    files,
    fn_fileloader=my_loader,  # (file_path: str) -> (ndarray, ndarray)
    fn_encode_label=None,
    random_shuffle=42,
)
```

### Decision: which pipeline?

- File contains raw continuous signal that needs windowing -> `fileseg_pipeline`
- File already contains pre-segmented data or needs custom logic -> `fileloader_pipeline`
- Data is already in memory as (X, y) numpy arrays -> skip pipelines, go to DataLoader layer

## Split at file level (recommended)

For EEG/biosignal data, one file typically = one subject. Splitting at segment level causes **data leakage** (same subject in both train and val). Always prefer file-level splitting.

### train_test_pipeline

Splits files first, then runs pipeline on each split independently.

```python
# 3-way split (train/val/test)
X_train, X_val, X_test, y_train, y_val, y_test = train_test_pipeline(
    split_size=(0.7, 0.15, 0.15),
    all_files=(asd_files, td_files),  # grouped -> stratified split
    random_seed=42,
    fn_pipeline=my_load_fn,           # (files: list[str]) -> (X, y)
)

# 2-way split (train/test)
X_train, X_test, y_train, y_test = train_test_pipeline(
    split_size=0.8,
    all_files=all_files,
    random_seed=42,
    fn_pipeline='fileseg',
    pipeline_kwargs=dict(fn_read=read_mat, fn_segment=seg, label=0),
)
```

Key points:
- `all_files` accepts flat list or grouped lists (for stratified split per class)
- `fn_pipeline` can be a callable or a string preset (`'fileseg'`, `'fileloader'`)
- When using string preset, pass pipeline args via `pipeline_kwargs`

### kfold_pipeline

File-level K-fold cross-validation. Returns a generator.

```python
for X_train, X_val, y_train, y_val in kfold_pipeline(
    k=5,
    all_files=(asd_files, td_files),
    random_seed=42,
    fn_pipeline=my_load_fn,
):
    loaders = make_basic_dataloader(
        (X_train, y_train), (X_val, y_val), batch_size=32
    )
    # train this fold...
```

## Build DataLoaders

### make_basic_dataloader

The primary DataLoader factory. Accepts one or more (X, y) tuples or Dataset objects.

```python
# Multiple datasets -> multiple loaders (tuple unpacking)
train_loader, val_loader, test_loader = make_basic_dataloader(
    (X_train, y_train),
    (X_val, y_val),
    (X_test, y_test),
    batch_size=64,
    shuffle='shuffle-first',  # only train shuffles
)

# Single dataset -> single loader (no tuple)
loader = make_basic_dataloader((X, y), batch_size=64, shuffle=True)
```

Shuffle modes: `True`, `False`, `'shuffle-first'` (default), `'shuffle-all'`, `'shuffle-train'`, or `list[bool]`.

### make_split_dataloader / make_kfold_dataloader

Use `make_split_dataloader` to split one dataset into multiple loaders (segment-level).
Use `make_kfold_dataloader` for segment-level K-fold (note: not file-level).

These are segment-level operations. For file-level splitting, use `train_test_pipeline` / `kfold_pipeline` instead.

## Utility functions

- `BasicXyDataset(data, target)` — wraps numpy/tensor into a PyTorch Dataset with optional transform
- `analyze_dataloader(loader)` — prints label distribution, balance check, sample count
- `count_dataloader_labels(loader)` — returns label distribution dict
- `random_split(data, proportion, seed)` — generic random split (returns actual data, not Subset)
- `summary_labels(y)` — prints label Counter from numpy array

## Read references only when needed

Each reference file covers a specific concern. Read only the one you need:

- `references/examples.md` — **5 complete code skeletons** covering the most common workflows:
  - Scenario A: Fixed train/val/test split from EEG files (recommended pattern)
  - Scenario B: File-level K-fold cross-validation
  - Scenario C: Building DataLoaders from in-memory (X, y) arrays
  - Scenario D: Using `fileloader_pipeline` for custom file formats (e.g. HDF5)
  - Scenario E: Using `BasicXyDataset` with custom transform/augmentation

- `references/api-details.md` — **Full parameter documentation** for every public function:
  - Complete function signatures with type annotations
  - Callback protocol signatures table (`fn_read`, `fn_segment`, `label`, etc.)
  - `split_size` semantics table (float/int/tuple variants and their return shapes)
  - `ShuffleMode` enum table (all 6 modes and their behavior)
  - `dtype` parameter usage (single dtype vs (data_dtype, target_dtype) tuple)
  - `all_files` grouping mechanism for stratified splits
  - Return value rules (single DataLoader vs tuple)

- `references/pitfalls.md` — **8 common mistakes** and a best-practice quick-reference table:
  - #1 Data leakage: segment-level vs file-level split (with wrong/correct code)
  - #2 `make_kfold_dataloader` vs `kfold_pipeline` confusion
  - #3 `fn_encode_label` default behavior (onehot with squeeze_2_class)
  - #4 `make_basic_dataloader` return type (single vs tuple)
  - #5 Shuffle misconfiguration (val set shuffling, ordering dependency)
  - #6 dtype mismatch (float32 target vs CrossEntropyLoss expecting long)
  - #7 `fn_pipeline` parameter modes (callable vs string preset, don't mix)
  - #8 `random_split` vs `torch.utils.data.random_split` (data slice vs Subset)
  - Best-practice combination lookup table (scenario -> pipeline -> split -> dataloader)

## Finish with these checks

1. Is the split at file level when data comes from per-subject files?
2. Is `shuffle='shuffle-first'` or equivalent used (train shuffles, val/test don't)?
3. Does the pipeline choice match the data format (continuous signal vs pre-segmented)?
4. Are `fn_encode_label` defaults acceptable, or does the task need custom encoding?
5. Is `random_seed` set for reproducibility?
6. For grouped files (`all_files=(group_a, group_b)`), is stratified splitting intended?
