# Dataset Registry Redesign: Per-Dataset Versioning

## Status

Draft

## Authors

- Koji Noshita

## Last Updated

2026-02-20

## Overview

### Problem Statement

The ktch dataset registry currently versions all remote datasets in lockstep with the ktch package version. Adding or updating a single dataset requires either a ktch release or the publication of a "phantom" version entry that has no corresponding code change. As the number of remote datasets grows (image, outline, and landmark variants for Passiflora leaves are all planned), this coupling becomes increasingly awkward.

### Goals

- Decouple dataset versioning from ktch release versioning
- Allow each dataset to evolve independently with its own version history
- Keep the user-facing API simple: `version=None` uses a sensible default; `version="2"` selects a specific dataset version
- Make the registry update workflow explicit and per-dataset rather than per-release
- Mirror the R2 storage structure in the local pooch cache for clarity

### Non-Goals

- Changing the loader function signatures beyond the `version` parameter semantics
- Versioning bundled datasets (outline_mosquito_wings, landmark_mosquito_wings, etc.) that are shipped inside the wheel
- Introducing a remote configuration endpoint for default version resolution
- Semver-style compatibility ranges for dataset versions

## Background

The current registry structure, introduced at ktch 0.7.0, maps ktch package versions to filename-to-hash dictionaries:

```python
versioned_registry = {
    "0.7.0": {
        "image_passiflora_leaves.zip": "d23b22252d0b...",
    },
}
```

When a user calls `load_image_passiflora_leaves()`, `_get_default_version()` derives a version string from the installed ktch package (e.g., `"0.7.1"`), and `_resolve_data_version()` falls back to the latest registered version that is less than or equal to the requested version (e.g., `"0.7.0"`). The download URL is formed as `releases/v{version}/{filename}`.

This design works for a single dataset but has several limitations:

- Updating `image_passiflora_leaves` to include more species (planned for the next release) requires either publishing the new zip under the next ktch version key or adding a new registry entry unrelated to any real code release.
- The fallback logic (`_resolve_data_version`) compares version tuples across the entire registry, meaning all datasets share the same version namespace. If `outline_passiflora_leaves` is registered at `"0.8.0"` and `image_passiflora_leaves` has only a `"0.7.0"` entry, the fallback logic works correctly by accident, but the mental model is confusing.
- Adding `outline_passiflora_leaves` and `landmark_passiflora_leaves` (both planned) would populate the same version-keyed dict, making each version entry a heterogeneous mixture of unrelated dataset updates.
- The `version` parameter exposed to users accepts a ktch version string (e.g., `"0.7.0"`), which is conceptually wrong: users are selecting a data version, not a code version.

## Proposed Solution

### Design Principles

Each dataset has its own independent version namespace. Versions are simple positive integers (`"1"`, `"2"`, `"3"`). The registry maps dataset name to version to filename hashes. A separate `default_versions` table records which dataset version each ktch release ships with. Users can override the default by passing `version="2"` to any loader function.

Integer versioning rather than semver is chosen because:

- The number of remote datasets in ktch is small and changes infrequently.
- The boundary between a "major" and "minor" data change is subjective for morphometric datasets (is adding 8 species "major"?).
- Integer version keys keep the user API clean: `version="2"` is unambiguous.
- Changes are documented in the dataset changelog; the version number itself does not need to encode change type.
- Projects that do use semver for datasets (e.g., TensorFlow Datasets) in practice rely only on the major component.

Data immutability: once a dataset version is published to R2, its contents never change. Updates always create a new version.

### _registry.py Structure

The current `versioned_registry` dict and `method_files_map` are replaced by three data structures and four helper functions.

```python
"""Dataset registry for ktch.datasets module."""

# Registry is auto-updated via:
#   uv run python scripts/update_registry.py <dataset_name> <version>
# See ref/dataset_registry_redesign.md for design rationale.

BASE_URL = "https://pub-c1d6dba6c94843f88f0fd096d19c0831.r2.dev"

# Per-dataset registry: {dataset_name: {version: {filename: sha256_hash}}}
dataset_registry = {
    "image_passiflora_leaves": {
        "1": {
            "image_passiflora_leaves.zip":
                "d23b22252d0baf3b60ef1986e6518f0550e2e1aa81f677b5991938ab65e2e45b",
        },
    },
}

# Default dataset versions for the current ktch release.
# Updated at ktch release time to pin the recommended dataset version.
default_versions = {
    "image_passiflora_leaves": "1",
}


def get_dataset_hash(dataset_name, version, filename):
    """Get the SHA256 hash for a specific dataset file."""
    ...


def get_dataset_url(dataset_name, version, filename):
    """Get the download URL for a specific dataset file.

    URL format: BASE_URL/datasets/{dataset_name}/v{version}/{filename}
    """
    return f"{BASE_URL}/datasets/{dataset_name}/v{version}/{filename}"


def get_default_version(dataset_name):
    """Get the default version for a dataset."""
    return default_versions[dataset_name]


def get_available_versions(dataset_name):
    """List all available versions for a dataset in sorted order.

    Note: versions must be sorted numerically (by int value), not
    lexicographically. Lexicographic sort of ["1", "2", "10"] yields
    ["1", "10", "2"], which is incorrect.
    """
    ...
```

The old `get_registry(version)` and `get_url(filename, version)` functions are removed. `method_files_map` is also removed; it was used only in `update_registry.py` and is not referenced from application code.

### R2 Storage Structure

Current layout:

```
releases/
└── v0.7.0/
    ├── manifest.json
    └── image_passiflora_leaves.zip
```

Proposed layout:

```
datasets/
├── image_passiflora_leaves/
│   ├── v1/
│   │   ├── manifest.json
│   │   └── image_passiflora_leaves.zip
│   └── v2/
│       ├── manifest.json
│       └── image_passiflora_leaves.zip
└── outline_passiflora_leaves/
    └── v1/
        ├── manifest.json
        └── outline_passiflora_leaves.zip
```

Path pattern: `datasets/{dataset_name}/v{version}/{filename}`

Each version directory is self-contained with its own `manifest.json`. The manifest format is unchanged: a flat JSON object mapping filename to lowercase hex SHA256 hash.

This layout groups all versions of a dataset together, making it straightforward to list, audit, or remove specific dataset histories without affecting others. It also makes the URL structure self-documenting.

### _base.py Changes

The functions `_get_default_version`, `_resolve_data_version`, and `_fetch_remote_data` are replaced.

`_get_default_version` is removed entirely. Its purpose was to derive a ktch version string from the installed package version; with independent dataset versioning this derivation is no longer needed.

`_resolve_data_version` is replaced by `_resolve_dataset_version`:

```python
def _resolve_dataset_version(dataset_name, version=None):
    """Resolve the dataset version to use.

    Parameters
    ----------
    dataset_name : str
        The dataset name (e.g., "image_passiflora_leaves").
    version : str or None
        Explicit version string (e.g., "2"). If None, uses the default
        from default_versions.

    Returns
    -------
    str
        Resolved version string.

    Raises
    ------
    ValueError
        If the dataset is not registered or the version does not exist.
    """
    if version is None:
        return get_default_version(dataset_name)

    if dataset_name not in dataset_registry:
        raise ValueError(f"Unknown dataset: '{dataset_name}'")

    if version not in dataset_registry[dataset_name]:
        available = ", ".join(
            sorted(dataset_registry[dataset_name].keys(), key=int)
        )
        raise ValueError(
            f"Version '{version}' not found for dataset '{dataset_name}'. "
            f"Available versions: {available}"
        )

    return version
```

The fallback logic in the old `_resolve_data_version` (find the latest registered version that is <= the requested version) is removed. With independent dataset versioning there is no meaningful ordering relationship to exploit across the global registry, and the concept of "compatibility" between a ktch version and a data version no longer applies.

`_fetch_remote_data` is replaced by `_fetch_remote_dataset`:

```python
def _fetch_remote_dataset(dataset_name, version, filename):
    """Fetch a remote dataset file using pooch.

    Parameters
    ----------
    dataset_name : str
        Dataset name (e.g., "image_passiflora_leaves").
    version : str
        Dataset version string (e.g., "1").
    filename : str
        Filename within the dataset (e.g., "image_passiflora_leaves.zip").

    Returns
    -------
    str
        Local path to the downloaded file.
    """
    if pooch is None:
        raise ImportError(
            "Missing optional dependency 'pooch' for downloading remote datasets. "
            "Please install it using: pip install ktch[data]"
        )

    known_hash = get_dataset_hash(dataset_name, version, filename)
    url = get_dataset_url(dataset_name, version, filename)
    cache_path = pooch.os_cache("ktch-data") / dataset_name / f"v{version}"

    return pooch.retrieve(
        url=url,
        known_hash=f"sha256:{known_hash}",
        path=cache_path,
        fname=filename,
    )
```

### Loader Function Changes

The `load_image_passiflora_leaves` function changes minimally. The version resolution call is updated, and the `_fetch_remote_data` call is replaced with `_fetch_remote_dataset`. The rest of the function body (zip extraction, metadata loading, image reading) is unchanged.

```python
# Current
def load_image_passiflora_leaves(
    *, return_paths=False, as_frame=False, version=None
) -> Bunch:
    if version is None:
        version = _get_default_version()
    version = _resolve_data_version(version)
    archive_path = _fetch_remote_data("image_passiflora_leaves.zip", version)
    ...

# Proposed
def load_image_passiflora_leaves(
    *, return_paths=False, as_frame=False, version=None
) -> Bunch:
    dataset_name = "image_passiflora_leaves"
    version = _resolve_dataset_version(dataset_name, version)
    archive_path = _fetch_remote_dataset(
        dataset_name, version, "image_passiflora_leaves.zip"
    )
    ...
```

The `version` parameter semantics change for users: previously it accepted a ktch version string such as `"0.7.0"`; after this change it accepts a dataset version string such as `"1"` or `"2"`. The default (`None`) continues to select the recommended version for the installed ktch release.

### Pooch Cache Layout

Current:

```
~/.cache/ktch-data/0.7.0/image_passiflora_leaves.zip
```

Proposed:

```
~/.cache/ktch-data/image_passiflora_leaves/v1/image_passiflora_leaves.zip
```

The cache layout mirrors the R2 directory structure. This eliminates ambiguity when multiple datasets and versions coexist in the cache, and makes manual cache inspection straightforward.

Existing cache entries under the old layout are not automatically cleaned up. Users who upgrade ktch will re-download the data to the new cache path on first use. The old path can be deleted manually or by a future cache-management utility.

### update_registry.py Changes

The script is updated to accept a dataset name and version rather than a ktch release version.

Current usage:

```
uv run python scripts/update_registry.py 0.8.0
uv run python scripts/update_registry.py --dry-run 0.8.0
```

Proposed usage:

```
uv run python scripts/update_registry.py image_passiflora_leaves 2
uv run python scripts/update_registry.py --dry-run image_passiflora_leaves 2
```

The script fetches `datasets/{dataset_name}/v{version}/manifest.json` from R2, validates the hashes, and updates the corresponding entry in `dataset_registry` in `_registry.py`. It does not modify `default_versions`; that is updated manually at release time.

The internal functions `read_current_registry`, `build_method_files_map`, `render_registry`, and `validate_manifest` are updated to reflect the new data structure. `build_method_files_map` is removed since `method_files_map` no longer exists in `_registry.py`.

## Alternatives Considered

### Alternative 1: Keep lockstep versioning, add a per-dataset override

Add a second registry structure alongside `versioned_registry` that maps dataset-name to a pinned version, used only when a dataset has been independently updated. The fallback logic in `_resolve_data_version` would check the per-dataset override first.

Pros: smaller diff; preserves backward compatibility of the `version` parameter.

Cons: two parallel structures with overlapping semantics; the mental model becomes harder to explain. The fallback logic grows more complex. The fundamental coupling between ktch releases and data versions is not resolved.

Why not chosen: the complexity cost outweighs the migration cost of the clean redesign.

### Alternative 2: Semver for dataset versions

Use three-part version strings (`"1.0.0"`, `"1.1.0"`, `"2.0.0"`) for dataset versions, with the major component indicating breaking changes.

Pros: familiar convention; encodes change type.

Cons: for infrequently updated morphometric datasets, the minor/patch distinction adds overhead without benefit. The definition of "breaking" for a dataset (new specimens added? column renamed?) is ambiguous and context-dependent. `version="1.0.0"` is more verbose than `version="1"` in the user API with no added clarity.

Why not chosen: integer versioning is simpler and sufficient for the expected update frequency.

### Alternative 3: Remote default version configuration

Store `default_versions` in a JSON file on R2. The loader fetches the remote config on first call to determine which dataset version to use, allowing default version updates without a ktch release.

Pros: dataset curators can push a "recommended" update without requiring a ktch release.

Cons: introduces a network dependency on every first call; complicates error handling (what if the config fetch fails?); makes behavior non-deterministic relative to the installed package version.

Why not chosen: the simplicity of a static `default_versions` table in the package outweighs the flexibility of remote configuration. Dataset updates that require ktch code changes (new loader logic, new return fields) already require a release; dataset updates that are purely additive (more specimens, same format) can be released on a fast cycle since ktch releases are inexpensive.

## Security Considerations

Hash verification via pooch is unchanged. Every downloaded file is verified against a SHA256 hash stored in `_registry.py`, which is distributed as part of the ktch package. A compromised R2 bucket cannot deliver malicious data without also compromising the ktch package release.

The zip path traversal guard (`_safe_extractall`) in `_base.py` is unaffected by this redesign and remains in place.

## Operational Considerations

### Publishing a New Dataset Version

1. Prepare the dataset archive and compute its SHA256 hash.
2. Upload to R2 at `datasets/{dataset_name}/v{version}/{filename}`.
3. Upload a `manifest.json` to `datasets/{dataset_name}/v{version}/manifest.json` with content `{"filename": "sha256_hash"}`.
4. Run `uv run python scripts/update_registry.py {dataset_name} {version}` to update `_registry.py`.
5. If this version should become the new default, update `default_versions` in `_registry.py` manually.
6. Commit and release ktch if `default_versions` changed or if the dataset update accompanies a code change.

### Old R2 Paths

The existing `releases/v0.7.0/` path on R2 remains accessible to users running ktch 0.7.x. It should not be deleted until ktch 0.7.x is outside the supported range. A deprecation period of at least one major release cycle is recommended.

### Monitoring

No changes to monitoring are required. The download path changes (new URL pattern, new cache path), but pooch's retry and hash-verification behavior is unchanged.

## Migration Plan

### Phase 1: Data Migration on R2

1. Copy `releases/v0.7.0/image_passiflora_leaves.zip` to `datasets/image_passiflora_leaves/v1/image_passiflora_leaves.zip`.
2. Create `datasets/image_passiflora_leaves/v1/manifest.json` with the SHA256 hash of the zip.
3. Keep `releases/v0.7.0/` intact for users on ktch 0.7.x.

### Phase 2: Code Changes

The following files require changes. All changes are in a single commit or pull request.

`ktch/datasets/_registry.py`:
- Replace `versioned_registry` and `method_files_map` with `dataset_registry` and `default_versions`.
- Replace `get_registry` and `get_url` with `get_dataset_hash`, `get_dataset_url`, `get_default_version`, and `get_available_versions`.

`ktch/datasets/_base.py`:
- Remove `_get_default_version` and `_resolve_data_version`.
- Add `_resolve_dataset_version` and `_fetch_remote_dataset`.
- Update `load_image_passiflora_leaves` to use the new functions.
- Update imports from `_registry`.

`scripts/update_registry.py`:
- Update argument parsing (dataset_name + version instead of version alone).
- Update `fetch_manifest` to use the new R2 path pattern.
- Update registry reading and rendering to handle `dataset_registry` and `default_versions`.
- Remove `build_method_files_map`.

### Phase 3: Test Updates

`ktch/datasets/tests/test_image_passiflora_leaves.py`:
- Update `TestRegistry` to test `dataset_registry`, `default_versions`, and the new registry functions.
- Update `TestVersionDetection` to remove tests for `_get_default_version` and `_resolve_data_version`.
- Add tests for `_resolve_dataset_version`.
- Update URL assertions to use the new path pattern (`datasets/image_passiflora_leaves/v1/`).

`scripts/tests/test_update_registry.py`:
- Update `TestRenderRegistry` for the new render function signatures.
- Remove `TestBuildMethodFilesMap`.
- Update `TestValidateManifest` (unchanged in behavior, but the calling context changes).

### Backward Compatibility

Users on ktch 0.7.x are unaffected; they use the old code and old R2 paths.

Users who upgrade to the next ktch release and have called `load_image_passiflora_leaves(version="0.7.0")` explicitly will receive a `ValueError` because `"0.7.0"` is not a valid integer dataset version. This is a breaking change in the `version` parameter semantics. Since explicit version usage is rare (the parameter was introduced as an escape hatch and the default `None` covers the vast majority of calls), the impact is expected to be minimal. The change should be documented in the release notes and changelog.

The local cache at `~/.cache/ktch-data/0.7.0/` is not cleaned up automatically. Users will accumulate stale cache entries until they clear the cache directory manually. A future `ktch.datasets.clear_cache()` utility could address this, but it is out of scope for this redesign.

## Open Questions

- [ ] Should the old `releases/v0.7.0/` path on R2 be kept permanently or removed after a deprecation period? If removed, what is the minimum deprecation window?
- [ ] Should `default_versions` be updatable without a ktch release (e.g., via a remote config)? Current recommendation: no; keep it static and update at release time only.
- [ ] Do `outline_passiflora_leaves` and `landmark_passiflora_leaves` need remote versioning in the future? Currently planned as bundled, but if data size grows they could migrate to remote delivery. This design supports that transition naturally by adding entries to `dataset_registry`.

## References

- Current registry implementation: `ktch/datasets/_registry.py`
- Current base loader: `ktch/datasets/_base.py`
- Registry update script: `scripts/update_registry.py`
- Passiflora datasets design (planned datasets): `ref/passiflora_datasets_design.md`
