# Resource Interface (v2)

## Overview

Resources represent steps in a dataset preparation pipeline. They form a
directed acyclic graph (DAG) where each resource can depend on other resources.

Key concepts:

- **Two-path system**: resources write to `transient_path` during download,
  then the framework moves data to `path` and marks the resource as COMPLETE.
- **Three states**: NONE, PARTIAL, COMPLETE (persisted in `.state.json`)
- **Transient resources**: intermediate resources that can be deleted after all
  dependents are COMPLETE (eager cleanup)
- **`can_recover` property**: subclasses override to preserve PARTIAL data on error

## Modern API: Class-based datasets (preferred)

```python
from datamaestro.definitions import dataset
from datamaestro.download.single import FileDownloader

@dataset(url="http://yann.lecun.com/exdb/mnist/")
class ProcessedMNIST(ImageClassification):
    """The MNIST database of handwritten digits."""

    # Resources are class attributes — no decorators needed
    TRAIN_IMAGES = FileDownloader(
        "train_images.idx",
        "http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz",
        transient=True,
    )
    TRAIN_IMAGES_NP = NumpyTensorFile.from_idx(TRAIN_IMAGES)

    TRAIN_LABELS = FileDownloader(
        "train_labels.idx",
        "http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz",
        transient=True,
    )
    TRAIN_LABELS_NP = NumpyTensorFile.from_idx(TRAIN_LABELS)

    TEST_IMAGES = FileDownloader(
        "test_images.idx",
        "http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz",
        transient=True,
    )
    TEST_IMAGES_NP = NumpyTensorFile.from_idx(TEST_IMAGES)

    TEST_LABELS = FileDownloader(
        "test_labels.idx",
        "http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz",
        transient=True,
    )
    TEST_LABELS_NP = NumpyTensorFile.from_idx(TEST_LABELS)

    @classmethod
    def __create_dataset__(cls, dataset: AbstractDataset):
        return cls.C(
            train=LabelledImages(
                images=NumpyTensorFile(path=cls.TRAIN_IMAGES_NP.path),
                labels=NumpyTensorFile(path=cls.TRAIN_LABELS_NP.path),
            ),
            test=LabelledImages(
                images=NumpyTensorFile(path=cls.TEST_IMAGES_NP.path),
                labels=NumpyTensorFile(path=cls.TEST_LABELS_NP.path),
            ),
        )
```

Advantages:

1. **Explicit pipeline** — dependencies between resources are visible
2. **Transient intermediaries** — intermediate files can be deleted after processing
3. **No varname** — resource names are auto-detected from class attribute names
4. **Two-path safety** — incomplete downloads never appear at the final path

## Resource hierarchy

```
Resource (ABC)
├── FileResource      — produces a single file
├── FolderResource    — produces a directory
├── ValueResource     — produces an in-memory value (no files)
├── reference         — references another dataset
└── Download          — (deprecated alias for Resource)
```

### `ResourceState`

```python
class ResourceState(str, Enum):
    NONE = "none"          # Not started
    PARTIAL = "partial"    # Started but incomplete
    COMPLETE = "complete"  # Fully available
```

### `Resource` base class

| Property / Method | Description |
|---|---|
| `name: str` | Resource name (auto-set from class attribute name) |
| `dataset` | Back-reference to the owning `AbstractDataset` |
| `transient: bool` | Whether data can be deleted after dependents complete |
| `can_recover: bool` | Property. If True, PARTIAL data is preserved on error |
| `dependencies` | List of resources that must be COMPLETE first |
| `dependents` | Computed inverse of dependencies |
| `path: Path` | Final storage path (after COMPLETE) |
| `transient_path: Path` | Temp path where `download()` writes |
| `state: ResourceState` | Current state (from `.state.json` metadata file) |
| `download(force)` | Abstract. Execute download/processing step |
| `prepare()` | Abstract. Return value for dataset construction |
| `cleanup()` | Remove data from disk, set state to NONE |
| `has_files() -> bool` | Whether this resource produces files on disk |
| `bind(name, dataset)` | Bind to a dataset (called by framework) |
| `stream() -> IO | None` | (FileResource only) Return byte stream or None |

### `FileResource`

Base for resources that produce a single file. Subclasses implement
`_download(destination: Path)`.

```python
class MyFileResource(FileResource):
    def __init__(self, filename, url, **kw):
        super().__init__(filename, **kw)
        self.url = url

    def _download(self, destination: Path):
        # Write to destination (which is self.transient_path)
        ...
```

### `FolderResource`

Base for resources that produce a directory. Subclasses implement
`_download(destination: Path)`.

### `ValueResource`

Base for resources that produce in-memory values (no files on disk).
`has_files()` returns False.

## Custom resource handlers (modern)

```python
from datamaestro.download import FileResource

class MyProcessor(FileResource):
    """Process a source file into a numpy array."""

    @property
    def can_recover(self) -> bool:
        return False  # or True for resumable downloads

    def __init__(self, filename, source, **kw):
        super().__init__(filename, **kw)
        self._dependencies = [source]

    def _download(self, destination):
        # Read from dependency, write to destination
        source_path = self.dependencies[0].path
        data = load(source_path)
        save(process(data), destination)

    @classmethod
    def from_source(cls, source):
        return cls("processed.npy", source)

# Factory alias
my_processor = MyProcessor.from_source
```

## Built-in resource types

| Class | Module | Factory alias | Base |
|---|---|---|---|
| `FileDownloader` | `download.single` | `filedownloader` | `FileResource` |
| `ConcatDownloader` | `download.single` | `concatdownload` | `FileResource` |
| `ZipDownloader` | `download.archive` | `zipdownloader` | `FolderResource` |
| `TarDownloader` | `download.archive` | `tardownloader` | `FolderResource` |
| `HFDownloader` | `download.huggingface` | `hf_download` | `ValueResource` |
| `custom_download` | `download.custom` | — | `Resource` |
| `links` | `download.links` | — | `Resource` |
| `linkfolder` | `download.links` | — | `Resource` |
| `linkfile` | `download.links` | — | `Resource` |
| `reference` | `download` | — | `Resource` |

## Two-path download flow

The framework (in `AbstractDataset.download()`) orchestrates:

```
1. Topological sort resources by dependencies
2. For each resource:
   a. COMPLETE and not force → skip
   b. PARTIAL and not can_recover → delete transient_path, set NONE
   c. Call resource.download(force)
      → Resource writes to transient_path
   d. On success: move transient_path → path, set COMPLETE
   e. On failure: if can_recover → set PARTIAL, else delete → NONE
   f. Eager cleanup: for each transient dependency with all
      dependents COMPLETE → cleanup
```

## State metadata file

Location: `<dataset.datapath>/.downloads/.state.json`

```json
{
  "version": 1,
  "resources": {
    "TRAIN_IMAGES": {"state": "complete"},
    "TRAIN_LABELS": {"state": "partial"}
  }
}
```

---

## Deprecated: decorator-based datasets

> **Deprecated.** The decorator-based API still works but emits deprecation
> warnings. Migrate to the class-based approach above.

```python
# DEPRECATED — use class-based approach instead
@filedownloader("train_images.idx", "http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz")
@filedownloader("train_labels.idx", "http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz")
@filedownloader("test_images.idx", "http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz")
@filedownloader("test_labels.idx", "http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz")
@dataset(
    ImageClassification,
    url="http://yann.lecun.com/exdb/mnist/",
)
def MNIST(train_images, train_labels, test_images, test_labels):
    """The MNIST database"""
    return {
        "train": LabelledImages(
            images=IDX(path=train_images),
            labels=IDX(path=train_labels)
        ),
        "test": LabelledImages(
            images=IDX(path=test_images),
            labels=IDX(path=test_labels)
        ),
    }
```

### Deprecated names

| Deprecated | Replacement |
|---|---|
| `Download` (base class) | `Resource` |
| `hasfiles()` | `has_files()` |
| `Resource.definition` | `Resource.dataset` |
| `Resource.varname` | `Resource.name` |
| `@filedownloader(...)` (decorator) | `FileDownloader(...)` (class attr) |
| `SingleDownload` | `FileDownloader` |

### Deprecated custom handler pattern

```python
# DEPRECATED
class MyDownload(Download):
    def __init__(self, varname, custom_param):
        super().__init__(varname)
        self.custom_param = custom_param

    def prepare(self):
        return self._download_and_process()

    def download(self, force=False):
        if force or not self._is_cached():
            self._do_download()

    def hasfiles(self) -> bool:
        return True

def mydownloader(varname, custom_param):
    def decorator(dataset):
        download = MyDownload(varname, custom_param)
        download.register(dataset)
        return dataset
    return decorator
```

Modern equivalent:

```python
class MyDownload(FileResource):
    def __init__(self, filename, custom_param, **kw):
        super().__init__(filename, **kw)
        self.custom_param = custom_param

    def _download(self, destination):
        # Write output to destination (self.transient_path)
        self._do_download(destination)

mydownloader = MyDownload.apply
```
