Managing Artifacts - Infernet ML
Description: Infernet ML Documentation

Skip to content

# Artifact Management

Infernet ML 2.0 introduces a new utility class, `RitualArtifactManager`. It provides nice abstractions for managing artifacts across the Ritual ecosystem. Depending on the type of computation, various artifacts might be needed. In the context of AI/ML this could mean the machine learning models. In the context of ZKPs this could mean the circuit files. This class provides a way to easily manage these artifacts.

This documentation will go over how to define a new artifact class, and use it with the Artifact Manager. Inside Infernet ML, the `EZKLArtifact` is one such artifact class that is used to manage the ZKP circuit files.

## Usage

To define a new artifact type all you need to do is create a new Pydantic model. The Artifact Manager will handle the uploading/downloading of the artifact to/from different storage layers.

### Pydantic Model's Structure

Generally, artifacts will be composed of: 1. A collection of files, and 2. Metadata about the artifact.

The Pydantic model should have the following structure: * All the fields of type `Path` or `List[Path]` will be treated as artifact files and are uploaded to the storage layer. In addition, the `sha256` hash of the file is also calculated & stored in the manifest file. * All other fields are treated as metadata and are stored in the manifest file.

Here's an example of what a ZK artifact & its integration with the `RitualArtifactManager` might look like:

    class MyZkMlArtifact(BaseModel):
        circuit_file: Path
        model_file: Path
        version: str
        num_params: str

    my_artifact = RitualArtifactManager(
        artifact=MyZkMlArtifact(
            circuit_file=Path("path/to/circuit_file"),
            model_file=Path("path/to/model_file"),
            version="v1.0",
            num_params="1000"
        )
    )

### Uploading to a Storage Layer

With your instance of Artifact `RitualArtifactManager` , you can now easily upload/download your artifacts to Arweave or Huggingface:

HuggingFace & Arweave:

    # Upload artifact to HuggingFace
    my_artifact.to_huggingface_hub(
      repo_name="my-hf-username/my-repo-name",
      token=os.getenv("HF_TOKEN")
    )

    # Upload artifact to Arweave
    my_artifact.to_arweave(
      repo_name="my-repo-name",
      wallet_path="path/to/wallet.json"
    )

Using `RitualRepoId:`

More generally, you use the `to_repo()` method to upload to any storage layer. You will have to pass in a Generic Repo Id.

Note: Both `RitualArtifactManager` and `ModelManager` use the same notion of repository ids.

    # Upload artifact to any storage layer
    my_artifact.to_repo(
      repo_id="arweave/my-username/my-model",
      repo_type="arweave",
      wallet_path="path/to/wallet.json" # or hf_token if you're uploading to a huggingface repo
    )

### Downloading from a Storage Layer

You can use the same base class to download the artifact from the storage layer.

HuggingFace & Arweave:

    # Downloading from Huggingface Hub
    artifact = RitualArtifactManager.from_huggingface_hub(
      artifact_class=MyZkMlArtifact,
      repo_id="my-hf-username/my-repo-name",
      token=os.getenv("HF_TOKEN")
    )

    # Downloading from Arweave
    artifact = RitualArtifactManager.from_arweave(
      artifact_class=MyZkMlArtifact,
      repo_id="my-arweave-address/arweave-id"
    )

Using `RitualRepoId:`

    manager = RitualArtifactManager[EZKLArtifact].from_repo(
      artifact_class=MyZkMlArtifact,
      repo_id="arweave/my-username/my-model"
    )

Check out RitualArtifactManager for more options & configurations.