Repository API — Renku 0.15.2.dev42-g8ea0d1bb-dirty documentation

# Repository API

This API is built on top of Git and Git-LFS.

Renku repository management.

class `renku.core.management.` `LocalClient`(path=<function default_path>, renku_home='.renku', parent=None, commit_activity_cache=NOTHING, activity_index=None, remote_cache=NOTHING, migration_type=<MigrationType.ALL: 7>, database=None, external_storage_requested=True, *, data_dir='data')

  A low-level client for communicating with a local Renku repository.

  Method generated by attrs for class LocalClient.

## Datasets

Client for handling datasets.

class `renku.core.management.datasets.` `DatasetsApiMixin`

  Client for handling datasets.

  Method generated by attrs for class DatasetsApiMixin.

  `CACHE` = 'cache'

    Directory to cache transient data.

  `DATASETS` = 'datasets'

    Directory for storing dataset metadata in Renku.

  `DATASETS_PROVENANCE` = 'dataset.json'

    File for storing datasets’ provenance.

  `DATASET_IMAGES` = 'dataset_images'

    Directory for dataset images.

  `POINTERS` = 'pointers'

    Directory for storing external pointer files.

  `add_data_to_dataset`(dataset, urls, force=False, overwrite=False, sources=(), destination='', ref=None, external=False, extract=False, all_at_once=False, destination_names=None)

    Import the data into the data directory.

  `add_dataset_tag`(dataset, tag, description='', force=False)

    Adds a new tag to a dataset.

    Validates if the tag already exists and that the tag follows the same rules as docker tags. See https://docs.docker.com/engine/reference/commandline/tag/ for a documentation of docker tag syntax.

    Raises

      errors.ParameterError

  `clear_temporary_datasets_path`()

    Clear path to Renku dataset metadata directory.

  `create_dataset`(name=None, title=None, description=None, creators=None, keywords=None, images=None, safe_image_paths=None)

    Create a dataset.

  `dataset_commits`(dataset, max_results=None)

    Gets the newest commit for a dataset or its files.

    Commits are returned sorted from newest to oldest.

  property `datasets`

    Return mapping from path to dataset.

  `datasets_from_commit`(commit=None)

    Return datasets defined in a commit.

  property `datasets_provenance`

    Return dataset provenance if available.

  property `datasets_provenance_path`

    Path to store activity files.

  `get_dataset_path`(name)

    Get dataset path from name.

  `get_datasets_metadata_files`()

    Return a generator of datasets metadata files.

  `has_datasets_provenance`()

    Return true if dataset provenance exists.

  `has_external_files`()

    Return True if project has external files.

  `initialize_datasets_provenance`()

    Create empty dataset provenance file.

  `is_protected_path`(path)

    Checks if a path is a protected path.

  `is_using_temporary_datasets_path`()

    Return true if temporary datasets path is set.

  `load_dataset`(name=None, strict=False)

    Load dataset reference file.

  `load_dataset_from_path`(path, commit=None)

    Return a dataset from a given path.

  `load_dataset_from_provenance`(name, strict=False)

    Load latest dataset’s metadata from dataset provenance file.

  `move_files`(files, to_dataset, commit)

    Move files and their metadata from one or more datasets to a target dataset.

  `prepare_git_repo`(url, ref=None, gitlab_token=None)

    Clone and cache a Git repo.

  `remove_dataset_tags`(dataset, tags)

    Removes tags from a dataset.

  `remove_datasets_provenance_file`()

    Remove dataset provenance.

  static `remove_file`(filepath)

    Remove a file/symlink and its pointer file (for external files).

  property `renku_dataset_images_path`

    Return a `Path` instance of Renku dataset metadata folder.

  property `renku_datasets_path`

    Return a `Path` instance of Renku dataset metadata folder.

  property `renku_pointers_path`

    Return a `Path` instance of Renku pointer files folder.

  `set_dataset_images`(dataset, images, safe_image_paths=None)

    Set the images on a dataset.

  `set_temporary_datasets_path`(path)

    Set path to Renku dataset metadata directory.

  `update_dataset_git_files`(files, ref, delete=False)

    Update files and dataset metadata according to their remotes.

    Parameters

      - files – List of files to be updated

      - delete – Indicates whether to delete files or not

    Returns

      List of files that should be deleted

  `update_dataset_local_files`(records, delete=False)

    Update files metadata from the git history.

  `update_datasets_provenance`(dataset, remove=False)

    Update datasets provenance for a dataset.

  `update_external_files`(records)

    Update files linked to external storage.

  `with_dataset`(name=None, create=False, immutable=False)

    Yield an editable metadata object for a dataset.

  `with_dataset_provenance`(name=None, create=False)

    Yield a dataset’s metadata from dataset provenance.

## Repository

Client for handling a local repository.

class `renku.core.management.repository.` `PathMixin`(path=<function default_path>)

  Define a default path attribute.

  Method generated by attrs for class PathMixin.

class `renku.core.management.repository.` `RepositoryApiMixin`(renku_home='.renku', parent=None, commit_activity_cache=NOTHING, activity_index=None, remote_cache=NOTHING, migration_type=<MigrationType.ALL: 7>, database=None, *, data_dir='data')

  Client for handling a local repository.

  Method generated by attrs for class RepositoryApiMixin.

  `ACTIVITY_INDEX` = 'activity_index.yaml'

    Caches activities that generated a path.

  `DATABASE_PATH`: str = 'metadata'

    Directory for metadata storage.

  `DEPENDENCY_GRAPH` = 'dependency.json'

    File for storing dependency graph.

  `DOCKERFILE` = 'Dockerfile'

    Name of the Dockerfile in the repo.

  `LOCK_SUFFIX` = '.lock'

    Default suffix for Renku lock file.

  `METADATA` = 'metadata.yml'

    Default name of Renku config file.

  `PROVENANCE_GRAPH` = 'provenance.json'

    File for storing ProvenanceGraph.

  `WORKFLOW` = 'workflow'

    Directory for storing workflow in Renku.

  `activities_for_paths`(paths, file_commit=None, revision='HEAD')

    Get all activities involving a path.

  property `activity_index_path`

    Path to the activity filepath cache.

  `add_to_activity_index`(activity)

    Add an activity and it’s generations to the cache.

  property `cwl_prefix`

    Return a CWL prefix.

  `data_dir`

    Define a name of the folder for storing datasets.

  property `database_path`

    Path to the metadata storage directory.

  property `dependency_graph_path`

    Path to the dependency graph file.

  property `docker_path`

    Path to the Dockerfile.

  `find_previous_commit`(paths, revision='HEAD', return_first=False, full=False)

    Return a previous commit for a given path starting from `revision`.

    Parameters

      - revision – revision to start from, defaults to `HEAD`

      - return_first – show the first commit in the history

      - full – return full history

    Raises

      KeyError – if path is not present in the given commit

  `get_template_files`(template_path, metadata)

    Gets paths in a rendered renku template.

  `has_graph_files`()

    Return true if dependency or provenance graph exists.

  `import_from_template`(template_path, metadata, force=False)

    Render template files from a template directory.

  `init_repository`(force=False, user=None, initial_branch=None)

    Initialize an empty Renku repository.

  `initialize_graph`(database)

    Create empty graph files.

  `is_project_set`()

    Return if project is set for the client.

  `is_workflow`(path)

    Check if the path is a valid CWL file.

  property `latest_agent`

    Returns latest agent version used in the repository.

  property `lock`

    Create a Renku config lock.

  property `migration_type`

    Type of migration that is being executed on this client.

  `parent`

    Store a pointer to the parent repository.

  property `path_activity_cache`

    Cache of all activities and their generated paths.

  `process_and_store_run`(command_line_tool, name, description, keywords)

    Create Plan and Activity from CommandLineTool and store them.

  `process_commit`(commit=None, path=None)

    Build an `Activity`.

    Parameters

      - commit – Commit to process. (default: `HEAD`)

      - path – Process a specific CWL file.

  property `project`

    Return the Project instance.

  property `provenance_graph_path`

    Path to store activity files.

  property `remote`

    Return host, owner and name of the remote if it exists.

  `remove_graph_files`()

    Remove all graph files.

  `renku_home`

    Define a name of the Renku folder (default: `.renku`).

  property `renku_metadata_path`

    Return a `Path` instance of Renku metadata file.

  `renku_path`

    Store a `Path` instance of the Renku folder.

  `resolve_in_submodules`(commit, path)

    Resolve filename in submodules.

  `subclients`(parent_commit)

    Return mapping from submodule to client.

  property `submodules`

    Return list of submodules it belongs to.

  property `template_checksums`

    Return a `Path` instance to the template checksums file.

  `update_graphs`(activity, database)

    Update Dependency and Provenance graphs from a ProcessRun/WorkflowRun.

  `with_commit`(commit)

    Yield the state of the repo at a specific commit.

  `with_metadata`(read_only=False, name=None)

    Yield an editable metadata object.

  property `workflow_names`

    Return index of workflow names.

  property `workflow_path`

    Return a `Path` instance of the workflow folder.

`renku.core.management.repository.` `default_path`(path='.')

  Return default repository path.

`renku.core.management.repository.` `path_converter`(path)

  Converter for path in PathMixin.

## Git Internals

Wrap Git client.

class `renku.core.management.git.` `GitCore`

  Wrap Git client.

  Method generated by attrs for class GitCore.

  property `candidate_paths`

    Return all paths in the index and untracked files.

  `commit`(commit_only=None, commit_empty=True, raise_if_empty=False, commit_message=None, abbreviate_message=True, skip_dirty_checks=False)

    Automatic commit.

  property `dirty_paths`

    Get paths of dirty files in the repository.

  `ensure_clean`(ignore_std_streams=False)

    Make sure the repository is clean.

  `ensure_unstaged`(path)

    Ensure that path is not part of git staged files.

  `ensure_untracked`(path)

    Ensure that path is not part of git untracked files.

  `find_attr`(*paths)

    Return map with path and its attributes.

  `find_ignored_paths`(*paths)

    Return ignored paths matching `.gitignore` file.

  property `modified_paths`

    Return paths of modified files.

  `remove_unmodified`(paths, autocommit=True)

    Remove unmodified paths and return their names.

  `repo`

    Store an instance of the Git repository.

  `setup_credential_helper`()

    Setup git credential helper to `cache` if not set already.

  `worktree`(path=None, branch_name=None, commit=None, merge_args=('--ff-only'))

    Create new worktree.

`renku.core.management.git.` `finalize_commit`(client, project_metadata_path, diff_before, commit_only=None, commit_empty=True, raise_if_empty=False, commit_message=None, abbreviate_message=True)

  Commit modified/added paths.

`renku.core.management.git.` `get_mapped_std_streams`(lookup_paths, streams=('stdin', 'stdout', 'stderr'))

  Get a mapping of standard streams to given paths.

`renku.core.management.git.` `prepare_commit`(client, commit_only=None, skip_dirty_checks=False)

  Gather information about repo needed for committing later on.

Git utilities.

class `renku.core.models.git.` `GitURL`(href, pathname=None, protocol='ssh', hostname='localhost', username=None, password=None, port=None, owner=None, name=None, regex=None)

  Parser for common Git URLs.

  Method generated by attrs for class GitURL.

  property `image`

    Return image name.

  classmethod `parse`(href)

    Derive URI components.

class `renku.core.models.git.` `Range`(start, stop)

  Represent parsed Git revision as an interval.

  Method generated by attrs for class Range.

  classmethod `rev_parse`(git, revision)

    Parse revision string.

`renku.core.models.git.` `filter_repo_name`(repo_name)

  Remove the .git extension from the repo name.

`renku.core.models.git.` `get_user_info`(git)

  Get Git repository’s owner name and email.