Low-level API — Renku 0.11.2.dev14+dirty documentation

Join us on GitHub

# Low-level API

This API is built on top of Git and Git-LFS.

Renku repository management.

class `renku.core.management.` `LocalClient`(path=<function default_path>, renku_home='.renku', parent=None, external_storage_requested=True, *, data_dir='data')

  A low-level client for communicating with a local Renku repository.

## Datasets

Client for handling datasets.

class `renku.core.management.datasets.` `DatasetsApiMixin`

  Client for handling datasets.

  `CACHE` = 'cache'

    Directory to cache transient data.

  `DATASETS` = 'datasets'

    Directory for storing dataset metadata in Renku.

  `POINTERS` = 'pointers'

    Directory for storing external pointer files.

  `add_data_to_dataset`(dataset, urls, force=False, overwrite=False, sources=(), destination='', ref=None, external=False, extract=False, all_at_once=False, destination_names=None, progress=None)

    Import the data into the data directory.

  `add_dataset_tag`(dataset, tag, description='', force=False)

    Adds a new tag to a dataset.

    Validates if the tag already exists and that the tag follows the same rules as docker tags. See https://docs.docker.com/engine/reference/commandline/tag/ for a documentation of docker tag syntax.

    Raises

      errors.ParameterError

  `create_dataset`(name=None, title=None, description=None, creators=None, keywords=None)

    Create a dataset.

  `dataset_commits`(dataset, max_results=None)

    Gets the newest commit for a dataset or its files.

    Commits are returned sorted from newest to oldest.

  property `datasets`

    Return mapping from path to dataset.

  `datasets_from_commit`(commit=None)

    Return datasets defined in a commit.

  `get_dataset_path`(name)

    Get dataset path from name.

  `has_external_files`()

    Return True if project has external files.

  `load_dataset`(name=None)

    Load dataset reference file.

  `load_dataset_from_path`(path, commit=None)

    Return a dataset from a given path.

  `prepare_git_repo`(url, ref=None)

    Clone and cache a Git repo.

  `remove_dataset_tags`(dataset, tags)

    Removes tags from a dataset.

  `remove_file`(filepath)

    Remove a file/symlink and its pointer file (for external files).

  property `renku_datasets_path`

    Return a `Path` instance of Renku dataset metadata folder.

  property `renku_pointers_path`

    Return a `Path` instance of Renku pointer files folder.

  `update_dataset_files`(files, ref, delete=False)

    Update files and dataset metadata according to their remotes.

    Parameters

      - files – List of files to be updated

      - delete – Indicates whether to delete files or not

    Returns

      List of files that should be deleted

  `update_external_files`(records)

    Update files linked to external storage.

  `with_dataset`(name=None, create=False)

    Yield an editable metadata object for a dataset.

class `renku.core.management.datasets.` `DownloadProgressCallback`(description, total_size)

  Interface to report various stages of a download.

  Default initializer.

  `finalize`()

    Called once when the download is finished.

  `update`(size)

    Update the status.

## Repository

Client for handling a local repository.

class `renku.core.management.repository.` `PathMixin`(path=<function default_path>)

  Define a default path attribute.

class `renku.core.management.repository.` `RepositoryApiMixin`(renku_home='.renku', parent=None, *, data_dir='data')

  Client for handling a local repository.

  `ACTIVITY_INDEX` = 'activity_index.yaml'

    Caches activities that generated a path.

  `LOCK_SUFFIX` = '.lock'

    Default suffix for Renku lock file.

  `METADATA` = 'metadata.yml'

    Default name of Renku config file.

  `WORKFLOW` = 'workflow'

    Directory for storing workflow in Renku.

  `activities_for_paths`(paths, file_commit=None, revision='HEAD')

    Get all activities involving a path.

  property `activity_index_path`

    Path to the activity filepath cache.

  `add_to_activity_index`(activity)

    Add an activity and it’s generations to the cache.

  property `cwl_prefix`

    Return a CWL prefix.

  `data_dir` = None

    Define a name of the folder for storing datasets.

  `find_previous_commit`(paths, revision='HEAD', return_first=False, full=False)

    Return a previous commit for a given path starting from `revision`.

    Parameters

      - revision – revision to start from, defaults to `HEAD`

      - return_first – show the first commit in the history

      - full – return full history

    Raises

      KeyError – if path is not present in the given commit

  `import_from_template`(template_path, metadata, force=False)

    Render template files from a template directory.

  `init_repository`(force=False, user=None)

    Initialize an empty Renku repository.

  `is_project_set`()

    Return if project is set for the client.

  `is_workflow`(path)

    Check if the path is a valid CWL file.

  property `latest_agent`

    Returns latest agent version used in the repository.

  property `lock`

    Create a Renku config lock.

  `parent` = None

    Store a pointer to the parent repository.

  property `path_activity_cache`

    Cache of all activities and their generated paths.

  `process_commit`(commit=None, path=None)

    Build an `Activity`.

    Parameters

      - commit – Commit to process. (default: `HEAD`)

      - path – Process a specific CWL file.

  property `project`

    Return the Project instance.

  property `remote`

    Return host, owner and name of the remote if it exists.

  `renku_home` = None

    Define a name of the Renku folder (default: `.renku`).

  property `renku_metadata_path`

    Return a `Path` instance of Renku metadata file.

  `renku_path` = None

    Store a `Path` instance of the Renku folder.

  `resolve_in_submodules`(commit, path)

    Resolve filename in submodules.

  `subclients`(parent_commit)

    Return mapping from submodule to client.

  property `submodules`

    Return list of submodules it belongs to.

  `with_commit`(commit)

    Yield the state of the repo at a specific commit.

  `with_metadata`(read_only=False, name=None)

    Yield an editable metadata object.

  `with_workflow_storage`()

    Yield a workflow storage.

  property `workflow_names`

    Return index of workflow names.

  property `workflow_path`

    Return a `Path` instance of the workflow folder.

`renku.core.management.repository.` `default_path`()

  Return default repository path.

`renku.core.management.repository.` `path_converter`(path)

  Converter for path in PathMixin.

## Git Internals

Wrap Git client.

class `renku.core.management.git.` `GitCore`

  Wrap Git client.

  property `candidate_paths`

    Return all paths in the index and untracked files.

  `commit`(commit_only=None, commit_empty=True, raise_if_empty=False, commit_message=None)

    Automatic commit.

  property `dirty_paths`

    Get paths of dirty files in the repository.

  `ensure_clean`(ignore_std_streams=False)

    Make sure the repository is clean.

  `ensure_unstaged`(path)

    Ensure that path is not part of git staged files.

  `ensure_untracked`(path)

    Ensure that path is not part of git untracked files.

  `find_attr`(*paths)

    Return map with path and its attributes.

  `find_ignored_paths`(*paths)

    Return ignored paths matching `.gitignore` file.

  property `modified_paths`

    Return paths of modified files.

  `remove_unmodified`(paths, autocommit=True)

    Remove unmodified paths and return their names.

  `repo` = None

    Store an instance of the Git repository.

  `setup_credential_helper`()

    Setup git credential helper to `cache` if not set already.

  `transaction`(clean=True, commit=True, commit_empty=True, commit_message=None, commit_only=None, ignore_std_streams=False, raise_if_empty=False)

    Perform Git checks and operations.

  `worktree`(path=None, branch_name=None, commit=None, merge_args=('--ff-only', ))

    Create new worktree.

`renku.core.management.git.` `get_mapped_std_streams`(lookup_paths, streams=('stdin', 'stdout', 'stderr'))

  Get a mapping of standard streams to given paths.

Git utilities.

class `renku.core.models.git.` `GitURL`(href, pathname=None, protocol='ssh', hostname='localhost', username=None, password=None, port=None, owner=None, name=None, regex=None)

  Parser for common Git URLs.

  property `image`

    Return image name.

  classmethod `parse`(href)

    Derive basic informations.

class `renku.core.models.git.` `Range`(start, stop)

  Represent parsed Git revision as an interval.

  classmethod `rev_parse`(git, revision)

    Parse revision string.

`renku.core.models.git.` `filter_repo_name`(repo_name)

  Remove the .git extension from the repo name.

`renku.core.models.git.` `get_user_info`(git)

  Get Git repository’s owner name and email.