rse.main.database package — The Research Software Encyclopedia 11 documentation

# rse.main.database package

## Submodules

## rse.main.database.base module

Copyright (C) 2020 Vanessa Sochat.

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

class rse.main.database.base.Database

  Bases: `object`

  A rse database holds common functions to represent repos, and metadata.

  add(uid)

    Add a new software repository to the database.

  clear()

    clear (delete) all software repositories.

  database = 'notimplemented'
  delete_parser(name)

    delete all repos for a parser based on the parser name.

  delete_repo(uid)

    delete a repo based on a specific identifier.

  exists()

    determine if a software repository exists in the database.

  get()

    get a software repository from the database.

  get_or_create(uid)

    Determine if a repo exists.

  list_repos(name=None)

    list software repositories, either under a particular parser name or just under all parsers.

  search(query)

    search is only available to non-filesystem databases

  update(uid)

    update a software repository.

## rse.main.database.filesystem module

Copyright (C) 2020 Vanessa Sochat.

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

class rse.main.database.filesystem.FileSystemDatabase(config_dir, config=None, **kwargs)

  Bases: `rse.main.database.base.Database`

  A FileSystemDatabase writes raw json to files to a database.

  add(uid)

    Add a new software repository to the database.

  clear()

    clear (delete) all software repositories.

  create_database(config_dir)

    Create the database. The parent folder must exist.

  database = 'filesystem'
  delete_parser(name)

    delete all repos for a parser, based on executor’s name (str).

  delete_repo(uid)

    delete a repo based on a specific identifier.

  exists(uid)

    Determine if a repo exists.

  get(uid=None, exact=False)

    Get a software repo based on a uid. If exact is not needed, we can search for a match based on the partial uid. If exact is False, and a uid is not provided, get the last repository created.

  get_or_create(uid)

    Determine if a repo exists.

  iter_parsers(fullpath=False)

    list executors based on the subfolders in the base database folder.

  label(repo, key, value, force=False)

    Update a repository with a specific key/value pair.

  list_repos(name=None)

    list software repositories, either under a particular parser name or just under all parsers. This returns repos in rows to be printed (or otherwise parsed).

  search(query, taxonomy=None, criteria=None)

    A filesystem search can only support returning results with filenames. For taxonomy and criteria items, we load them and search. We organize results based on the query, taxonomy, and criteria The results returned are separate (e.g., a single repo can appear in more than one list).

  update(repo, rewrite=False)

    Update a repository by retrieving metadata, and then calling update on the software repository to save it.

class rse.main.database.filesystem.SoftwareRepository(parser, data_base, exists=False)

  Bases: `object`

  A software repository is a filesystem representation of a repo. It can take a uid, determine if the repo exists, and then interact with the metadata for it. If the repo is instantiated without a unique id it is assumed to not exist yet, otherwise it must already exist.

  property avatar
  create(should_exist=False)

    create the filename if it doesn’t exist, otherwise if it should (and does not) exit on error.

  property description
  export()

    wrapper to expose the executor.export function

  property filename
  get_criteria()

    Get loaded criteria

  get_taxonomy()

    Get loaded taxonomy

  has_criteria_annotation(uid, username)

    Determine if a repository has been annotated by a user.

  has_taxonomy_annotation(username)

    Determine if a repository has been annotated by a user.

  load()

    Given a software uid, load data from filename.

  load_criteria()

    Given a repository directory, load criteria files if they exist

  load_taxonomy()

    Given a repository directory, load taxonomy annotations if they exist The taxonomy.tsv file should be a tab separated file with: username category-unique-id. This means that we keep a record of who has categorized what, and load this information into the taxonomy dictionary (organized by the category-unique-id which then has a total count and list of users).

  property parser_dir
  save()

    Save a json object (metadata.json) for the software repository.

  save_criteria()

    Save criteria to file. Each file is named based on the criteria id, and is a tab separated file that includes the username and response.

  save_taxonomy()

    Save taxonomy to file. Each file is named taxonomy.tsv, and is a tab separated file that includes the username and response.

  summary()
  update(updates=None)

    Update a data file. This means reading, updating, and writing.

  update_criteria(uid, username, response)

    Update a criteria, meaning adding a True/False answer to the unique id for the user. We are currently assuming that criteria have yes/no responses, and True == yes, False == no.

  property url

## rse.main.database.models module

Copyright (C) 2020 Vanessa Sochat.

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

class rse.main.database.models.SoftwareRepository(uid=None, parser=None, data={})

  Bases: `sqlalchemy.orm.decl_api.Base`

  A software repository.

  property avatar
  criteria
  data
  property description
  export()

    Export removes the outer wrapper, and just returns the data

  get_criteria()

    load criteria into a dictionary

  get_taxonomy()

    load taxonomy into a dictionary

  has_criteria_annotation(uid, username)

    Determine if a repository has been annotated by a user.

  has_taxonomy_annotation(username)

    Determine if a repository has been annotated by a user.

  load()

    loading a software repository means exporting as json

  parser_name
  summary()
  taxonomy
  timestamp
  uid
  update_criteria(uid, username, response)

    Given a username and unique id update criteria

  update_taxonomy(username, uids)

    Given a username and unique id update taxonomy items

  property url

## rse.main.database.relational module

Copyright (C) 2020 Vanessa Sochat.

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

class rse.main.database.relational.RelationalDatabase(config_dir, config=None, **kwargs)

  Bases: `rse.main.database.base.Database`

  A RelationalDatabase is a more robust relational datbase (to sqlite). Since the global database property can be any of postgresql, mysql+pysq;, it is defined on init. The sqlite database also uses this class, but defines a custom init function to handle the rse.db file.

  add(uid)

    Create a new repo based on a uid that matches to a parser.

  clear()

    clear (delete) all repos. This could be improved to cascade instead.

  create_database()

    create the databsae based on the string, whether it’s relational or sqlite. self.db must be defined.

  delete_parser(name)

    delete all repos for a parser, based on parser’s name (str).

  delete_repo(uid)

    delete a repo based on a specific repo id.

  exists(uid)

    Determine if a repo exists.

  get(uid=None)

    Get a repo based on a uid. Exits on error if doesn’t exist. If a uid is not provided, get the last updated repository.

  get_or_create(uid)

    Determine if a repo exists.

  label(repo, key, value, force=False)

    Update a repository with a specific key/value pair.

  list_repos(name=None)

    list repos, either under a particular parser name (if provided) or just the parsers.

  search(query, taxonomy=None, criteria=None)

    Search across the database for a particular query.

  update(repo, updates=None, rewrite=False)

    update a repo with a json dictionary.

## rse.main.database.sqlite module

Copyright (C) 2020 Vanessa Sochat.

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

class rse.main.database.sqlite.SqliteDatabase(config_dir, config=None, **kwargs)

  Bases: `rse.main.database.relational.RelationalDatabase`

  A SqliteDatabase writes to a rse.db file in $HOME/.rse. This is the suggested database backend for QueueMe, as it doesn’t require anything beyond a filesystem and still allows for relational type queries.

  database = 'sqlite'

## Module contents

Copyright (C) 2020 Vanessa Sochat.

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

rse.main.database.init_db(database, config_dir=None, database_string='', config=None)

  Initialize the database, meaning a base client and appropriate functions to save, or generate a unique ID based on the backend being used. Each client has it’s own init to check for a connection (or filesystem path existence) and then functions to interact with entities.