aiscalator.core package — AIscalator 0.1.0 documentation

### Navigation

- ☰
- AIscalator-0.1.0 »
- aiscalator »
- aiscalator package »

Fork me on GitHub

# aiscalator.core package

## Submodules

## aiscalator.core.config module

Handles configurations files for the application

class `aiscalator.core.config.` `AiscalatorConfig`(user_config_override=None, step_config=None, steps_selection=None)

  Bases: `object`

  A configuration object for the Aiscalator application.

  This object stores:
    - global configuration for the whole application
    - configuration for a particular context specified in a step configuration file.
    - In this case, we might even focus on a particular step.

  …

  Variables: | _user_config_override (str) – path to the specified user config _app_conf – global configuration object for the application _step_config (str) – path to the step configuration (or plain configuration as string) _focused_steps (list) – list of selected steps _step_it (int) – index of the step being processed _step – configuration object for the currently processed step

  Methods

  `app_config`() |
  returns: | str – the configuration object for the aiscalator application
  `app_config_has`(field) | Tests if the applicatin config has a configuration value for the field.
  `container_name`() | Return the docker container name to execute this step
  `extract_parameters`() | Returns a list of docker parameters
  `file_path`(string) | Returns absolute path of a file from a field of the currently focused step.
  `find_user_config_file`(filename) | Looks for configuration files in the user configuration folder
  `has_step_field`(field) | Tests if the currently focused step has a configuration value for the field.
  `next_step`() | Iterates to the next configuration step from the list of selected steps
  `notebook_output_path`(notebook) | Generates the name of the output notebook
  `root_dir`() |
  returns: | str – Returns the path to the folder containing the step
  `setup_app_config`() | Setup global application configuration.
  `step_config_path`() |
  returns: | str – Returns the path to the step configuration file.
  `step_field`(field) | Returns the value associated with the field for the currently focused step.
  `step_name`() | Returns the name of the currently focused step
  `timestamp_now`() | Depending on how the timezone is configured, returns the timestamp for this instant.
  `user_env_file`() | Find a list of env files to pass to docker containers
  `user_id`() |
  returns: | str – the user id stored when the application was first setup
  `validate_config`() | Check if all the fields in the reference config are defined in focused steps too.

  `app_config`()

    Returns: | str – the configuration object for the aiscalator application

  `app_config_has`(field) → bool

    Tests if the applicatin config has a configuration value for the field.

  `container_name`() → str

    Return the docker container name to execute this step

  `extract_parameters`() → list

    Returns a list of docker parameters

  `file_path`(string)

    Returns absolute path of a file from a field of the currently focused step.

  `find_user_config_file`(filename) → str

    Looks for configuration files in the user configuration folder

    Parameters: | filename (str) – file to search for
    Returns: | str – path to the filename in the user configuration folder

  `has_step_field`(field) → bool

    Tests if the currently focused step has a configuration value for the field.

  `next_step`()

    Iterates to the next configuration step from the list of selected steps

    Returns: | the next configuration object

  `notebook_output_path`(notebook) → str

    Generates the name of the output notebook

  `root_dir`()

    Returns: | str – Returns the path to the folder containing the step configuration file

  `setup_app_config`()

    Setup global application configuration. If not found in the default location, this method will generate a brand new one.

  `step_config_path`()

    Returns: | str – Returns the path to the step configuration file. If it was an URL, it will return the path to the temporary downloaded version of it. If it was a plain string, then returns None

  `step_field`(field)

    Returns the value associated with the field for the currently focused step.

  `step_name`()

    Returns the name of the currently focused step

  `timestamp_now`() → str

    Depending on how the timezone is configured, returns the timestamp for this instant.

  `user_env_file`() → list

    Find a list of env files to pass to docker containers

    Returns: | List – env files

  `user_id`() → str

    Returns: | str – the user id stored when the application was first setup

  `validate_config`()

    Check if all the fields in the reference config are defined in focused steps too. Otherwise raise an Exception (either pyhocon.ConfigMissingException or pyhocon.ConfigWrongTypeException)

`aiscalator.core.config.` `convert_to_format`(file: str, output: str, output_format: str)

  Converts a HOCON file to another format

  Parameters: | file (str) – hocon file to convert output (str) – output file to produce output_format (str) – format of the output file
  Returns: | str – the output file

`aiscalator.core.config.` `find_tasks`(tree: pyhocon.config_tree.ConfigTree, path='')

  Find all Tasks objects in the Configuration object and report their paths.

  Parameters: | tree (pyhocon.ConfigTree) – Configuration object path (str) – path that was traversed to get to this tree
  Returns: | list – list of names of Configuration objects containing a definition of a section ‘task’

`aiscalator.core.config.` `generate_global_config`() → str

  Generate a standard configuration file for the application in the user’s home folder ~/.aiscalator/config/aiscalator.conf from the template file in aiscalator/config/template/aiscalator.conf

`aiscalator.core.config.` `generate_user_id`() → str

  Returns: | str – Returns a string identifying this user when the setup was run first

`aiscalator.core.config.` `load_logging_conf`(file)

  Reads and loads the logging configuration file

`aiscalator.core.config.` `parse_step_config`(step_config)

  Interpret the step_config to produce a step configuration object. It could be provided as: - a path to a local file - a url to a remote file - the plain configuration stored as string

  Returns: | Step configuration object

`aiscalator.core.config.` `select_steps`(step_conf, steps_selection: list) → list

  Extract the list of step objects corresponding to the list of names provided.

  Parameters: | step_conf – step configuration object steps_selection (list) – list of names of step to extract
  Returns: | list – list of tuples of (step_name, step) of selected configuration objects

`aiscalator.core.config.` `setup_logging`()

  Setup the logging configuration of the application

`aiscalator.core.config.` `validate_configs`(test, reference, path, missing_exception=True, type_mismatch_exception=True)

  Recursively check two configs if they match

  Parameters: | test – configuration object to test reference – reference configuration object path (str) – this accumulates the recursive path for details in Exceptions missing_exception (bool) – when a missing field is found, raise xception? type_mismatch_exception (bool) – when a field has type mismatch, raise xception?

## aiscalator.core.log_regex_analyzer module

Class to parse output logs from subprocess and catch particular expressions

class `aiscalator.core.log_regex_analyzer.` `LogRegexAnalyzer`(pattern=None)

  Bases: `object`

  A regular expression analyzer object to parse logs and extract values from patterns in the logs. …

  Variables: | _artifact (str) – Value of the pattern found in the logs _pattern (bytes) – Regular expression to search for in the logs

  Methods

  `artifact`() | Returns the artifact extracted from the logs.
  `grep_logs`(pipe) | Reads the logs and extract values defined by the pattern

  `artifact`()

    Returns the artifact extracted from the logs.

  `grep_logs`(pipe)

    Reads the logs and extract values defined by the pattern

    Parameters: | pipe – Stream of logs to analyze

## aiscalator.core.utils module

Various Utility functions

class `aiscalator.core.utils.` `BackgroundThreadRunner`(command, log_function, no_redirect=False)

  Bases: `object`

  Worker Thread to run logging output in the background

  …

  Variables: | _process – Process object of the command running in the background _log_function (function(stream -> bool)) – callback function to log the output of the command _no_redirect (bool) – whether the subprocess STDOUT and STDERR should be redirected to logs _worker (Thread) – Thread object

  Methods

  `process`() | Returns the process object.
  `run`() | Starts the Thread, process the output of the process.

  `process`()

    Returns the process object.

  `run`()

    Starts the Thread, process the output of the process.

`aiscalator.core.utils.` `copy_replace`(src, dst, pattern='', replace_value='')

  Copies a file from src to dst replacing pattern by replace_value

  Parameters: | src (string) – Path to the source filename to copy from dst (string) – Path to the output filename to copy to pattern (string) – Pattern to replace inside the src file replace_value – Value to replace by in the dst file

`aiscalator.core.utils.` `data_file`(path)

  Utility function to find resources data file packaged along with code

  Parameters: | path (path) – path to the resource file in the package
  Returns: | absolute path to the resource data file

`aiscalator.core.utils.` `find`(collection, item, field='name')

  Finds an element in a collection which has a field equal to particular item value

  Parameters: | collection (Set) – Collection of objects item – value of the item that we are looking for field (string) – Name of the field from the object to inspect
  Returns: | object – Corresponding element that has a field matching item in the collection

`aiscalator.core.utils.` `log_info`(pipe)

  Default logging function

`aiscalator.core.utils.` `subprocess_run`(command, log_function=<function log_info>, no_redirect=False, wait=True)

  Run command in a subprocess while redirecting output to log_function.

  The subprocess either runs synchroneoulsy or in the background depending on the wait parameter.

  Parameters: | command (List) – Command to run in the subprocess log_function (function) – Callback function to log the output of the subprocess no_redirect (bool) – whether the subprocess STDOUT and STDERR should be redirected to logs wait (bool) – Whether the subprocess should be run synchroneously or in the background
  Returns: | int – return code of the subprocess BackgroundThreadRunner – the thread running in the background

## Module contents

### Navigation

- index
- modules |
- next |
- previous |