GC3Libs modules — gc3pie v1.0rc4 documentation

### Navigation

- index
- modules |
- previous |
- gc3pie v1.0rc4 documentation »

# GC3Libs modules

## gc3libs

GC3Libs is a python package for controlling the life-cycle of a Grid or batch computational job.

GC3Libs provides services for submitting computational jobs to Grids and batch systems, controlling their execution, persisting job information, and retrieving the final output.

GC3Libs takes an application-oriented approach to batch computing. A generic `Application` class provides the basic operations for controlling remote computations, but different `Application` subclasses can expose adapted interfaces, focusing on the most relevant aspects of the application being represented.

class `gc3libs.` `Application`(executable, arguments, inputs, outputs, output_dir, **kw)

  Support for running a generic application with the GC3Libs. The following parameters are required to create an Application instance:

  executable
    (string) name of the application binary to be launched on the remote resource; the specifics of how this is handled are dependent on the submission backend, but you may always run a script that you upload through the inputs mechanism by specifying `./scriptname` as executable.
  arguments
    list of command-line arguments to pass to executable; any object in the list will be converted to string via Python `str()`. Note that, in contrast with the UNIX `execvp()` usage, the first argument in this list will be passed as `argv[1]`, i.e., `argv[0]` will always be equal to executable.
  inputs

    files that will be copied from the local computer to the remote execution node before execution starts.

    There are two possible ways of specifying the inputs parameter:

    > - It can be a Python dictionary: keys are local file paths, values are remote file names.
    > - It can be a Python list: each item in the list should be a pair (local_file_name, remote_file_name); specifying a single string file_name is allowed as a shortcut and will result in both local_file_name and remote_file_name being equal. If an absolute path name is specified as remote_file_name, then an `InvalidArgument` exception is thrown.

  outputs

    list of files that will be copied back from the remote execution node back to the local computer after execution has completed.

    There are two possible ways of specifying the outputs parameter:

    - It can be a Python dictionary: keys are local file paths, values are remote file names.
    - It can be a Python list: each item in the list should be a pair (remote_file_name, local_file_name); specifying a single string file_name is allowed as a shortcut and will result in both local_file_name and remote_file_name being equal. If an absolute path name is specified as remote_file_name, then an `InvalidArgument` exception is thrown.
  output_dir
    Path to the base directory where output files will be downloaded. Output file names are interpreted relative to this base directory.

  The following optional parameters may be additionally specified as keyword arguments and will be given special treatment by the Application class logic:

  requested_cores,`requested_memory`,`requested_walltime`
    specify resource requirements for the application: the number of independent execution units (CPU cores), amount of memory (in GB; will be converted to a whole number by truncating any decimal digits), amount of wall-clock time to allocate for the computational job (in hours; will be converted to a whole number by truncating any decimal digits).
  environment
    a list of pairs (name, value): the environment variable whose name is given by the contents of the string name will be defined as the content of string value (i.e., as if “export name=value” was executed prior to starting the application). Alternately, one can pass in a list of strings of the form “name=value”.
  grid
    if not None, equivalent to calling Task.attach(self, grid); enables the use of the Task/active job control interface
  stdin
    file name of a file whose contents will be fed as standard input stream to the remote-executing process.
  stdout
    name of a file where the standard output stream of the remote executing process will be redirected to; will be automatically added to outputs.
  stderr
    name of a file where the standard error stream of the remote executing process will be redirected to; will be automatically added to outputs.
  join
    if this evaluates to True, then standard error is redirected to the file specified by stdout and stderr is ignored. (join has no effect if stdout is not given.)
  tags
    list of tag names (string) that must be present on a resource in order to be eligible for submission; in the ARC backend, tags are interpreted as run-time environments (RTE) to request.

  Any other keyword arguments will be set as instance attributes, but otherwise ignored by the Application constructor.

  After successful construction, an Application object is guaranteed to have the following instance attributes:

  executable
    a string specifying the executable name
  arguments
    list of strings specifying command-line arguments for executable invocation; possibly empty
  inputs
    dictionary mapping local file name (a string) to a remote file name (a string); remote file names are relative paths (root directory is the remote job folder)
  outputs
    dictionary mapping remote file name (a string) to a local file name (a string); remote file names are relative paths (root directory is the remote job folder)
  output_dir
    Path to the base directory where output files will be downloaded. Output file names are interpreted relative to this base directory.
  final_output_retrieved
    boolean, indicating whether job output has been fetched from the remote resource; if True, you cannot assume that data is still available remotely. (Note: for jobs in `TERMINATED` state, the output can be retrieved only once!)
  execution
    a Run instance; its state attribute is initially set to `NEW`
  environment
    dictionary mapping environment variable names to the requested value (string); possibly empty
  stdin
    None or a string specifying a (local) file name. If stdin is not None, then it matches a key name in inputs
  stdout
    None or a string specifying a (remote) file name. If stdout is not None, then it matches a key name in outputs
  stderr
    None or a string specifying a (remote) file name. If stdout is not None, then it matches a key name in outputs
  join
    boolean value, indicating whether stdout and stderr are collected into the same file
  tags
    list of strings specifying the tags to request in each resource for submission; possibly empty.
  `clone`()
    Return a deep copy of this Application object, with the .execution instance variable reset to a fresh new instance of Run.
  `cmdline`(resource)

    Return a string, suitable for invoking the application from a UNIX shell command-line.

    The default implementation just concatenates executable and arguments separating them with whitespace; this is hardly correct for any application, so you should override this method in derived classes to provide appropriate invocation templates.

  `fetch_output_error`(ex)

    Invocation of Core.fetch_output() on this object failed; ex is the Exception that describes the error.

    If this method returns an exception object, that is raised as a result of the Core.fetch_output(), otherwise the return value is ignored and Core.fetch_output returns None.

    Default is to return ex unchanged; override in derived classes to change this behavior.

  `new`()

    Called when the job state is (re)set to NEW.

    Note this will not be called when the application object is created, rather if the state is reset to NEW after it has already been submitted.

    The default implementation does nothing, override in derived classes to implement additional behavior.

  `postprocess`(dir)

    Called when the final output of the job has been retrieved to local directory dir.

    The default implementation does nothing, override in derived classes to implement additional behavior.

  `qsub`(resource, _suppress_warning=False, **kw)

    Get an SGE `qsub` command-line invocation for submitting an instance of this application. Return a pair (cmd, script), where cmd is the command to run to submit an instance of this application to the SGE batch system, and script –if it’s not None– is written to a new file, whose name is then appended to cmd.

    In the construction of the command-line invocation, one should assume that all the input files (as named in Application.inputs) have been copied to the current working directory, and that output files should be created in this same directory.

    The default implementation just prefixes any output from the cmdline method with an SGE `qsub` invocation of the form `qsub -cwd -S /bin/sh` + resource limits. Note that there is no generic way of requesting a certain number of cores in SGE: it all depends on the installed parallel environment, and these are totally under control of the local sysadmin; therefore, any request for cores is ignored and a warning is logged.

    Override this method in application-specific classes to provide appropriate invocation templates.

  `running`()

    Called when the job state transitions to RUNNING, i.e., the job has been successfully started on a (possibly) remote resource.

    The default implementation does nothing, override in derived classes to implement additional behavior.

  `stopped`()

    Called when the job state transitions to STOPPED, i.e., the job has been remotely suspended for an unknown reason and cannot automatically resume execution.

    The default implementation does nothing, override in derived classes to implement additional behavior.

  `submit_error`(exs)

    Invocation of Core.submit() on this object failed; exs is a list of Exception objects, one for each attempted submission.

    If this method returns an exception object, that is raised as a result of the Core.submit(), otherwise the return value is ignored and Core.submit returns None.

    Default is to always return None; override in derived classes to change this behavior.

  `submitted`()

    Called when the job state transitions to SUBMITTED, i.e., the job has been successfully sent to a (possibly) remote execution resource and is now waiting to be scheduled.

    The default implementation does nothing, override in derived classes to implement additional behavior.

  `terminated`()

    Called when the job state transitions to TERMINATED, i.e., the job has finished execution (with whatever exit status, see returncode) and its execution cannot resume.

    The default implementation does nothing, override in derived classes to implement additional behavior.

  `update_job_state_error`(ex)

    Invocation of Core.update_job_state() on this object failed; ex is the Exception that describes the error.

    If this method returns an exception object, that is raised as a result of the Core.update_job_state(), otherwise the return value is ignored and Core.update_job_state returns None.

    Note that returning an Exception instance interrupts the normal flow of Core.update_job_state: in particular, the execution state is not updated and state transition methods will not be called.

    Default is to return None; override in derived classes to change this behavior.

  `xrsl`(resource)

    Return a string containing an xRSL sequence, suitable for submitting an instance of this application through ARC’s `ngsub` command.

    The default implementation produces XRSL content based on the construction parameters; you should override this method to produce XRSL tailored to your application.

class `gc3libs.` `Run`(initializer=None, **keywd)

  A specialized dict-like object that keeps information about the execution state of an Application instance.

  A Run object is guaranteed to have the following attributes:

  > log
  >   A gc3libs.utils.Log instance, recording human-readable text messages on events in this job’s history.
  > info
  >   A simplified interface for reading/writing messages to Run.log. Reading from the info attribute returns the last message appended to log. Writing into info appends a message to log.
  > timestamp
  >   Dictionary, recording the most recent timestamp when a certain state was reached. Timestamps are given as UNIX epochs.

  For properties state, signal and returncode, see the respective documentation.

  Run objects support attribute lookup by both the `[...]` and the `.` syntax; see gc3libs.utils.Struct for examples.

  `exitcode`
    The “exit code” part of a Run.returncode, see os.WEXITSTATUS. This is an 8-bit integer, whose meaning is entirely application-specific. However, the value -1 is used to mean that an error has occurred and the application could not end its execution normally.
  `info`

    A simplified interface for reading/writing entries into log.

    Setting the info attribute appends a message to the log:

        >>> j1.info = 'a message'
        >>> j1.info = 'a second message'

    Getting the value of the info attribute returns the last message entered in the log:

        >>> j1.info
        'a second message'

  `returncode`

    The returncode attribute of this job object encodes the Run termination status in a manner compatible with the POSIX termination status as implemented by os.WIFSIGNALED and os.WIFEXITED.

    However, in contrast with POSIX usage, the exitcode and the signal part can both be significant: in case a Grid middleware error happened after the application has successfully completed its execution. In other words, os.WEXITSTATUS(job.returncode) is meaningful iff os.WTERMSIG(job.returncode) is 0 or one of the pseudo-signals listed in Run.Signals.

    Run.exitcode and Run.signal are combined to form the return code 16-bit integer as follows (the convention appears to be obeyed on every known system):

    > Bit | Encodes...
    > --- | ---
    > 0..7 | signal number
    > 8 | 1 if program dumped core.
    > 9..16 | exit code

    Note: the “core dump bit” is always 0 here.

    Setting the returncode property sets exitcode and signal; you can either assign a (signal, exitcode) pair to returncode, or set returncode to an integer from which the correct exitcode and signal attribute values are extracted:

        >>> j = Run()
        >>> j.returncode = (42, 56)
        >>> j.signal
        42
        >>> j.exitcode
        56

        >>> j.returncode = 137
        >>> j.signal
        9
        >>> j.exitcode
        0

    See also Run.exitcode and Run.signal.

  `signal`

    The “signal number” part of a Run.returncode, see os.WTERMSIG for details.

    The “signal number” is a 7-bit integer value in the range 0..127; value 0 is used to mean that no signal has been received during the application runtime (i.e., the application terminated by calling `exit()`).

    The value represents either a real UNIX system signal, or a “fake” one that GC3Libs uses to represent Grid middleware errors (see Run.Signals).

  `state`

    The state a Run is in.

    The value of Run.state must always be a value from the Run.State enumeration, i.e., one of the following values.

    Run.State value | purpose | can change to
    --- | --- | ---
    NEW | Job has not yet been submitted/started (i.e., gsub not called) | SUBMITTED (by gsub)
    SUBMITTED | Job has been sent to execution resource | RUNNING, STOPPED
    STOPPED | Trap state: job needs manual intervention (either user- or sysadmin-level) to resume normal execution | TERMINATED (by gkill), SUBMITTED (by miracle)
    RUNNING | Job is executing on remote resource | TERMINATED
    TERMINATED | Job execution is finished (correctly or not) and will not be resumed | None: final state

    When a `Run` object is first created, it is assigned the state NEW. After a successful invocation of Core.submit(), it is transitioned to state SUBMITTED. Further transitions to RUNNING or STOPPED or TERMINATED state, happen completely independently of the creator progra; the Core.update_job_state() call provides updates on the status of a job.

    The STOPPED state is a kind of generic “run time error” state: a job can get into the STOPPED state if its execution is stopped (e.g., a SIGSTOP is sent to the remote process) or delayed indefinitely (e.g., the remote batch system puts the job “on hold”). There is no way a job can get out of the STOPPED state automatically: all transitions from the STOPPED state require manual intervention, either by the submitting user (e.g., cancel the job), or by the remote systems administrator (e.g., by releasing the hold).

    The TERMINATED state is the final state of a job: once a job reaches it, it cannot get back to any other state. Jobs reach TERMINATED state regardless of their exit code, or even if a system failure occurred during remote execution; actually, jobs can reach the TERMINATED status even if they didn’t run at all, for example, in case of a fatal failure during the submission step.

class `gc3libs.` `Task`(grid=None)

  Mix-in class implementing a facade for job control.

  A Task can be described as an “active” job, in the sense that all job control is done through methods on the Task instance itself; contrast this with operating on Application objects through a Core or Engine instance.

  The following pseudo-code is an example of the usage of the Task interface for controlling a job. Assume that GamessApplication is inheriting from Task (and it actually is):

      t = GamessApplication(input_file)
      t.submit()
      # ... do other stuff
      t.update()
      # ... take decisions based on t.execution.state
      t.wait() # blocks until task is terminated

  Each Task object has an execution attribute: it is an instance of class `Run`, initialized with a new instance of Run, and at any given time it reflects the current status of the associated remote job. In particular, execution.state can be checked for the current task status.

  exception `DetachedFromGridError`(msg, do_log=False)
    Raised when a method (other than `attach()`) is called on a detached Task instance.
  exception `Task.` `Error`(msg, do_log=False)
    Generic error condition in a Task object.
  exception `Task.` `UnexpectedStateError`(msg, do_log=False)
    Raised by `Task.progress()` when a job lands in STOPPED or TERMINATED state.
  `Task.` `attach`(grid)
    Use the given Grid interface for operations on the job associated with this task.
  `Task.` `detach`()
    Remove any reference to the current grid interface. After this, calling any method other than `attach()` results in an exception `Task.DetachedFromGridError` being thrown.
  `Task.` `fetch_output`(output_dir=None, overwrite=False)

    Retrieve the outputs of the computational job associated with this task into directory output_dir, or, if that is None, into the directory whose path is stored in instance attribute .output_dir.

    See `gc3libs.Core.fetch_output()` for a full explanation.

    Returns: | Path to the directory where the job output has been collected.

  `Task.` `kill`()

    Terminate the computational job associated with this task.

    See `gc3libs.Core.kill()` for a full explanation.

  `Task.` `peek`(what='stdout', offset=0, size=None)

    Download size bytes (at offset offset from the start) from the associated job standard output or error stream, and write them into a local file. Return a file-like object from which the downloaded contents can be read.

    See `gc3libs.Core.peek()` for a full explanation.

  `Task.` `progress`()

    Advance the associated job through all states of a regular lifecycle. In detail:

    > 1. If execution.state is NEW, the associated job is started.
    > 2. The state is updated until it reaches TERMINATED
    > 3. Output is collected and the final returncode is returned.

    An exception Task.Error is raised if the job hits state STOPPED or UNKNOWN during an update in phase 2.

    When the job reaches TERMINATED state, the output is retrieved, and the return code (stored also in .returncode) is returned; if the job is not yet in TERMINATED state, calling progress returns None.

    Raises: | exception `Task.UnexpectedStateError` if the associated job goes into state STOPPED or UNKNOWN
    Returns: | job final returncode, or None if the execution state is not TERMINATED.

  `Task.` `submit`()
    Start the computational job associated with this Task instance.
  `Task.` `update_state`()
    In-place update of the execution state of the computational job associated with this Task. After successful completion, .execution.state will contain the new state.
  `Task.` `wait`(interval=60)

    Block until the associated job has reached TERMINATED state, then return the job’s return code. Note that this does not automatically fetch the output.

    Parameter: | integer interval – Poll job state every this number of seconds

`gc3libs.` `configure_logger`(level=40, name=None, format='sphinx-build:[, %(asctime)s], %(levelname)-8s: %(message)s', datefmt='%Y-%m-%d %H:%M:%S')

  Configure the `gc3libs` logger.

  Arguments level, format and datefmt set the corresponding arguments in the logging.basicConfig() call.

  If a user configuration file exists in `~/.gc3/logging.conf`, it is read and used for more advanced configuration; if it does not exist, then a sample one is created.

## gc3libs.core

Top-level interface to Grid functionality.

class `gc3libs.core.` `Engine`(grid, tasks=[], store=None, can_submit=True, can_retrieve=True, max_in_flight=0, max_submitted=0, output_dir=None, fetch_output_overwrites=False)

  Submit tasks in a collection, and update their state until a terminal state is reached. Specifically:

  > - tasks in NEW state are submitted;
  > - the state of tasks in SUBMITTED, RUNNING or STOPPED state is updated;
  > - when a task reaches TERMINATED state, its output is downloaded.

  The behavior of Engine instances can be further customized by setting the following instance attributes:

  > can_submit
  >   Boolean value: if False, no task will be submitted.
  > can_retrieve
  >   Boolean value: if False, no output will ever be retrieved.
  > max_in_flight
  >   If >0, limit the number of tasks in SUBMITTED or RUNNING state: if the number of tasks in SUBMITTED, RUNNING or STOPPED state is greater than max_in_flight, then no new submissions will be attempted.
  > max_submitted
  >   If >0, limit the number of tasks in SUBMITTED state: if the number of tasks in SUBMITTED, RUNNING or STOPPED state is greater than max_submitted, then no new submissions will be attempted.
  > output_dir
  >   Base directory for job output; if not None, each task’s results will be downloaded in a subdirectory named after the task’s permanent_id.
  > fetch_output_overwrites
  >   Default value to pass as the overwrite argument to `Core.fetch_output()` when retrieving results of a terminated task.

  Any of the above can also be set by passing a keyword argument to the constructor:

      >>> e = Engine(can_submit=False)
      >>> e.can_submit
      False

  `add`(app)
    Add app to the list of tasks managed by this Engine.
  `fetch_output`(task, output_dir=None, overwrite=False)
    Proxy for Core.fetch_output (which see).
  `free`(task)
    Proxy for Core.free (which see); in addition, remove task from the list of managed tasks.
  `kill`(task)
    Schedule a task for killing on the next perform run.
  `peek`(task, what='stdout', offset=0, size=None, **kw)
    Proxy for Core.peek (which see).
  `progress`()

    Update state of all registered tasks and take appropriate action. Specifically:

    > - tasks in NEW state are submitted;
    > - the state of tasks in SUBMITTED, RUNNING or STOPPED state is updated;
    > - when a task reaches TERMINATED state, its output is downloaded.

    The max_in_flight and max_submitted limits (if >0) are taken into account when attempting submission of tasks.

  `remove`(app)
    Remove a app from the list of tasks managed by this Engine.
  `submit`(task)
    Submit task at the next invocation of perform. Actually, the task is just added to the collection of managed tasks, regardless of its state.
  `update_job_state`(*tasks)
    Return list of current states of the given tasks. States will only be updated at the next invocation of perform; in particular, no state-change handlers are called as a result of calling this method.

`gc3libs.core.` `read_config`(*locations)
  Read each of the configuration files listed in locations, and return a (defaults, resources, auths) triple that can be passed to the Core class constructor.

## gc3libs.Default

Warning

This module is deprecated and will be removed in a future release. Do not depend on it.

A namespace for all constants and default values used in the GC3Libs package.

## gc3libs.Exceptions

Exceptions specific to the gc3libs package.

exception `gc3libs.Exceptions.` `ConfigurationError`(msg, do_log=True)
  Raised when the configuration file (or parts of it) could not be read/parsed. Also used to signal that a required parameter is missing or has an unknown/invaliud value.

exception `gc3libs.Exceptions.` `DataStagingError`(msg, do_log=False)
  Raised when problems with copying data to or from the remote execution site occurred.

exception `gc3libs.Exceptions.` `Error`(msg, do_log=False)

  A non-fatal error.

  Depending on the nature of the task, steps could be taken to continue, but users must be aware that an error condition occurred, so the message is sent to the logs at the ERROR level.

  This is the base class for all error-level exceptions.

exception `gc3libs.Exceptions.` `FatalError`(msg, do_log=True)

  A fatal error; execution cannot continue and program should report to user and then stop.

  The message is sent to the logs at CRITICAL level when the exception is first constructed.

  This is the base class for all fatal exceptions.

exception `gc3libs.Exceptions.` `InputFileError`(msg, do_log=True)
  Raised when an input file is specified, which does not exist or cannot be read.

exception `gc3libs.Exceptions.` `InternalError`(msg, do_log=False)
  Raised when some function cannot fulfill its duties, for reasons that do not depend on the library client code. For instance, when a response string gotten from an external command cannot be parsed as expected.

exception `gc3libs.Exceptions.` `InvalidArgument`(msg, do_log=False)
  Raised when the arguments passed to a function do not honor some required contract. For instance, either one of two optional arguments must be provided, but none of them was.

exception `gc3libs.Exceptions.` `InvalidOperation`(msg, do_log=False)
  Raised when an operation is attempted, that is not considered valid according to the system state. For instance, trying to retrieve the output of a job that has not yet been submitted.

exception `gc3libs.Exceptions.` `InvalidResourceName`(msg, do_log=False)
  Raised to signal that no computational resource with the given name is defined in the configuration file.

exception `gc3libs.Exceptions.` `InvalidUsage`(msg, do_log=True)

  Raised when a command is not provided all required arguments on the command line, or the arguments do not match the expected syntax.

  Since the exception message is the last thing a user will see, try to be specific about what is wrong on the command line.

exception `gc3libs.Exceptions.` `LoadError`(msg, do_log=False)
  Raised upon errors loading a job from the persistent storage.

exception `gc3libs.Exceptions.` `NoConfigurationFile`(msg, do_log=True)
  Raised when the configuration file cannot be read (e.g., does not exist or has wrong permissions), or cannot be parsed (e.g., is malformed).

exception `gc3libs.Exceptions.` `NoResources`(msg, do_log=False)
  Raised to signal that no resources are defined, or that none are compatible with the request.

exception `gc3libs.Exceptions.` `OutputNotAvailableError`(msg, do_log=False)
  Raised upon attempts to retrieve the output for jobs that are still in NEW or SUBMITTED state.

exception `gc3libs.Exceptions.` `UnknownJobState`(msg, do_log=False)
  Raised when a job state is gotten from the Grid middleware, that is not handled by the GC3Libs code. Might actually mean that there is a version mismatch between GC3Libs and the Grid middleware used.

## gc3libs.persistence

Facade to store and retrieve Job information from permanent storage.

class `gc3libs.persistence.` `FilesystemStore`(directory='/home/rmurri/.gc3/jobs', idfactory=<class 'gc3libs.persistence.Id'>, protocol=2)

  Save and load objects in a given directory. Uses Python’s standard pickle module to serialize objects onto files.

  All objects are saved as files in the given directory (default: gc3libs.Default.JOBS_DIR). The file name is the object ID.

  If an object contains references to other Persistable objects, these are saved in the file they would have been saved if the save method was called on them in the first place, and only an “external reference” is saved in the pickled container. This ensures that: (1) only one copy of a shared object is ever saved, and (2) any shared reference to Persistable objects is correctly restored when restoring the container.

  The default idfactory assigns object IDs by appending a sequential number to the class name; see class Id for details.

  The protocol argument specifies the pickle protocol to use (default: pickle protocol 0). See the pickle module documentation for details.

  class `Pickler`(parent, file, root_obj)
    Pickle a Python object, saving the Persistable instances contained in it as external references through the same FilesystemStore.
  `FilesystemStore.` `list`()

    Return list of IDs of saved Job objects.

    This is an optional method; classes that do not implement it should raise a NotImplementedError exception.

  `FilesystemStore.` `load`(id_)
    Load a saved object given its ID, and return it.
  `FilesystemStore.` `remove`(id_)
    Delete a given object from persistent storage, given its ID.
  `FilesystemStore.` `replace`(id_, obj)
    Replace the object already saved with the given ID with a copy of obj.
  `FilesystemStore.` `save`(obj)
    Save an object, and return an ID.

class `gc3libs.persistence.` `Id`

  An automatically-generated “unique identifier” (a string-like object). Object identifiers are temporally unique: no identifier will (ever) be re-used, even in different invocations of the program.

  The unique job identifier has the form “PREFIX.XXX” where “XXX” is a decimal number, and “PREFIX” defaults to the object class name but can be overridden in the Id constructor.

  Two object IDs can be compared iff they have the same prefix; in which case, the result of the comparison is the same as comparing the two sequence numbers.

  static `reserve`(n)
    Pre-allocate n IDs. Successive invocations of the Id constructor will return one of the pre-allocated, with a potential speed gain if many Id objects are constructed in a loop.

class `gc3libs.persistence.` `JobIdFactory`
  Override :py:class:`Id` behavior and generate IDs starting with a lowercase `job` prefix.

class `gc3libs.persistence.` `Persistable`

  A mix-in class to mark that an object should be persisted by its ID.

  Any instance of this class is saved as an “external reference” when a container holding a reference to it is saved.

class `gc3libs.persistence.` `Store`

  Interface for storing and retrieving objects on permanent storage.

  Each save operation returns a unique “ID”; each ID is a Python string value, which is guaranteed to be temporally unique, i.e., no two save operations in the same persistent store can result in the same IDs being assigned to different objects. The “ID” is also stored in the instance attribute _id.

  Any Python object can stored, provided it meets the following conditions:

  > - it can be pickled with Python’s standard module pickle.
  > - the instance attribute persistent_id is reserved for use by the Store class: it should not be set or altered by other parts of the code.

  `list`(**kw)

    Return list of IDs of saved Job objects.

    This is an optional method; classes that do not implement it should raise a NotImplementedError exception.

  `load`(id_)
    Load a saved object given its ID, and return it.
  `remove`(id_)
    Delete a given object from persistent storage, given its ID.
  `replace`(id_, obj)
    Replace the object already saved with the given ID with a copy of obj.
  `save`(obj)
    Save an object, and return an ID.

## gc3libs.application

Support for running a generic application with the GC3Libs.

`gc3libs.application.` `get`(tag, *args, **kwargs)

  Return an instance of the specific application class associated with tag. Example:

      >>> app = get('gamess')
      >>> isinstance(app, GamessApplication)
      True

  The returned object is always an instance of a sub-class of Application:

      >>> isinstance(app, Application)
      True

`gc3libs.application.` `register`(application_class, tag)
  Register an application class with name tag. After registration, application factories can be retrieved by tag name with `gc3libs.application.get('tag')`.

## gc3libs.application.gamess

Specialized support for computational jobs running GAMESS-US.

class `gc3libs.application.gamess.` `GamessApplication`(inp_file_path, *other_input_files, **kw)

  Specialized Application object to submit computational jobs running GAMESS-US.

  The only required parameter for construction is the input file name; any other argument names an additional input file, that is added to the Application.inputs list, but not otherwise treated specially.

  Any other kyword parameter that is valid in the Application class can be used here as well, with the exception of input and output. Note that a GAMESS-US job is always submitted with join = True, therefore any stderr setting is ignored.

  `qgms`(resource, **kw)
    Return a qgms invocation to run GAMESS-US with the parameters embedded in this object.
  `qsub`(resource, **kw)
    Return a qgms invocation to run GAMESS-US with the parameters embedded in this object.

## gc3libs.application.rosetta

Specialized support for computational jobs running programs in the Rosetta suite.

class `gc3libs.application.rosetta.` `RosettaApplication`(application, inputs, outputs=[], flags_file=None, database=None, arguments=[], **kw)

  Specialized Application object to submit one run of a single application in the Rosetta suite.

  Required parameters for construction:

  > - application: name of the Rosetta application to call (e.g., “docking_protocol” or “relax”)
  > - inputs: a dict instance, keys are Rosetta `-in:file:*` options, values are the (local) path names of the corresponding files. (Example: `inputs={"-in:file:s":"1brs.pdb"}`)
  > - outputs: list of output file names to fetch after Rosetta has finished running.

  Optional parameters:

  > - flags_file: path to a local file containing additional flags for controlling Rosetta invocation; if None, a local configuration file will be used.
  > - database: (local) path to the Rosetta DB; if this is not specified, then it is assumed that the correct location will be available at the remote execution site as environment variable `ROSETTA_DB_LOCATION`
  > - arguments: If present, they will be appended to the Rosetta application command line.

  `postprocess`(output_dir)
    Extract output files from the tar archive created by the ‘rosetta.sh’ script.

class `gc3libs.application.rosetta.` `RosettaDockingApplication`(pdb_file_path, native_file_path=None, number_of_decoys_to_create=1, flags_file=None, **kw)

  Specialized Application class for executing a single run of the Rosetta “docking_protocol” application.

  Currently used in the grosetta app.

## gc3libs.authentication

Authentication support for the GC3Libs.

class `gc3libs.authentication.` `NoneAuth`(**auth_dicts)
  Auth proxy to use when no auth is needed.

## gc3libs.authentication.ssh

Authentication support for accessing resources through the SSH protocol.

## gc3libs.authentication.grid

Authentication support with Grid proxy certificates.

## gc3libs.backends

Interface to different resource management systems for the GC3Libs.

class `gc3libs.backends.` `LRMS`

  Base class for interfacing with a computing resource.

  `cancel_job`(app)
    Cancel a running job. If app is associated to a queued or running remote job, tell the execution middleware to cancel it.
  `free`(app)

    Free up any remote resources used for the execution of app. In particular, this should delete any remote directories and files.

    Call this method when app.execution.state is anything other than TERMINATED results in undefined behavior and will likely be the cause of errors later on. Be cautious.

  `get_resource_status`()
    Update the status of the resource associated with this LRMS instance in-place. Return updated Resource object.
  `get_results`(job, download_dir, overwrite=False)
    Retrieve job output files into local directory download_dir (which must already exists). Will not overwrite existing files, unless the optional argument overwrite is True.
  `is_valid`()
    Determine if a provided LRMS instance is valid. Returns True or False.
  `peek`(app, remote_filename, local_file, offset=0, size=None)

    Download size bytes (at offset offset from the start) from remote file remote_filename and write them into local_file. If size is None (default), then snarf contents of remote file from offset unto the end.

    Argument local_file is either a local path name (string), or a file-like object supporting a .write() method. If local_file is a path name, it is created if not existent, otherwise overwritten.

    Argument remote_filename is the name of a file in the remote job “sandbox”.

    Any exception raised by operations will be passed through.

  `submit_job`(application, job)

    Submit an Application instance to the configured computational resource; return a gc3libs.Job instance for controlling the submitted job.

    This method only returns if the job is successfully submitted; upon any failure, an exception is raised.

    Note:

    > 1. job.state is not altered; it is the caller’s responsibility to update it.
    > 2. the job object may be updated with any information that is necessary for this LRMS to perform further operations on it.

  `update_job_state`(app)
    Query the state of the remote job associated with app and update app.execution.state accordingly. Return the corresponding Run.State; see Run.State for more details.

## gc3libs.backends.arc

Job control on ARC0 resources.

class `gc3libs.backends.arc.` `ArcLrms`(resource, auths)

  Manage jobs through the ARC middleware.

  `cancel_job`(app)
    Cancel a running job. If app is associated to a queued or running remote job, tell the execution middleware to cancel it.
  `free`(app)

    Free up any remote resources used for the execution of app. In particular, this should delete any remote directories and files.

    Call this method when app.execution.state is anything other than TERMINATED results in undefined behavior and will likely be the cause of errors later on. Be cautious.

  `get_resource_status`()
    Update the status of the resource associated with this LRMS instance in-place. Return updated Resource object.
  `get_results`(app, download_dir, overwrite=False)
    Retrieve job output files into local directory download_dir (which must already exists). Will not overwrite existing files, unless the optional argument overwrite is True.
  `peek`(app, remote_filename, local_file, offset=0, size=None)

    Download size bytes (at offset offset from the start) from remote file remote_filename and write them into local_file. If size is None (default), then snarf contents of remote file from offset unto the end.

    Argument local_file is either a local path name (string), or a file-like object supporting a .write() method. If local_file is a path name, it is created if not existent, otherwise overwritten.

    Argument remote_filename is the name of a file in the remote job “sandbox”.

    Any exception raised by operations will be passed through.

  `submit_job`(app)

    Submit an Application instance to the configured computational resource; return a gc3libs.Job instance for controlling the submitted job.

    This method only returns if the job is successfully submitted; upon any failure, an exception is raised.

    Note:

    > 1. job.state is not altered; it is the caller’s responsibility to update it.
    > 2. the job object may be updated with any information that is necessary for this LRMS to perform further operations on it.

  `update_job_state`(app)

    Query the state of the ARC job associated with app and update app.execution.state accordingly. Return the corresponding Run.State; see Run.State for more details.

    The mapping of ARC job statuses to Run.State is as follows:

    > ARC job status | Run.State
    > --- | ---
    > ACCEPTED | SUBMITTED
    > ACCEPTING | SUBMITTED
    > SUBMITTING | SUBMITTED
    > PREPARING | SUBMITTED
    > INLRMS:Q | SUBMITTED
    > INLRMS:R | RUNNING
    > INLRMS:O | RUNNING
    > INLRMS:E | RUNNING
    > INLRMS:X | RUNNING
    > INLRMS:S | STOPPED
    > INLRMS:H | STOPPED
    > FINISHING | RUNNING
    > EXECUTED | RUNNING
    > FINISHED | TERMINATED
    > CANCELING | TERMINATED
    > FINISHED | TERMINATED
    > KILLED | TERMINATED
    > FAILED | TERMINATED
    > DELETED | TERMINATED

    Any other ARC job status is mapped to Run.State.UNKNOWN. In particular, querying a job ID that is not found in the ARC information system will result in UNKNOWN state, as will querying a job that has just been submitted and has not yet found its way to the infosys.

## gc3libs.backends.sge

Job control on SGE clusters (possibly connecting to the front-end via SSH).

class `gc3libs.backends.sge.` `SgeLrms`(resource, auths)

  Job control on SGE clusters (possibly by connecting via SSH to a submit node).

  `cancel_job`(app)
    Cancel a running job. If app is associated to a queued or running remote job, tell the execution middleware to cancel it.
  `free`(app)

    Free up any remote resources used for the execution of app. In particular, this should delete any remote directories and files.

    Call this method when app.execution.state is anything other than TERMINATED results in undefined behavior and will likely be the cause of errors later on. Be cautious.

  `get_resource_status`()
    Update the status of the resource associated with this LRMS instance in-place. Return updated Resource object.
  `get_results`(app, download_dir, overwrite=False)
    Retrieve job output files into local directory download_dir (which must already exists). Will not overwrite existing files, unless the optional argument overwrite is True.
  `peek`(app, remote_filename, local_file, offset=0, size=None)

    Download size bytes (at offset offset from the start) from remote file remote_filename and write them into local_file. If size is None (default), then snarf contents of remote file from offset unto the end.

    Argument local_file is either a local path name (string), or a file-like object supporting a .write() method. If local_file is a path name, it is created if not existent, otherwise overwritten.

    Argument remote_filename is the name of a file in the remote job “sandbox”.

    Any exception raised by operations will be passed through.

  `submit_job`(app)

    Submit an Application instance to the configured computational resource; return a gc3libs.Job instance for controlling the submitted job.

    This method only returns if the job is successfully submitted; upon any failure, an exception is raised.

    Note:

    > 1. job.state is not altered; it is the caller’s responsibility to update it.
    > 2. the job object may be updated with any information that is necessary for this LRMS to perform further operations on it.

  `update_job_state`(app)
    Query the state of the remote job associated with app and update app.execution.state accordingly. Return the corresponding Run.State; see Run.State for more details.

`gc3libs.backends.sge.` `compute_nr_of_slots`(qstat_output)

  Compute the number of total, free, and used/reserved slots from the output of SGE’s `qstat -F`.

  Return a dictionary instance, mapping each host name into a dictionary instance, mapping the strings `total`, `available`, and `unavailable` to (respectively) the the total number of slots on the host, the number of free slots on the host, and the number of used+reserved slots on the host.

  Cluster-wide totals are associated with key `global`.

  Note: The ‘available slots’ computation carried out by this function is unreliable: there is indeed no notion of a ‘global’ or even ‘per-host’ number of ‘free’ slots in SGE. Slot numbers can be computed per-queue, but a host can belong in different queues at the same time; therefore the number of ‘free’ slots available to a job actually depends on the queue it is submitted to. Since SGE does not force users to submit explicitly to a queue, rather encourages use of a sort of ‘implicit’ routing queue, there is no way to compute the number of free slots, as this entirely depends on how local policies will map a job to the available queues.

`gc3libs.backends.sge.` `count_jobs`(qstat_output, whoami)

  Parse SGE’s `qstat` output (as contained in string qstat_output) and return a quadruple (R, Q, r, q) where:

  > - R is the total number of running jobs in the SGE cell (from any user);
  > - Q is the total number of queued jobs in the SGE cell (from any user);
  > - r is the number of running jobs submitted by user whoami;
  > - q is the number of queued jobs submitted by user whoami

`gc3libs.backends.sge.` `get_qsub_jobid`(qsub_output)
  Parse the `qsub` output for the local jobid.

`gc3libs.backends.sge.` `parse_qhost_f`(qhost_output)
  Parse SGE’s `qhost -F` output (as contained in string qhost_output) and return a dict instance, mapping each host name to its attributes.

`gc3libs.backends.sge.` `parse_qstat_f`(qstat_output)
  Parse SGE’s `qstat -F` output (as contained in string qstat_output) and return a dict instance, mapping each queue name to its attributes.

## gc3libs.backends.transport

The Transport class hierarchy provides an abstraction layer to execute commands and copy/move files irrespective of whether the destination is the local computer or a remote front-end that we access via SSH.

## gc3libs.utils

Generic Python programming utility functions.

This module collects general utility functions, not specifically related to GC3Libs. A good rule of thumb for determining if a function or class belongs in here is the following: place a function or class in the `utils` if you could copy its code into the sources of a different project and it would not stop working.

class `gc3libs.utils.` `Enum`

  A generic enumeration class. Inspired by: http://stackoverflow.com/questions/36932/whats-the-best-way-to-implement-an-enum-in-python/2182437#2182437 with some more syntactic sugar added.

  An Enum class must be instanciated with a list of strings, that make the enumeration “label”:

      >>> Animal = Enum('CAT', 'DOG')

  Each label is available as an instance attribute, evaluating to itself:

      >>> Animal.DOG
      'DOG'

      >>> Animal.CAT == 'CAT'
      True

  As a consequence, you can test for presence of an enumeration label by string value:

      >>> 'DOG' in Animal
      True

  Finally, enumeration labels can also be iterated upon:

      >>> for a in Animal: print a
      DOG
      CAT

class `gc3libs.utils.` `Log`

  A list of messages with timestamps and (optional) tags.

  The append method should be used to add a message to the Log:

      >>> L = Log()
      >>> L.append('first message')
      >>> L.append('second one')

  The last method returns the text of the last message appended:

      >>> L.last()
      'second one'

  Iterating over a Log instance returns message texts in the temporal order they were added to the list:

      >>> for msg in L: print(msg)
      first message
      second one

  `append`(message, *tags)

    Append a message to this Log.

    The message is timestamped with the time at the moment of the call.

    The optional tags argument is a sequence of strings. Tags are recorded together with the message and may be used to filter log messages given a set of labels. (This feature is not yet implemented.)

  `last`()
    Return text of last message appended. If log is empty, return empty string.

class `gc3libs.utils.` `PlusInfinity`

  An object that is greater-than any other object.

  >     >>> x = PlusInfinity()

  >     >>> x > 1
  >     True
  >     >>> 1 < x
  >     True
  >     >>> 1245632479102509834570124871023487235987634518745 < x
  >     True

  >     >>> x > sys.maxint
  >     True
  >     >>> x < sys.maxint
  >     False
  >     >>> sys.maxint < x
  >     True

  PlusInfinity objects are actually larger than any given Python object:

      >>> x > 'azz'
      True
      >>> x > object()
      True

  Note that PlusInfinity is a singleton, therefore you always get the same instance when calling the class constructor:

      >>> x = PlusInfinity()
      >>> y = PlusInfinity()
      >>> x is y
      True

  Relational operators try to return the correct value when comparing PlusInfinity to itself:

      >>> x < y
      False
      >>> x <= y
      True
      >>> x == y
      True
      >>> x >= y
      True
      >>> x > y
      False

class `gc3libs.utils.` `Singleton`

  Derived classes of Singleton can have only one instance in the running Python interpreter.

      >>> x = Singleton()
      >>> y = Singleton()
      >>> x is y
      True

class `gc3libs.utils.` `Struct`(initializer=None, **kw)

  A dict-like object, whose keys can be accessed with the usual ‘[...]’ lookup syntax, or with the ‘.’ get attribute syntax.

  Examples:

      >>> a = Struct()
      >>> a['x'] = 1
      >>> a.x
      1
      >>> a.y = 2
      >>> a['y']
      2

  Values can also be initially set by specifying them as keyword arguments to the constructor:

      >>> a = Struct(z=3)
      >>> a['z']
      3
      >>> a.z
      3

class `gc3libs.utils.` `defaultdict`(default_factory)
  A backport of defaultdict to Python 2.4 See http://docs.python.org/library/collections.html

`gc3libs.utils.` `defproperty`(fn)
  Decorator to define properties with a simplified syntax in Python 2.4. See http://code.activestate.com/recipes/410698-property-decorator-for-python-24/#c6 for details and examples.

`gc3libs.utils.` `deploy_configuration_file`(filename, template_filename=None)

  Ensure that configuration file filename exists; possibly copying it from the specified template_filename.

  Return True if a file with the specified name exists in the configuration directory. If not, try to copy the template file over and then return False; in case the copy operations fails, a NoConfigurationFile exception is raised.

  If parameter filename is not an absolute path, it is interpreted as relative to gc3libs.Default.RCDIR; if template_filename is None, then it is assumed to be the same as filename.

`gc3libs.utils.` `dirname`(pathname)
  Same as os.path.dirname but return . in case of path names with no directory component.

`gc3libs.utils.` `first`(seq)

  Return the first element of sequence or iterator seq. Raise TypeError if the argument does not implement either of the two interfaces.

  Examples:

      >>> s = [0, 1, 2]
      >>> first(s)
      0

      >>> s = {'a':1, 'b':2, 'c':3}
      >>> first(sorted(s.keys()))
      'a'

`gc3libs.utils.` `from_template`(template, **kw)

  Return the contents of template, substituting all occurrences of Python formatting directives ‘%(key)s’ with the corresponding values taken from dictionary kw.

  If template is an object providing a read() method, that is used to gather the template contents; else, if a file named template exists, the template contents are read from it; otherwise, template is treated like a string providing the template contents itself.

`gc3libs.utils.` `get_and_remove`(dictionary, key, default=None)
  Return the value associated to key in dictionary, or default if there is no such key. Remove key from dictionary afterwards.

`gc3libs.utils.` `mkdir_with_backup`(path)
  Like os.makedirs, but if path already exists, rename the existing one appending a .NUMBER suffix.

`gc3libs.utils.` `progressive_number`(qty=None)

  Return a positive integer, whose value is guaranteed to be monotonically increasing across different invocations of this function, and also across separate instances of the calling program.

  Example:

      >>> n = progressive_number()
      >>> m = progressive_number()
      >>> m > n
      True

  If you specify a positive integer as argument, then a list of monotonically increasing numbers is returned. For example:

      >>> ls = progressive_number(5)
      >>> len(ls)
      5

  In other words, progressive_number(N) is equivalent to:

      nums = [ progressive_number() for n in range(N) ]

  only more efficient, because it has to obtain and release the lock only once.

  After every invocation of this function, the last returned number is stored into the file `~/.gc3/next_id.txt`.

  Note: as file-level locking is used to serialize access to the counter file, this function may block (default timeout: 30 seconds) while trying to acquire the lock, or raise a LockTimeout exception if this fails.

  @raise LockTimeout

  @return A positive integer number, monotonically increasing with every call.
    A list of such numbers if argument qty is a positive integer.

`gc3libs.utils.` `safe_repr`(obj)
  Return a string describing Python object obj. Avoids calling any Python magic methods, so should be safe to use as a “last resort” in implementation of __str__ and __repr__ magic.

`gc3libs.utils.` `same_docstring_as`(referenced_fn)

  Function decorator: sets the docstring of the following function to the one of referenced_fn.

  Intended usage is for setting docstrings on methods redefined in derived classes, so that they inherit the docstring from the corresponding abstract method in the base class.

`gc3libs.utils.` `string_to_boolean`(word)
  Convert word to a Python boolean value and return it. The strings true, yes, on, 1 (with any capitalization and any amount of leading and trailing spaces) are recognized as meaning Python True. Any other word is considered as boolean False.

`gc3libs.utils.` `to_bytes`(s)

  Convert string s to an integer number of bytes. Suffixes like ‘KB’, ‘MB’, ‘GB’ (up to ‘YB’), with or without the trailing ‘B’, are allowed and properly accounted for. Case is ignored in suffixes.

  Examples:

      >>> to_bytes('12')
      12
      >>> to_bytes('12B')
      12
      >>> to_bytes('12KB')
      12000
      >>> to_bytes('1G')
      1000000000

  Binary units ‘KiB’, ‘MiB’ etc. are also accepted:

      >>> to_bytes('1KiB')
      1024
      >>> to_bytes('1MiB')
      1048576

## gc3libs.Resource

Warning

This module is deprecated and will be removed in a future release. Do not depend on it.

A specialized dictionary for representing computational resource characteristics.

class `gc3libs.Resource.` `Resource`(initializer=None, **keywd)

  Resource objects are dictionaries, comprised of the following keys.

  Statically provided, i.e., specified at construction time and changed never after:

  > arc_ldap string auth string frontend string gamess_location string max_cores_per_job int * max_memory_per_core int * max_walltime int * name string * ncores int type int *

  Starred attributes are required for object construction.

  Dynamically provided (i.e., defined by the get_resource_status() method or similar):
    free_slots int user_run int user_queued int queued int

## gc3libs.scheduler

Simple-minded scheduling for GC3Libs.

## gc3libs.InformationContainer

Warning

This module is deprecated and will be removed in a future release. Do not depend on it.

A specialized dict class.

# GC3Utils modules

## gc3utils

## gc3utils.gcmd

gc3utils - A simple command-line frontend to distributed resources

This is a generic front-end code; actual implementation of commands can be found in gc3utils/commands.py

`gc3utils.gcmd.` `main`()
  Generic front-end function to invoke the commands in gc3utils/gcommands.py

## gc3utils.commands

Implementation of the core command-line front-ends.

`gc3utils.commands.` `gclean`(*args, **kw)
  The ‘glean’ command. gclean takes a list of jobids and tries to clean each of them. if any of the clean requests will fail, gclean will exit with exitcode 1 gclean will try anyway to process all requests

`gc3utils.commands.` `gget`(*args, **kw)
  The gget command.

`gc3utils.commands.` `ginfo`(*args, **kw)
  The ‘ginfo’ command.

`gc3utils.commands.` `gkill`(*args, **kw)
  The gkill command. gkill takes a list of jobids and tries to kill each of them. if any of the clean requests will fail, gkill will exit with exitcode 1 gkill will try anyway to process all requests

`gc3utils.commands.` `glist`(*args, **kw)
  The glist command.

`gc3utils.commands.` `gnotify`(*args, **kw)
  The gnotify command

`gc3utils.commands.` `gresub`(*args, **kw)
  The gresub command: resubmit an already-submitted job with different parameters.

`gc3utils.commands.` `gstat`(*args, **kw)
  The gstat command.

`gc3utils.commands.` `gsub`(*args, **kw)
  The gsub command.

`gc3utils.commands.` `gtail`(*args, **kw)
  The ‘gtail’ command.

### Navigation

- index
- modules |
- previous |
- gc3pie v1.0rc4 documentation »