fireworks.utilities package — FireWorks 1.4.4 documentation

# fireworks.utilities package

## Subpackages

- fireworks.utilities.tests package
  - Submodules
  - fireworks.utilities.tests.test_filepad module
  - fireworks.utilities.tests.test_fw_serializers module
  - fireworks.utilities.tests.test_update_collection module
  - Module contents

## Submodules

## fireworks.utilities.dict_mods module

`fireworks.utilities.dict_mods.` `apply_mod`(modification, obj)

  Note that modify makes actual in-place modifications. It does not return a copy.

  Parameters: | modification – Modification must be {action_keyword : settings}, where action_keyword is a supported DictMod obj – A dict to be modified

`fireworks.utilities.dict_mods.` `get_nested_dict`(input_dict, key)

## fireworks.utilities.filepad module

class `fireworks.utilities.filepad.` `FilePad`(host=u'localhost', port=27017, database=u'fireworks', username=None, password=None, filepad_coll_name=u'filepad', gridfs_coll_name=u'filepad_gfs', logdir=None, strm_lvl=None)

  Bases: `monty.json.MSONable`

  `add_file`(path, identifier=None, compress=True, metadata=None)

    Insert the file specified by the path into gridfs. The gridfs id and identifier are returned. Note: identifier must be unique, i.e, no insertion if the identifier already exists in the db.

    Parameters: | path (str) – path to the file identifier (str) – file identifier. If identifier = None then the identifier is set to the object id returned by gridfs insertion. compress (bool) – compress or not metadata (dict) – file metadata
    Returns: | the id returned by gridfs, identifier
    Return type: | (str, str)

  classmethod `auto_load`()

    Returns FilePad object

  `build_indexes`(indexes=None, background=True)

    Build the indexes.

    Parameters: | indexes (list) – list of single field indexes to be built. background (bool) – Run in the background or not.

  `count`(filter=None, **kwargs)

    Get the number of documents in filepad.

    Parameters: | filter (dict) – kwargs (dict) – see pymongo.Collection.count for the supported keyword arguments.
    Returns: | the count
    Return type: | int

  `delete_file`(identifier)

    Delete the document with the matching identifier. The contents in the gridfs as well as the associated document in the filepad are deleted.

    Parameters: | identifier (str) – the file identifier

  `delete_file_by_id`(gfs_id)

    Parameters: | gfs_id (str) – the file id

  `delete_file_by_query`(query)

    Parameters: | query (dict) – pymongo query dict

  classmethod `from_db_file`(db_file, admin=True)

    Parameters: | db_file (str) – path to the filepad cred file
    Returns: | FilePad object

  `get_file`(identifier)

    Get file by identifier

    Parameters: | identifier (str) – the file identifier
    Returns: | the file content as a string, document dictionary
    Return type: | (str, dict)

  `get_file_by_id`(gfs_id)

    Parameters: | gfs_id (str) – the gridfs id for the file.
    Returns: | the file content as a string, document dictionary
    Return type: | (str, dict)

  `get_file_by_query`(query)

    Parameters: | query (dict) – pymongo query dict
    Returns: | list of all (file content as a string, document dictionary)
    Return type: | list

  `reset`()

    Reset filepad and the gridfs collections

  `update_file`(identifier, path, compress=True)

    Update the filecontents in the gridfs, update the gfs_id in the document and retain the rest.

    Parameters: | identifier (str) – the unique file identifier path (str) – path to the new file whose contents will replace the existing one. compress (bool) – whether or not to compress the contents before inserting to gridfs
    Returns: | old file id , new file id
    Return type: | (str, str)

  `update_file_by_id`(gfs_id, path, compress=True)

    Update the file in the gridfs with the given id and retain the rest of the document.

    Parameters: | gfs_id (str) – the gridfs id for the file. path (str) – path to the new file whose contents will replace the existing one. compress (bool) – whether or not to compress the contents before inserting to gridfs
    Returns: | old file id , new file id
    Return type: | (str, str)

## fireworks.utilities.fw_serializers module

`fireworks.utilities.fw_serializers.` `DATETIME_HANDLER`(obj)

class `fireworks.utilities.fw_serializers.` `FWSerializable`

  Bases: `object`

  To create a serializable object within FireWorks, you should subclass this class and implement the to_dict() and from_dict() methods.

  If you want the load_object() implicit de-serialization to work, you must also:

  > - Use the @serialize_fw decorator on to_dict()
  > - Override the _fw_name parameter with a unique key.

  See documentation of load_object() for more details.

  The use of @classmethod for from_dict allows you to define a superclass that implements the to_dict() and from_dict() for all its subclasses.

  For an example of serialization, see the class QueueAdapterBase.

  `as_dict`()
  classmethod `from_dict`(m_dict)
  classmethod `from_file`(filename, f_format=None)

    Load a serialization of this object from a file.

    Parameters: | filename (str) – filename to read f_format (str) – serialization format, default checks the filename extension
    Returns: | FWSerializable

  classmethod `from_format`(f_str, f_format=u'json')

    convert from a String representation to its Object.

    Parameters: | f_str (str) – the String representation f_format (str) – serialization format of the String (default json)
    Returns: | FWSerializable

  `fw_name`
  `to_db_dict`()
  `to_dict`()
  `to_file`(filename, f_format=None, **kwargs)

    Write a serialization of this object to a file.

    Parameters: | filename (str) – filename to write to f_format (str) – serialization format, default checks the filename extension

  `to_format`(f_format=u'json', **kwargs)

    returns a String representation in the given format

    Parameters: | f_format (str) – the format to output to (default json)

`fireworks.utilities.fw_serializers.` `get_default_serialization`(cls)

`fireworks.utilities.fw_serializers.` `load_object`(obj_dict)

  Creates an instantiation of a class based on a dictionary representation. We implicitly determine the Class through introspection along with information in the dictionary.

  We search for a class with the _fw_name property equal to obj_dict[‘_fw_name’] If the @module key is set, that module is checked first for a matching class to improve speed of lookup. Afterwards, the modules in the USER_PACKAGES global parameter are checked.

  Refactoring class names, module names, etc. will not break object loading as long as:

  1. the _fw_name property is maintained the same AND
  2. the refactored module is kept within USER_PACKAGES

  You can get around these limitations if you really want: i) If you want to change the fw_name of an object you can set the FW_NAME_UPDATES key ii) If you want to put a refactored module in a new place add an entry to USER_PACKAGES

  Parameters: | obj_dict (dict) – the dict representation of the class

`fireworks.utilities.fw_serializers.` `load_object_from_file`(filename, f_format=None)

  Implicitly load an object from a file. just a friendly wrapper to load_object()

  Parameters: | filename (str) – the filename to load an object from f_format (str) – the serialization format (default is auto-detect based on filename extension)

`fireworks.utilities.fw_serializers.` `reconstitute_dates`(obj_dict)

`fireworks.utilities.fw_serializers.` `recursive_deserialize`(func)

  a decorator to add FW serializations keys see documentation of FWSerializable for more details

`fireworks.utilities.fw_serializers.` `recursive_dict`(obj, preserve_unicode=True)

`fireworks.utilities.fw_serializers.` `recursive_serialize`(func)

  a decorator to add FW serializations keys see documentation of FWSerializable for more details

`fireworks.utilities.fw_serializers.` `serialize_fw`(func)

  a decorator to add FW serializations keys see documentation of FWSerializable for more details

## fireworks.utilities.fw_utilities module

class `fireworks.utilities.fw_utilities.` `DataServer`(address=None, authkey=None, serializer='pickle')

  Bases: `multiprocessing.managers.BaseManager`

  Provide a server that can host shared objects between multiprocessing Processes (that normally can’t share data). For example, a common LaunchPad is shared between processes and pinging launches is coordinated to limit DB hits.

  classmethod `setup`(launchpad)

    Parameters: | launchpad (LaunchPad) –
    Returns: | DataServer

class `fireworks.utilities.fw_utilities.` `NestedClassGetter`

  Bases: `object`

  Used to help pickle inner classes, e.g. see Workflow.Links When called with the containing class as the first argument, and the name of the nested class as the second argument, returns an instance of the nested class.

`fireworks.utilities.fw_utilities.` `create_datestamp_dir`(root_dir, l_logger, prefix='block_')

  Internal method to create a new block or launcher directory. The dir name is based on the time and the FW_BLOCK_FORMAT

  Parameters: | root_dir – directory to create the new dir in l_logger – the logger to use prefix – the prefix for the new dir, default=”block_“

`fireworks.utilities.fw_utilities.` `explicit_serialize`(o)

`fireworks.utilities.fw_utilities.` `get_fw_logger`(name, l_dir=None, file_levels=('DEBUG', 'ERROR'), stream_level='DEBUG', formatter=<logging.Formatter object>, clear_logs=False)

  Convenience method to return a logger.

  Parameters: | name – name of the logger that sets the groups, e.g. ‘group1.set2’ l_dir – the directory to put the log file file_levels – iterable describing level(s) to log to file(s). default: (‘DEBUG’, ‘ERROR’) stream_level – level to log to standard output. default: ‘DEBUG’ formatter – logging format. default: FW_LOGGING_FORMATTER clear_logs – whether to clear the logger with the same name

`fireworks.utilities.fw_utilities.` `get_my_host`()

`fireworks.utilities.fw_utilities.` `get_my_ip`()

`fireworks.utilities.fw_utilities.` `get_slug`(m_str)

`fireworks.utilities.fw_utilities.` `log_exception`(m_logger, msgs)

  A shortcut wrapper around log_fancy for exceptions

  Parameters: | m_logger (logger) – The logger object msgs ([str]) – String or iterable of Strings, will be joined by newlines

`fireworks.utilities.fw_utilities.` `log_fancy`(m_logger, msgs, log_lvl='info', add_traceback=False)

  A wrapper around the logger messages useful for multi-line logs. Helps to group log messages by adding a fancy border around it, which enhances readability of log lines meant to be read as a unit.

  Parameters: | m_logger (logger) – The logger object log_lvl (str) – The level to log at msgs ([str]) – a String or iterable of Strings add_traceback (bool) – add traceback text, useful when logging exceptions (default False)

`fireworks.utilities.fw_utilities.` `log_multi`(m_logger, msg, log_lvl='info')

  Parameters: | m_logger (logger) – The logger object msg (str) – a String to log log_lvl (str) – The level to log at

`fireworks.utilities.fw_utilities.` `plot_wf`(wf, depth_factor=1.0, breadth_factor=2.0, labels_on=True, numerical_label=False, text_loc_factor=1.0, save_as=None, style='rD--', markersize=10, markerfacecolor='blue', fontsize=12)

  Generate a visual representation of the workflow. Useful for checking whether the firework connections are in order before launching the workflow.

  Parameters: | wf (Workflow) – workflow object. depth_factor (float) – adjust this to stretch the plot in y direction. breadth_factor (float) – adjust this to stretch the plot in x direction. labels_on (bool) – whether to label the nodes or not. The default is to lable the nodes using the firework names. numerical_label (bool) – set this to label the nodes using the firework ids. text_loc_factor (float) – adjust the label location. save_as (str) – save the figure to the given name. style (str) – marker style. markersize (int) – marker size. markerfacecolor (str) – marker face color. fontsize (int) – font size for the node label.

`fireworks.utilities.fw_utilities.` `redirect_local`(*args, **kwds)

  temporarily redirect stdout or stderr to fws.error and fws.out

## fireworks.utilities.timing module

class `fireworks.utilities.timing.` `NullTimer`

  Bases: `fireworks.utilities.timing.Timer`

  Support performance timer interface, but do absolutely nothing. This is useful to avoid many tiresome if/else blocks.

  `block`(stage)
  `start`(stage)
  `stop`(stage)
  `write`(**kwargs)

class `fireworks.utilities.timing.` `Timer`(name)

  Bases: `object`

  Simple performance timer.

  Usage:

    p = Timer(“myname”) for thing in all_things:

    > p.start(“stage1”) do_stage_1() p.stop(“stage1”) # alt. ‘with’ interface with p.block(“stage2”):

    > > do_something_else()

  Limitations: - Instances are not thread-safe. - The set_ns() class method is not thread-safe. - The ‘with’ block() cannot be nested,

  > instead use different stages with begin()/end() pairs.

  - The only output format is CSV.
  - There is no (easy) programmatic way to get the results.
  `block`(stage)
  classmethod `set_ns`(val)

    Set a namespace (prefix) for all timers. In output, the namespace will be separated by the timer name by a ”.”

  `start`(stage=u'null')

    Begin timing.

  `stop`(stage=u'null')

    Stop timing.

  `stop_all`()

    Stop all timers. Idempotent.

  `write`(stream=<open file '<stdout>', mode 'w'>)

    Write results (CSV) to a stream.

`fireworks.utilities.timing.` `any_fw_timers`()

  Whether any timers are enabled and non-empty

  Returns: | True if so, False if not
  Return type: | bool

`fireworks.utilities.timing.` `enable_fw_timer`(name, is_enabled)

  Enable or disable a timer.

  Parameters: | name (str) – Timer’s name, or glob-style name pattern is_enabled (bool) – Whether to enable (True) or disable (False)

`fireworks.utilities.timing.` `get_fw_timer`(name)

  G et timer, possibly a NullTimer, for a section of code.

  If the user did not enable timers for this section, will return a NullTimer. Otherwise will return a Timer.

  Usage:
    timer = get_fw_timer(“StarWars”) timer.start(“jumpToLightSpeed”) jumpToLightSpeed() timer.stop(“jumpToLightSpeed”) ... print_fw_timers() # prints results of all timers

  Parameters: | name (str) – Name of a given timer.
  Returns: | A timer instance
  Return type: | Timer

`fireworks.utilities.timing.` `print_fw_timers`(stream=<open file '<stdout>', mode 'w'>)

  Print results of all timers to the provided stream.

  Parameters: | stream – Output stream, only needs to support ‘write’
  Returns: | number of items (data rows) printed
  Return type: | int

`fireworks.utilities.timing.` `timer_env_var` = u'FW_TIMERS'

  Environment variable in which to list the enabled timers Use comma-separated strings, e.g.:

## fireworks.utilities.update_collection module

`fireworks.utilities.update_collection.` `update_launchpad_data`(lp, replacements, **kwargs)

  If you want to update a text string in your entire FireWorks database with a replacement, use this method. For example, you might want to update a directory name preamble like “/scratch/user1” to “/project/user2”. The algorithm does a text replacement over the entire BSON document. The original collection is backed up within the database with extension “_xiv_{Date}”.

  Parameters: | (LaunchPad) (lp) – a FireWorks LaunchPad object (dict) (replacements) – e.g. {“old_path1”: “new_path1”, “scratch/”:”project/”} kwargs – Additional arguments accepted by the update_path_in_collection method

`fireworks.utilities.update_collection.` `update_path_in_collection`(db, collection_name, replacements, query=None, dry_run=False, force_clear=False)

  updates the text specified in replacements for the documents in a MongoDB collection. This can be used to mass-update an outdated value (e.g., a directory path or tag) in that collection. The algorithm does a text replacement over the entire BSON document. The original collection is backed up within the database with extension “_xiv_{Date}”.

  Parameters: | db (Database) – MongoDB db object collection_name (str) – name of a MongoDB collection to update replacements (dict) – e.g. {“old_path1”: “new_path1”, “scratch/”:”project/”} query (dict) – criteria for query, default None if you want all documents to be updated dry_run (bool) – if True, only a new collection with new paths is created and original “collection” is not replaced force_clear (bool) – careful! If True, the collection “{}_temp_refactor”.format(collection) is removed!
  Returns: | None, but if dry_run==False it replaces the collection with the updated one

## Module contents