stormtracks Running in Cluster Mode — stormtracks 0.4.1 documentation

### Navigation

- index
- modules |
- next |
- previous |
- stormtracks 0.4.1 documentation »
- stormtracks API »

# stormtracks Running in Cluster Mode

## `stormtracks.pyro_cluster.pyro_nameserver` – Runs a Pyro4 Nameserver

`stormtracks.pyro_cluster.pyro_nameserver.` `main`()

  Launches a Pyro4 nameserver

## `stormtracks.pyro_cluster.pyro_starter` – Launches All pyro_workers

`stormtracks.pyro_cluster.pyro_starter.` `main`()

  Starts all pyro_workers defined in pyro_settings.pyro_workers

  ssh’s into each computer and issues a command to start pyro_worker

## `stormtracks.pyro_cluster.pyro_manager` – Manages All pyro_workers

`stormtracks.pyro_cluster.pyro_manager.` `main`(task_name, years, num_ensemble_members=56, use_best_config=True, compress=True)

  Established communication with all pyro_workers

  N.B. pyro_nameserver must be set up first, and pyro workers must be running on each of the servers defined by pyro_settings.worker servers (either by running pyro_start.py or by manually starting them).

  Starts off by finding each of the pyro_workers, then generates a task schedule before handing over to run_tasks(...).

`stormtracks.pyro_cluster.pyro_manager.` `run_tasks`(year, task_provider, workers, free_workers, task_name)

  Takes the given task_provider and list of workers and farms out the various tasks.

## `stormtracks.pyro_cluster.pyro_ender` – Stops All pyro_workers

`stormtracks.pyro_cluster.pyro_ender.` `main`()

  Kills all pyro_workers defined in pyro_settings.pyro_workers

  ssh’s into each computer and issues a command to kill pyro_worker

## `stormtracks.pyro_cluster.pyro_worker` – Processes Data

class `stormtracks.pyro_cluster.pyro_worker.` `PyroWorker`

  Runs on each of the worker servers

  Listens for do_work requests by pyro_manager, on receiving of a test does the required work and saves it to disk using a results manager.

  `do_work`(year, ensemble_member, task_name, task_data)

    Do the work required by pyro_manager

    Parameters: | year – year to analyse ensemble_member – ensemble_member to analyse task – task to do (currently must be ‘vort_track’)
    Returns: | dict containing details of task

`stormtracks.pyro_cluster.pyro_worker.` `main`()

  Sets up a PyroWorker and starts its event loop to wait for calls from a pyro_manager

## `stormtracks.pyro_cluster.pyro_task` – Helps Keep Track of Outstanding Tasks

class `stormtracks.pyro_cluster.pyro_task.` `PyroFieldCollectionAnalysis`(year, num_ensemble_members=56)

  Keeps track of all field collection analysis tasks to be done, and can issue the next outstanding task

  Parameters: | year – year to analyse

  `get_next_outstanding`()

    Returns the next outstanding task, None if there are no more

  `get_progress`()

    Returns a string representing the progress

  `print_progress`()

    Prints the current progress

class `stormtracks.pyro_cluster.pyro_task.` `PyroTask`(year, ensemble_member, name, data=None)

  Represents task that is to be done

  Parameters: | year – year of task ensemble_member – ensemble_member of task name – name of the task to be done

  `name`

    What name this is doing

  `status`

    Current status of task, must be in STATUSES

class `stormtracks.pyro_cluster.pyro_task.` `PyroTrackingAnalysis`(year, num_ensemble_members=56, config=None)

  Keeps track of all tracking analysis tasks to be done, and can issue the next outstanding class

  Parameters: | year – year to analyse

  `get_next_outstanding`()

    Returns the next outstanding task, None if there are no more

  `get_progress`()

    Returns a string representing the progress

  `print_progress`()

    Prints the current progress

class `stormtracks.pyro_cluster.pyro_task.` `PyroVortTracking`(start_year=2000, end_year=2012, num_ensemble_members=56)

  Keeps track of all vort tracking tasks to be done, and can issue the next outstanding class

  Parameters: | start_year – year from which to start tasks end_year – year from which to end tasks (inclusive) num_ensemble_members – how many ensemble members to keep tasks for

  `get_next_outstanding`()

    Returns the next outstanding task, None if there are no more

  `get_progress`(years=None, include_year=False)

    Returns a string representing the progress for all years

  `get_progress_for_year`(year, include_year=True)

    Returns a string representing the progress of the year

    - `-` :task to be done
    - `W` :task being worked on
    - `+` :task complete
    - `F` :task failure
    - `T` :task timeout
  `print_years`(years=None, include_year=True)

    Prints progress

### Navigation

- index
- modules |
- next |
- previous |
- stormtracks 0.4.1 documentation »
- stormtracks API »