stormtracks Running in Cluster Mode

stormtracks.pyro_cluster.pyro_nameserver – Runs a Pyro4 Nameserver

stormtracks.pyro_cluster.pyro_nameserver.main()

Launches a Pyro4 nameserver

stormtracks.pyro_cluster.pyro_starter – Launches All pyro_workers

stormtracks.pyro_cluster.pyro_starter.main()

Starts all pyro_workers defined in pyro_settings.pyro_workers

ssh’s into each computer and issues a command to start pyro_worker

stormtracks.pyro_cluster.pyro_manager – Manages All pyro_workers

stormtracks.pyro_cluster.pyro_manager.main(task_name, years, num_ensemble_members=56, use_best_config=True, compress=True)

Established communication with all pyro_workers

N.B. pyro_nameserver must be set up first, and pyro workers must be running on each of the servers defined by pyro_settings.worker servers (either by running pyro_start.py or by manually starting them).

Starts off by finding each of the pyro_workers, then generates a task schedule before handing over to run_tasks(...).

stormtracks.pyro_cluster.pyro_manager.run_tasks(year, task_provider, workers, free_workers, task_name)

Takes the given task_provider and list of workers and farms out the various tasks.

stormtracks.pyro_cluster.pyro_ender – Stops All pyro_workers

stormtracks.pyro_cluster.pyro_ender.main()

Kills all pyro_workers defined in pyro_settings.pyro_workers

ssh’s into each computer and issues a command to kill pyro_worker

stormtracks.pyro_cluster.pyro_worker – Processes Data

class stormtracks.pyro_cluster.pyro_worker.PyroWorker

Runs on each of the worker servers

Listens for do_work requests by pyro_manager, on receiving of a test does the required work and saves it to disk using a results manager.

do_work(year, ensemble_member, task_name, task_data)

Do the work required by pyro_manager

Parameters:

• year – year to analyse
• ensemble_member – ensemble_member to analyse
• task – task to do (currently must be ‘vort_track’)

Returns:

dict containing details of task

stormtracks.pyro_cluster.pyro_worker.main()

Sets up a PyroWorker and starts its event loop to wait for calls from a pyro_manager

stormtracks.pyro_cluster.pyro_task – Helps Keep Track of Outstanding Tasks

class stormtracks.pyro_cluster.pyro_task.PyroFieldCollectionAnalysis(year, num_ensemble_members=56)

Keeps track of all field collection analysis tasks to be done, and can issue the next outstanding task

Parameters:year – year to analyse

get_next_outstanding()

Returns the next outstanding task, None if there are no more

get_progress()

Returns a string representing the progress

print_progress()

Prints the current progress

class stormtracks.pyro_cluster.pyro_task.PyroTask(year, ensemble_member, name, data=None)

Represents task that is to be done

Parameters:

• year – year of task
• ensemble_member – ensemble_member of task
• name – name of the task to be done

name

What name this is doing

status

Current status of task, must be in STATUSES

class stormtracks.pyro_cluster.pyro_task.PyroTrackingAnalysis(year, num_ensemble_members=56, config=None)

Keeps track of all tracking analysis tasks to be done, and can issue the next outstanding class

Parameters:year – year to analyse

get_next_outstanding()

Returns the next outstanding task, None if there are no more

get_progress()

Returns a string representing the progress

print_progress()

Prints the current progress

class stormtracks.pyro_cluster.pyro_task.PyroVortTracking(start_year=2000, end_year=2012, num_ensemble_members=56)

Keeps track of all vort tracking tasks to be done, and can issue the next outstanding class

Parameters:

• start_year – year from which to start tasks
• end_year – year from which to end tasks (inclusive)
• num_ensemble_members – how many ensemble members to keep tasks for

get_next_outstanding()

Returns the next outstanding task, None if there are no more

get_progress(years=None, include_year=False)

Returns a string representing the progress for all years

get_progress_for_year(year, include_year=True)

Returns a string representing the progress of the year

• - :task to be done
• W :task being worked on
• + :task complete
• F :task failure
• T :task timeout

print_years(years=None, include_year=True)

Prints progress