starcluster.cluster.Cluster

Package starcluster :: Module cluster :: Class Cluster |
[hide private]
[frames] | no frames]

# Class Cluster

source code

Instance Methods | [hide private]
__init__(self, ec2_conn=None, spot_bid=None, cluster_tag=None, cluster_description=None, cluster_size=None, cluster_user=None, cluster_shell=None, master_image_id=None, master_instance_type=None, node_image_id=None, node_instance_type=None, node_instance_types=`[``]`, availability_zone=None, keyname=None, key_location=None, volumes=`[``]`, plugins=`[``]`, permissions=`[``]`, refresh_interval=30, disable_queue=False, disable_threads=False, cluster_group=None, **kwargs) x.__init__(...) initializes x; see x.__class__.__doc__ for signature | source code
__repr__(self) repr(x) | source code
load_volumes(self, vols) Iterate through vols and set device/partition settings automatically if not specified. | source code
load_plugins(self, plugins) | source code
update(self, kwargs) | source code
_validate_running_instances(self) Validate existing instances against this template's settings | source code
get(self, name) | source code
__str__(self) str(x) | source code
load_receipt(self, load_plugins=True) Load the original settings used to launch this cluster into this Cluster object. | source code
__getstate__(self) | source code
get_node_by_dns_name(self, dns_name) | source code
get_node_by_id(self, instance_id) | source code
get_node_by_alias(self, alias) | source code
_nodes_in_states(self, states) | source code
create_node(self, alias, image_id=None, instance_type=None, zone=None, placement_group=None) | source code
create_nodes(self, aliases, image_id=None, instance_type=None, count=1, zone=None, placement_group=None) Convenience method for requesting instances with this cluster's settings | source code
_get_next_node_num(self) | source code
add_node(self, alias=None) Add a single node to this cluster | source code
add_nodes(self, num_nodes, aliases=None) Add new nodes to this cluster | source code
remove_node(self, node) Remove a single node from this cluster | source code
remove_nodes(self, nodes) Remove a list of nodes from this cluster | source code
_get_launch_map(self) Groups all node-aliases that have similar instance types/image ids Returns a dictionary that's used to launch all similar instance types and image ids in the same request. | source code
_get_type_and_image_id(self, alias) Returns (instance_type,image_id) for a given alias based on the map returned from self._get_launch_map | source code
create_cluster(self) Launches all EC2 instances based on this cluster's settings. | source code
_create_flat_rate_cluster(self) Launches cluster using flat-rate instances. | source code
_create_spot_cluster(self) Launches cluster using all spot instances. | source code
is_ebs_cluster(self) Returns true if any instances in the cluster are EBS-backed | source code
is_cluster_compute(self) Returns true if any instances are a Cluster Compute type | source code
is_cluster_up(self) Check that all nodes are 'running' and that ssh is up on all nodes This method will return False if any spot requests are in an 'open' state. | source code
get_spinner(self, msg) Logs a status msg, starts a spinner, and returns the spinner object. | source code
wait_for_cluster(self, msg=`'``Waiting for cluster to come up...``'`) Wait for cluster to come up and display progress bar. | source code
is_cluster_stopped(self) Check whether all nodes are in the 'stopped' state | source code
is_cluster_terminated(self) Check whether all nodes are in a 'terminated' state | source code
attach_volumes_to_master(self) Attach each volume to the master node | source code
detach_volumes(self) Detach all volumes from all nodes | source code
restart_cluster(*arg, **kargs) Raw timing function | source code
stop_cluster(self) Stop this cluster by detaching all volumes, stopping/terminating all instances, cancelling all spot requests (if any), and removing this cluster's security group. | source code
terminate_cluster(self) Stop this cluster by first detaching all volumes, shutting down all instances, cancelling all spot requests (if any), removing this cluster's placement group (if any), and removing this cluster's security group. | source code
start(self, create=True, create_only=False, validate=True, validate_only=False, validate_running=False) Handles creating and configuring a cluster. | source code
_start(*arg, **kargs) Raw timing function | source code
_setup_cluster(self) This method waits for all nodes to come up and then runs the default StarCluster setup routines followed by any additional plugin setup routines | source code
run_plugins(self, plugins=None, method_name=`'``run``'`, node=None, reverse=False) Run all plugins specified in this Cluster object's self.plugins list Uses plugins list instead of self.plugins if specified. | source code
run_plugin(self, plugin, name=`'````'`, method_name=`'``run``'`, node=None) Run a StarCluster plugin. | source code
is_running_valid(self) Checks whether the current running instances are compatible with this cluster template's settings | source code
_validate(self, validate_running=False) Checks that all cluster template settings are valid. | source code
is_valid(self) Returns True if all cluster template settings are valid | source code
_validate_spot_bid(self) | source code
_validate_cluster_size(self) | source code
_validate_shell_setting(self) | source code
_validate_image_settings(self) | source code
_validate_zone(self) | source code
__check_platform(self, image_id, instance_type) Validates whether an image_id (AMI) is compatible with a given instance_type. | source code
_validate_instance_types(self) | source code
_validate_cluster_compute(self) | source code
_validate_ebs_aws_settings(self) Verify EBS volumes exists on Amazon and that each volume's zone matches this cluster's zone setting. | source code
_validate_permission_settings(self) | source code
_validate_ebs_settings(self) Check EBS vols for missing/duplicate DEVICE/PARTITION/MOUNT_PATHs and validate these settings. | source code
_has_all_required_settings(self) | source code
_validate_credentials(self) | source code
_validate_keypair(self) | source code
ssh_to_master(self, user=`'``root``'`) | source code
ssh_to_node(self, alias, user=`'``root``'`) | source code
Inherited from `object`: `__delattr__`, `__format__`, `__getattribute__`, `__hash__`, `__new__`, `__reduce__`, `__reduce_ex__`, `__setattr__`, `__sizeof__`, `__subclasshook__`

Properties | [hide private]
| zone If volumes are specified, this method determines the common availability zone between those volumes.
| _security_group
| cluster_group
| placement_group
| master_node
| nodes
| running_nodes
| stopped_nodes
| spot_requests
| progress_bar
Inherited from `object`: `__class__`

Method Details | [hide private]

### __init__(self, ec2_conn=None, spot_bid=None, cluster_tag=None, cluster_description=None, cluster_size=None, cluster_user=None, cluster_shell=None, master_image_id=None, master_instance_type=None, node_image_id=None, node_instance_type=None, node_instance_types=`[``]`, availability_zone=None, keyname=None, key_location=None, volumes=`[``]`, plugins=`[``]`, permissions=`[``]`, refresh_interval=30, disable_queue=False, disable_threads=False, cluster_group=None, **kwargs) (Constructor) | source code
x.__init__(...) initializes x; see x.__class__.__doc__ for signature Overrides: object.__init__ (inherited documentation)

### __repr__(self) (Representation operator) | source code
repr(x) Overrides: object.__repr__ (inherited documentation)

### load_volumes(self, vols) | source code
Iterate through vols and set device/partition settings automatically if not specified. This method assigns the first volume to /dev/sdz, second to /dev/sdy, etc for all volumes that do not include a device/partition setting

### __str__(self) (Informal representation operator) | source code
str(x) Overrides: object.__str__ (inherited documentation)

### load_receipt(self, load_plugins=True) | source code
Load the original settings used to launch this cluster into this Cluster object. The settings are loaded from the cluster group's description field.

### add_nodes(self, num_nodes, aliases=None) | source code
Add new nodes to this cluster aliases - list of aliases to assign to new nodes (len must equal num_nodes)

### _get_launch_map(self) | source code
    Groups all node-aliases that have similar instance types/image ids
    Returns a dictionary that's used to launch all similar instance types
    and image ids in the same request. Example return value:

    {('c1.xlarge', 'ami-a5c02dcc'): ['node001', 'node002'],
     ('m1.large', 'ami-a5c02dcc'): ['node003'],
     ('m1.small', 'ami-17b15e7e'): ['master', 'node005', 'node006'],
     ('m1.small', 'ami-19e17a2b'): ['node004']}

### _create_flat_rate_cluster(self) | source code
Launches cluster using flat-rate instances. This method attempts to minimize the number of launch requests by grouping nodes of the same type/ami and launching each group simultaneously within a single launch request. This is especially important for Cluster Compute instances given that Amazon *highly* recommends requesting all CCI in a single launch request.

### _create_spot_cluster(self) | source code
Launches cluster using all spot instances. This method makes a single spot request for each node in the cluster since spot instances *always* have an ami_launch_index of 0. This is needed in order to correctly assign aliases to nodes.

### is_cluster_compute(self) | source code
Returns true if any instances are a Cluster Compute type If no instances are currently running, this method checks the original settings used to launch this cluster and returns true if any of the instance type settings specified Cluster Compute instance types

### get_spinner(self, msg) | source code
    Logs a status msg, starts a spinner, and returns the spinner object.
    This is useful for long running processes:

        s = self.get_spinner("Long running process running...")
        (do something)
        s.stop()

### wait_for_cluster(self, msg=`'``Waiting for cluster to come up...``'`) | source code
Wait for cluster to come up and display progress bar. Waits for all spot requests to become 'active', all instances to be in a 'running' state, and for all SSH daemons to come up. msg - custom message to print out before waiting on the cluster

### restart_cluster(*arg, **kargs) | source code
Raw timing function Decorators: `@print_timing('Restarting cluster')`

### stop_cluster(self) | source code
Stop this cluster by detaching all volumes, stopping/terminating all instances, cancelling all spot requests (if any), and removing this cluster's security group. If a node is a spot instance, it will be terminated. Spot instances can not be 'stopped', they must be terminated.

### start(self, create=True, create_only=False, validate=True, validate_only=False, validate_running=False) | source code
Handles creating and configuring a cluster. Validates, creates, and configures a cluster. Passing validate=False will ignore validate_only and validate_running keywords and is effectively the same as running _start

### _start(*arg, **kargs) | source code
Raw timing function Decorators: `@print_timing("Starting cluster")`

### run_plugins(self, plugins=None, method_name=`'``run``'`, node=None, reverse=False) | source code
Run all plugins specified in this Cluster object's self.plugins list Uses plugins list instead of self.plugins if specified. plugins must be a tuple: the first element is the plugin's name, the second element is the plugin object (a subclass of ClusterSetup)

### run_plugin(self, plugin, name=`'````'`, method_name=`'``run``'`, node=None) | source code
Run a StarCluster plugin. plugin - an instance of the plugin's class name - a user-friendly label for the plugin method_name - the method to run within the plugin (default: "run") node - optional node to pass as first argument to plugin method (used for on_add_node/on_remove_node)

### _validate(self, validate_running=False) | source code
Checks that all cluster template settings are valid. Raises a ClusterValidationError exception if not. Passing validate_running=True will also check that the existing instances properties match the configuration of this cluster template.

### __check_platform(self, image_id, instance_type) | source code
Validates whether an image_id (AMI) is compatible with a given instance_type. image_id_setting and instance_type_setting are the setting labels in the config file.

### _validate_ebs_aws_settings(self) | source code
Verify EBS volumes exists on Amazon and that each volume's zone matches this cluster's zone setting. Requires AWS credentials.

### _validate_ebs_settings(self) | source code
Check EBS vols for missing/duplicate DEVICE/PARTITION/MOUNT_PATHs and validate these settings. Does not require AWS credentials.

Property Details | [hide private]

### zone If volumes are specified, this method determines the common availability zone between those volumes. If an availability zone is explicitly specified in the config and does not match the common availability zone of the volumes, an exception is raised. If all volumes are not in the same availabilty zone an exception is raised. If no volumes are specified, returns the user specified availability zone if it exists. Get Method: unreachable.zone(self) - If volumes are specified, this method determines the common availability zone between those volumes.

### _security_group Get Method: unreachable._security_group(self)

### cluster_group Get Method: unreachable.cluster_group(self)

### placement_group Get Method: unreachable.placement_group(self)

### master_node Get Method: unreachable.master_node(self)

### nodes Get Method: unreachable.nodes(self)

### running_nodes Get Method: unreachable.running_nodes(self)

### stopped_nodes Get Method: unreachable.stopped_nodes(self)

### spot_requests Get Method: unreachable.spot_requests(self)

### progress_bar Get Method: unreachable.progress_bar(self)