pilot

Package pilot |

# Source Code for Package pilot

      1  ''' B{BigJob Documentation}
      2
      3  This is the BigJob implementation of the Pilot API. Pilot-Jobs (also referred to as
      4  Pilot Compute) decouple system level and application level job management.
      5
      6  The main concepts and classes exposed by the Compute part of the API are:
      7
      8      - L{PilotCompute} (PC):
      9            a pilot job, which can execute some compute workload (L{ComputeUnit}).
     10
     11      - L{PilotComputeDescription} (PCD):
     12            description for specifying the requirements of a \L{PilotCompute}.
     13
     14      - L{PilotComputeService} (PCS):
     15            a factory for creating \L{PilotCompute}s.
     16
     17  The data side of the Pilot API is symmetric to the compute side.  The exposed
     18  classes for managing Pilot Data are:
     19
     20      - L{PilotData} (PD):
     21            a pilot that manages some data workload (L{DataUnit})
     22
     23      - L{PilotDataDescription} (PDD):
     24            a abstract description of the requirements of the PD
     25
     26      - L{PilotDataService} (PDS):
     27            a factory (service) which can create L{PilotData}s according to some
     28            specification
     29
     30
     31  The application workload is represented by so called L{ComputeUnit}s and  L{DataUnit}s:
     32
     33      - L{ComputeUnit} (CU):
     34            a work item executed on a L{PilotCompute}.
     35
     36
     37      - L{DataUnit} (DU):
     38            a data item managed by a L{PilotData}
     39
     40  Both Compute and Data Units are specified using an abstract description object:
     41
     42      - L{ComputeUnitDescription} (CUD):
     43              abstract description of a L{ComputeUnit}.
     44
     45      - L{DataUnitDescription} (DUD):
     46              abstract description of a L{DataUnit}.
     47
     48
     49  The L{ComputeDataService} represents the central entry point for the application
     50  workload:
     51
     52      - L{ComputeDataService} (CDS)
     53            a service which can map CUs and DUs to a set of Pilot Compute and Pilot Data.
     54
     55  The L{ComputeDataService} (CDS) takes care of the placement of Compute and Data Units.
     56  The set of L{PilotCompute}s and L{PilotData} available to the CDS can be changed during
     57  the application's runtime.  The CDS different data-compute affinity and will handle
     58  compute/data co-locationing for the requested data-compute workload.
     59
     60  Pilots, Compute and Data Units are associated with a L{State}.
     61
     62      - L{State<pilot.api.compute.api.State>}: State enumeration
     63
     64
     65  B{Example}::
     66
     67      from pilot import PilotComputeService, ComputeDataService, State
     68
     69      pilot_compute_service = PilotComputeService()
     70
     71      # create pilot job service and initiate a pilot job
     72      pilot_compute_description = {
     73                               "service_url": 'fork://localhost',
     74                               "number_of_processes": 1,
     75                               "working_directory": os.path.join(os.getcwd(),"work"),
     76                               'affinity_datacenter_label': "eu-de-south",
     77                               'affinity_machine_label': "mymachine"
     78                              }
     79
     80      pilotjob = pilot_compute_service.create_pilot(pilot_compute_description=pilot_compute_description)
     81
     82      compute_data_service = ComputeDataService()
     83      compute_data_service.add_pilot_compute_service(pilot_compute_service)
     84
     85      # start work unit
     86      compute_unit_description = {
     87              "executable": "/bin/date",
     88              "arguments": [""],
     89              "number_of_processes": 1,
     90              "output": "stdout.txt",
     91              "error": "stderr.txt",
     92              "affinity_datacenter_label": "eu-de-south",
     93              "affinity_machine_label": "mymachine"
     94      }
     95
     96      compute_unit = compute_data_service.submit_compute_unit(compute_unit_description)
     97
     98      compute_data_service.wait()
     99
    100      compute_data_service.cancel()
    101
    102
    103  All API objects that should be utilized by the application reside in the L{pilot} namespace. The implementation resides in L{pilot.impl}.
    104
    105  Please, refer to U{https://github.com/saga-project/BigJob/tree/master/examples/pilot-api} for an extensive set of examples.
    106  '''
    107
    108
    109  application_id = "bigjob"
    110
    111
    112  from pilot.impl.pilotcompute_manager import PilotComputeService
    113  from pilot.impl.pilotcompute_manager import PilotCompute
    114  from pilot.impl.pilotcompute_manager import ComputeUnit
    115  from pilot.impl.pilotdata_manager import PilotDataService
    116  from pilot.impl.pilotdata_manager import PilotData
    117  from pilot.impl.pilotdata_manager import DataUnit
    118  from pilot.impl.pilot_manager import ComputeDataService, ComputeUnitService, DataUnitService
    119  from pilot.impl.pilot_manager_decentral import ComputeDataServiceDecentral
    120  from pilot.api.api import PilotError
    121  from pilot.api import State
    122
    123  import bigjob
    124
    125  """ Version of Pilot-API/BigJob """
    126  version = bigjob.version
    127