pilot

Package pilot |

# Package pilot

source code

BigJob Documentation

This is the BigJob implementation of the Pilot API. Pilot-Jobs (also referred to as Pilot Compute) decouple system level and application level job management.

The main concepts and classes exposed by the Compute part of the API are:

- PilotCompute (PC): a pilot job, which can execute some compute workload (ComputeUnit).
- PilotComputeDescription (PCD): description for specifying the requirements of a \PilotCompute.
- PilotComputeService (PCS): a factory for creating \PilotComputes.

The data side of the Pilot API is symmetric to the compute side. The exposed classes for managing Pilot Data are:

- PilotData (PD): a pilot that manages some data workload (DataUnit)
- PilotDataDescription (PDD): a abstract description of the requirements of the PD
- PilotDataService (PDS): a factory (service) which can create PilotDatas according to some specification

The application workload is represented by so called ComputeUnits and DataUnits:

- ComputeUnit (CU): a work item executed on a PilotCompute.
- DataUnit (DU): a data item managed by a PilotData

Both Compute and Data Units are specified using an abstract description object:

- ComputeUnitDescription (CUD): abstract description of a ComputeUnit.
- DataUnitDescription (DUD): abstract description of a DataUnit.

The ComputeDataService represents the central entry point for the application workload:

- ComputeDataService (CDS) a service which can map CUs and DUs to a set of Pilot Compute and Pilot Data.

The ComputeDataService (CDS) takes care of the placement of Compute and Data Units. The set of PilotComputes and PilotData available to the CDS can be changed during the application's runtime. The CDS different data-compute affinity and will handle compute/data co-locationing for the requested data-compute workload.

Pilots, Compute and Data Units are associated with a State.

- State: State enumeration

Example:

       from pilot import PilotComputeService, ComputeDataService, State

       pilot_compute_service = PilotComputeService()

       # create pilot job service and initiate a pilot job
       pilot_compute_description = {
                                "service_url": 'fork://localhost',
                                "number_of_processes": 1,
                                "working_directory": os.path.join(os.getcwd(),"work"),
                                'affinity_datacenter_label': "eu-de-south",
                                'affinity_machine_label': "mymachine"
                               }

       pilotjob = pilot_compute_service.create_pilot(pilot_compute_description=pilot_compute_description)

       compute_data_service = ComputeDataService()
       compute_data_service.add_pilot_compute_service(pilot_compute_service)

       # start work unit
       compute_unit_description = {
               "executable": "/bin/date",
               "arguments": [""],
               "number_of_processes": 1,
               "output": "stdout.txt",
               "error": "stderr.txt",
               "affinity_datacenter_label": "eu-de-south",
               "affinity_machine_label": "mymachine"
       }

       compute_unit = compute_data_service.submit_compute_unit(compute_unit_description)

       compute_data_service.wait()

       compute_data_service.cancel()

All API objects that should be utilized by the application reside in the pilot namespace. The implementation resides in pilot.impl.

Please, refer to https://github.com/saga-project/BigJob/tree/master/examples/pilot-api for an extensive set of examples.

Submodules
pilot.api: Abstract API Definition of the Pilot-API; This is not the implementation! Please see pilot.impl pilot.api.api pilot.api.compute pilot.api.compute.api: Pilot Compute related entities. pilot.api.data pilot.api.data.api: This file contains the API for the PilotData Framework. pilot.coordination: Private: Implementation of the coordination subsystem. pilot.coordination.advert_adaptor pilot.coordination.nocoord_adaptor pilot.coordination.redis_adaptor pilot.filemanagement: Private: Implementation of the filemanagement subsystem. pilot.filemanagement.bliss_adaptor: Bliss based File Transfer Implementation pilot.filemanagement.globusonline_adaptor: Globus Online based File Transfer pilot.filemanagement.gs_adaptor: Google Storage based File Transfer Implementation pilot.filemanagement.irods_adaptor: iRods based File Transfer Implementation pilot.filemanagement.s3_adaptor: Amazon S3 based File Transfer Implementation pilot.filemanagement.ssh_adaptor: SSH-based coordination scheme between manager and agent pilot.filemanagement.webhdfs_adaptor pilot.impl: BigJob-based Implementation of Pilot-API pilot.impl.pilot_manager: ComputeDataService Module: A central implementation of the ComputeDataService pilot.impl.pilot_manager_decentral: ComputeDataServiceDecentral Module: A decentral implementation of a ComputeDataService (see ComputeDataServiceDecentral). pilot.impl.pilotcompute_manager: PilotCompute Module: Implementation of PilotComputeService, PilotCompute, ComputeUnit. pilot.impl.pilotdata_manager: PilotData Module: Implementation of PilotData, PilotDataService and DataUnit pilot.scheduler pilot.scheduler.data_compute_affinity_scheduler: Affinity-aware scheduler that evaluates affinity labels and input/output data flow

Variables
| application_id = ``'``bigjob``'``
| version = ``'``0.4.123``'``
| __package__ = ``'``pilot``'``