RSE Classes — Rucio 1.2 documentation

# RSE Classes

## Client

class `rucio.client.rseclient.` `RSEClient`(rucio_host=None, auth_host=None, account=None, ca_cert=None, auth_type=None, creds=None, timeout=600, user_agent='rucio-clients')

  Bases: `rucio.client.baseclient.BaseClient`

  RSE client class for working with rucio RSEs

  `RSE_BASEURL` = 'rses'
  `add_distance`(source, destination, parameters)

    Add a src-dest distance.

    Parameters: | source – The source. destination – The destination. parameters – A dictionnary with property.

  `add_protocol`(rse, params)

    Sends the request to create a new protocol for the given RSE.

    Parameters: | rse – the name of the rse. scheme – identifier of this protocol params – Attributes of the protocol. Supported are: hostname: hostname for this protocol (default = localhost) port: port for this protocol (default = 0) prefix: string used as a prfeix for this protocol when generating the PFN (default = None) impl: qualified name of the implementation class for this protocol (mandatory) read: integer representing the priority of this procotol for read operations (default = -1) write: integer representing the priority of this procotol for write operations (default = -1) delete: integer representing the priority of this procotol for delete operations (default = -1) extended_attributes: miscellaneous protocol specific information e.g. spacetoken for SRM (default = None)
    Returns: | True if protocol was created successfully else False.
    Raises: | Duplicate – if protocol with same hostname, port and protocol identifier already exists for the given RSE. RSENotFound – if the RSE doesn’t exist. KeyNotFound – if params is missing manadtory attributes to create the protocol. AccessDenied – if not authorized.

  `add_rse`(rse, **kwargs)

    Sends the request to create a new RSE.

    Parameters: | rse – the name of the rse. deterministic – Boolean to know if the pfn is generated deterministically. volatile – Boolean for RSE cache. city – City for the RSE. region_code – The region code for the RSE. country_name – The country. continent – The continent. time_zone – Timezone. staging_area – Staging area. ISP – Internet service provider.
    Returns: | True if location was created successfully else False.
    Raises: | Duplicate – if rse already exists.

  `add_rse_attribute`(rse, key, value)

    Sends the request to add a RSE attribute.

    Parameters: | rse – the name of the rse. key – the attribute key. value – the attribute value.
    Returns: | True if RSE attribute was created successfully else False.
    Raises: | Duplicate – if RSE attribute already exists.

  `delete_protocols`(rse, scheme, hostname=None, port=None)

    Deletes matching protocols from RSE. Protocols using the same identifier can be distinguished by hostname and port.

    Parameters: | rse – the RSE name. scheme – identifier of the protocol. hostname – hostname of the protocol. port – port of the protocol.
    Returns: | True if success.
    Raises: | RSEProtocolNotSupported – if no matching protocol entry could be found. RSENotFound – if the RSE doesn’t exist. AccessDenied – if not authorized.

  `delete_rse`(rse)

    Sends the request to delete a rse.

    Parameters: | rse – the name of the rse.
    Returns: | True if location was created successfully else False.

  `delete_rse_attribute`(rse, key)

    Sends the request to delete a RSE attribute.

    Parameters: | rse – the RSE name. key – the attribute key.
    Returns: | True if RSE attribute was deleted successfully else False.

  `get_distance`(source, destination)

    Get distances between rses.

    Parameters: | source – The source RSE. destination – The destination RSE.
    Returns distance:
    | List of dictionaries.

  `get_protocols`(rse, protocol_domain='ALL', operation=None, default=False, scheme=None)

    Returns protocol information. Parameter comibantions are: (operation OR default) XOR protocol.

    Parameters: | rse – the RSE name. protocol_domain – The scope of the protocol. Supported are ‘LAN’, ‘WAN’, and ‘ALL’ (as default). operation – The name of the requested operation (read, write, or delete). If None, all operations are queried. default – Indicates if only the default operations should be returned. scheme – The identifier of the requested protocol.
    Returns: | A list with details about each matching protocol.
    Raises: | RSENotFound – if the RSE doesn’t exist. RSEProtocolNotSupported – if no matching protocol entry could be found. RSEOperationNotSupported – if no matching protocol entry for the requested operation could be found.

  `get_rse`(rse)

    Returns details about the referred RSE.

    Parameters: | rse – Name of the referred RSE
    Returns: | A dict containing all attributes of the referred RSE
    Raises: | RSENotFound – if the referred RSE was not found in the database

  `get_rse_limits`(rse)

    Get RSE limits.

    Parameters: | rse – The RSE name.
    Returns: | True if successful, otherwise false.

  `get_rse_usage`(rse, filters=None)

    Get RSE usage information.

    Parameters: | rse – the RSE name. filters – dictionary of attributes by which the results should be filtered
    Returns: | True if successful, otherwise false.

  `lfns2pfns`(rse, lfns, protocol_domain='ALL', operation=None, scheme=None)

    Returns PFNs that should be used at a RSE, corresponding to requested LFNs. The PFNs are generated for the RSE regardless of whether a replica exists for the LFN.

    Parameters: | rse – the RSE name lfns – A list of LFN strings to translate to PFNs. protocol_domain – The scope of the protocol. Supported are ‘LAN’, ‘WAN’, and ‘ALL’ (as default). operation – The name of the requested operation (read, write, or delete). If None, all operations are queried. scheme – The identifier of the requested protocol (gsiftp, https, davs, etc).
    Returns: | A dictionary of LFN / PFN pairs.
    Raises: | RSENotFound – if the RSE doesn’t exist. RSEProtocolNotSupported – if no matching protocol entry could be found. RSEOperationNotSupported – if no matching protocol entry for the requested operation could be found.

  `list_rse_attributes`(rse)

    Sends the request to get RSE attributes.

    Parameters: | rse – The RSE name.
    Returns: | True if RSE attribute was created successfully else False.

  `list_rse_usage_history`(rse, filters=None)

    List RSE usage history information.

    Parameters: | rse – The RSE name. filters – dictionary of attributes by which the results should be filtered.
    Returns: | list of dictionnaries.

  `list_rses`(rse_expression=None)

    Sends the request to list all rucio locations(RSEs).

    Rse_expression: | RSE Expression to use as filter.
    Returns: | a list containing the names of all rucio locations.

  `set_rse_limits`(rse, name, value)

    Set RSE limit information.

    Parameters: | rse – The RSE name. name – The name of the limit. value – The feature value. Set to -1 to remove the limit.
    Returns: | True if successful, otherwise false.

  `set_rse_usage`(rse, source, used, free)

    Set RSE usage information.

    Parameters: | rse – the RSE name. source – the information source, e.g. srm. used – the used space in bytes. free – the free in bytes.
    Returns: | True if successful, otherwise false.

  `swap_protocols`(rse, domain, operation, scheme_a, scheme_b)

    Swaps the priorities of the provided operation.

    Parameters: | rse – the RSE name. domain – the domain in which priorities should be swapped i.e. wan or lan. operation – the operation that should be swapped i.e. read, write, or delete. scheme_a – the scheme of one of the two protocols to be swapped, e.g. srm. scheme_b – the scheme of the other of the two protocols to be swapped, e.g. http.
    Returns: | True if success.
    Raises: | RSEProtocolNotSupported – if no matching protocol entry could be found. RSENotFound – if the RSE doesn’t exist. KeyNotFound – if invalid data was provided for update. AccessDenied – if not authorized.

  `update_distance`(source, destination, parameters)

    Update distances with the given RSE ids.

    Parameters: | source – The source. destination – The destination. parameters – A dictionnary with property.

  `update_protocols`(rse, scheme, data, hostname=None, port=None)

    Updates matching protocols from RSE. Protocol using the same identifier can be distinguished by hostname and port.

    Parameters: | rse – the RSE name. scheme – identifier of the protocol. data – A dict providing the new values of the protocol attibutes. Keys must match column names in database. hostname – hostname of the protocol. port – port of the protocol.
    Returns: | True if success.
    Raises: | RSEProtocolNotSupported – if no matching protocol entry could be found. RSENotFound – if the RSE doesn’t exist. KeyNotFound – if invalid data was provided for update. AccessDenied – if not authorized.

  `update_rse`(rse, parameters)

    Update RSE properties like availability or name.

    Parameters: | rse – the name of the new rse. parameters – A dictionnary with property (name, read, write, delete as keys).

## Manager

`rucio.rse.rsemanager.` `create_protocol`(rse_settings, operation, scheme=None, domain='wan')

  Instanciates the protocol defined for the given operation.

  Parameters: | rse_attr – RSE attributes operation – Intended operation for this protocol scheme – Optional filter if no specific protocol is defined in rse_setting for the provided operation domain – Optional specification of the domain
  Returns: | An instance of the requested protocol

`rucio.rse.rsemanager.` `delete`(rse_settings, lfns)

  Delete a file from the connected storage. Providing a list indicates the bulk mode.

  Parameters: | lfns – a single dict or a list with dicts containing ‘scope’ and ‘name’. E.g. [{‘name’: ‘1_rse_remote_delete.raw’, ‘scope’: ‘user.jdoe’}, {‘name’: ‘2_rse_remote_delete.raw’, ‘scope’: ‘user.jdoe’}]
  Returns: | True/False for a single file or a dict object with ‘scope:name’ as keys and True or the exception as value for each file in bulk mode
  Raises: | RSENotConnected – no connection to a specific storage has been established SourceNotFound – remote source file can not be found on storage ServiceUnavailable – for any other reason

`rucio.rse.rsemanager.` `download`(rse_settings, files, dest_dir=None, force_scheme=None, ignore_checksum=False, printstatements=False, domain='wan', transfer_timeout=None)

  Copy a file from the connected storage to the local file system. Providing a list indicates the bulk mode.

  Parameters: | rse_settings – RSE to use files – a single dict or a list with dicts containing ‘scope’ and ‘name’ if LFNs are provided and additional ‘pfn’ if PFNs are provided. Examples: [ {‘name’: ‘2_rse_remote_get.raw’, ‘scope’: ‘user.jdoe’}, {‘name’:‘3_rse_remote_get.raw’, ‘scope’: ‘user.jdoe’, ‘pfn’: ‘user/jdoe/5a/98/3_rse_remote_get.raw’} ] dest_dir – path to the directory where the downloaded files will be stored. If not given, each scope is represented by its own directory. force_scheme – normally the scheme is dictated by the RSE object, when specifying the PFN it must be forced to the one specified in the PFN, overruling the RSE description. ignore_checksum – do not verify the checksum - caution: should only be used for rucio download –pfn transfer_timeout – set this timeout (in seconds) for the transfers, for protocols that support it
  Returns: | True/False for a single file or a dict object with ‘scope:name’ for LFNs or ‘name’ for PFNs as keys and True or the exception as value for each file in bulk mode
  Raises: | SourceNotFound – remote source file can not be found on storage DestinationNotAccessible – local destination directory is not accessible FileConsistencyMismatch – the checksum of the downloaded file does not match the provided one ServiceUnavailable – for any other reason

`rucio.rse.rsemanager.` `exists`(rse_settings, files)

  Checks if a file is present at the connected storage. Providing a list indicates the bulk mode.

  Parameters: | files – a single dict or a list with dicts containing ‘scope’ and ‘name’ if LFNs are used and only ‘name’ if PFNs are used. E.g. {‘name’: ‘2_rse_remote_get.raw’, ‘scope’: ‘user.jdoe’}, {‘name’: ‘user/jdoe/5a/98/3_rse_remote_get.raw’}
  Returns: | True/False for a single file or a dict object with ‘scope:name’ for LFNs or ‘name’ for PFNs as keys and True or the exception as value for each file in bulk mode
  Raises: | RSENotConnected – no connection to a specific storage has been established

`rucio.rse.rsemanager.` `find_matching_scheme`(rse_settings_dest, rse_settings_src, operation_src, operation_dest, domain='wan', scheme=None)

  Find the best matching scheme between two RSEs

  Parameters: | rse_settings_dest – RSE settings for the destination RSE. rse_settings_src – RSE settings for the src RSE. operation_src – Source Operation such as read, write. operation_dest – Dest Operation such as read, write. domain – Domain such as lan, wan. scheme – List of supported schemes.
  Returns: | Tuple of matching schemes (dest_scheme, src_scheme).

`rucio.rse.rsemanager.` `get_protocols_ordered`(rse_settings, operation, scheme=None, domain='wan')

`rucio.rse.rsemanager.` `get_rse_info`(rse, session=None)

  Returns all protocol related RSE attributes.

  Parameters: | rse – Name of the requested RSE session – The eventual database session.
  Returns: | a dict object with the following attributes: id … an internal identifier rse … the name of the RSE as string type … the storage type odf the RSE e.g. DISK volatile … boolean indictaing if the RSE is volatile verify_checksum … boolean indicating whether RSE supports requests for checksums deteministic … boolean indicating of the nameing of the files follows the defined determinism domain … indictaing the domain that should be assumed for transfers. Values are ‘ALL’, ‘LAN’, or ‘WAN’ protocols … all supported protocol in form of a list of dict objects with the followig structure - scheme … protocol scheme e.g. http, srm, … - hostname … hostname of the site - prefix … path to the folder where the files are stored - port … port used for this protocol - impl … naming the python class of the protocol implementation - extended_attributes … additional information for the protocol - domains … a dict naming each domain and the priority of the protocol for each operation (lower is better, zero is not upported)
  Raises: | RSENotFound – if the provided RSE coud not be found in the database.

`rucio.rse.rsemanager.` `get_space_usage`(rse_settings, scheme=None)

  Get RSE space usage information.

  Parameters: | scheme – optional filter to select which protocol to be used.
  Returns: | a list with dict containing ‘totalsize’ and ‘unusedsize’
  Raises: | ServiceUnavailable – if some generic error occured in the library.

`rucio.rse.rsemanager.` `lfns2pfns`(rse_settings, lfns, operation='write', scheme=None, domain='wan')

  Convert the lfn to a pfn

  Parameters: | lfns – logical file names as a dict containing ‘scope’ and ‘name’ as keys. For bulk a list of dicts can be provided protocol – instance of the protocol to be used to create the PFN
  Returns: | a dict with scope:name as key and the PFN as value

`rucio.rse.rsemanager.` `parse_pfns`(rse_settings, pfns, operation='read', domain='wan')

  Checks if a PFN is feasible for a given RSE. If so it splits the pfn in its various components.

  Parameters: | pfns – list of PFNs protocol – instance of the protocol to be used to create the PFN
  Returns: | A dict with the parts known by the selected protocol e.g. scheme, hostname, prefix, path, name
  Raises: | RSEFileNameNotSupported – if provided PFN is not supported by the RSE/protocol RSENotFound – if the referred storage is not found i the repository (rse_id) InvalidObject – If the properties parameter doesn’t include scheme, hostname, and port as keys RSEOperationNotSupported – If no matching protocol was found for the requested operation

`rucio.rse.rsemanager.` `rename`(rse_settings, files)

  Rename files stored on the connected storage. Providing a list indicates the bulk mode.

  Parameters: | files – a single dict or a list with dicts containing ‘scope’, ‘name’, ‘new_scope’ and ‘new_name’ if LFNs are used or only ‘name’ and ‘new_name’ if PFNs are used. If ‘new_scope’ or ‘new_name’ are not provided, the current one is used. Examples: [ {‘name’: ‘3_rse_remote_rename.raw’, ‘scope’: ‘user.jdoe’, ‘new_name’: ‘3_rse_new.raw’, ‘new_scope’: ‘user.jdoe’}, {‘name’: ‘user/jdoe/d9/cb/9_rse_remote_rename.raw’, ‘new_name’: ‘user/jdoe/c6/4a/9_rse_new.raw’} ]
  Returns: | True/False for a single file or a dict object with LFN (key) and True/False (value) in bulk mode
  Raises: | RSENotConnected – no connection to a specific storage has been established SourceNotFound – remote source file can not be found on storage DestinationNotAccessible – remote destination directory is not accessible ServiceUnavailable – for any other reason

`rucio.rse.rsemanager.` `select_protocol`(rse_settings, operation, scheme=None, domain='wan')

`rucio.rse.rsemanager.` `upload`(rse_settings, lfns, source_dir=None, force_pfn=None, force_scheme=None, transfer_timeout=None)

  Uploads a file to the connected storage. Providing a list indicates the bulk mode.

  Parameters: | lfns – a single dict or a list with dicts containing ‘scope’ and ‘name’. Examples: [ {‘name’: ‘1_rse_local_put.raw’, ‘scope’: ‘user.jdoe’, ‘filesize’: 42, ‘adler32’: ‘87HS3J968JSNWID’}, {‘name’: ‘2_rse_local_put.raw’, ‘scope’: ‘user.jdoe’, ‘filesize’: 4711, ‘adler32’: ‘RSSMICETHMISBA837464F’} ] If the ‘filename’ key is present, it will be used by Rucio as the actual name of the file on disk (separate from the Rucio ‘name’). source_dir – path to the local directory including the source files force_pfn – use the given PFN – can lead to dark data, use sparingly force_scheme – use the given protocol scheme, overriding the protocol priority in the RSE description transfer_timeout – set this timeout (in seconds) for the transfers, for protocols that support it
  Returns: | True/False for a single file or a dict object with ‘scope:name’ as keys and True or the exception as value for each file in bulk mode
  Raises: | RSENotConnected – no connection to a specific storage has been established SourceNotFound – local source file can not be found DestinationNotAccessible – remote destination directory is not accessible ServiceUnavailable – for any other reason

## Storage

class `rucio.rse.protocols.protocol.` `RSEProtocol`(protocol_attr, rse_settings)

  Bases: `object`

  This class is virtual and acts as a base to inherit new protocols from. It further provides some common functionality which applies for the amjority of the protocols.

  `close`()

    Closes the connection to RSE.

  `connect`()

    Establishes the actual connection to the referred RSE.

    Raises: | RSEAccessDenied – if no connection could be established.

  `delete`(path)

    Deletes a file from the connected RSE.

    Parameters: | path – path to the to be deleted file
    Raises: | ServiceUnavailable – if some generic error occured in the library. SourceNotFound – if the source file was not found on the referred storage.

  `exists`(path)

    Checks if the requested file is known by the referred RSE.

    Parameters: | path – Physical file name
    Returns: | True if the file exists, False if it doesn’t
    Raises: | SourceNotFound – if the source file was not found on the referred storage.

  `get`(path, dest, transfer_timeout=None)

    Provides access to files stored inside connected the RSE.

    Parameters: | path – Physical file name of requested file dest – Name and path of the files when stored at the client transfer_timeout – Transfer timeout (in seconds)
    Raises: | DestinationNotAccessible – if the destination storage was not accessible. ServiceUnavailable – if some generic error occured in the library. SourceNotFound – if the source file was not found on the referred storage.

  `get_space_usage`()

    Get RSE space usage information.

    Returns: | a list with dict containing ‘totalsize’ and ‘unusedsize’
    Raises: | ServiceUnavailable – if some generic error occured in the library.

  `lfns2pfns`(lfns)

    Retruns a fully qualified PFN for the file referred by path.

    Parameters: | path – The path to the file.
    Returns: | Fully qualified PFN.

  `parse_pfns`(pfns)

    Splits the given PFN into the parts known by the protocol. It is also checked if the provided protocol supportes the given PFNs.

    Parameters: | pfns – a list of a fully qualified PFNs
    Returns: | dic with PFN as key and a dict with path and name as value
    Raises: | RSEFileNameNotSupported – if the provided PFN doesn’t match with the protocol settings

  `put`(source, target, source_dir, transfer_timeout=None)

    Allows to store files inside the referred RSE.

    Parameters: | source – path to the source file on the client file system target – path to the destination file on the storage source_dir – Path where the to be transferred files are stored in the local file system transfer_timeout – Transfer timeout (in seconds)
    Raises: | DestinationNotAccessible – if the destination storage was not accessible. ServiceUnavailable – if some generic error occured in the library. SourceNotFound – if the source file was not found on the referred storage.

  `rename`(path, new_path)

    Allows to rename a file stored inside the connected RSE.

    Parameters: | path – path to the current file on the storage new_path – path to the new file on the storage
    Raises: | DestinationNotAccessible – if the destination storage was not accessible. ServiceUnavailable – if some generic error occured in the library. SourceNotFound – if the source file was not found on the referred storage.

  `stat`(path)

    Returns the stats of a file.

    Parameters: | path – path to file
    Raises: | ServiceUnavailable – if some generic error occured in the library. SourceNotFound – if the source file was not found on the referred storage.
    Returns: | a dict with two keys, filesize and adler32 of the file provided in path.