Components — sfrout 0.0.17 documentation

Here you can find exact details regarding app’s components.

# Components

## Config

class components.config.ConfigProtocol(*args, **kwargs)

  Protocol class for config object.

  Parameters:

    - domain (str) – argument for SFDC domain

    - reports_csv_path (str) – argument for input report csv path.

    - reports_list (list[dict[str, Any]]) – argument for list of reports as dictionary

    - summary_filepath (str) – argument for summary file path

    - log_filepath (str) – argument for log file path

    - report (str) – argument for single report parameters

    - path (str) – argument for save location path override

    - threads (int) – argument for number of threads to use

    - stdout_loglevel (str) – argument for stdout log level

    - file_loglevel (str) – argument for stdout log level

    - verbose (bool) – Flag, toggles progress bar/stdout

class components.config.Config(*, domain: str, reports_csv_path: str, reports_list: list[dict[str, Any]] = [], summary_filepath: str | None = None, log_path: str | None = None, report: str = '', path: str = '', threads: int = 0, stdout_loglevel: str = 'WARNNING', file_loglevel: str = 'INFO', verbose=False)

  Concrete class representing Config object. Contains entire configuration required for a program.

  Parameters:

    - domain (str) – argument for SFDC domain

    - reports_csv_path (str) – argument for input report csv path.

    - reports_list (list[dict[str, Any]], optional) – argument for list of reports as dictionary, defaults to []

    - summary_filepath (str, optional) – argument for summary file path, defaults to None

    - log_filepath (str, optional) – argument for log file path, defaults to None

    - report (str, optional) – argument for single report parameters, defaults to “”

    - path (str, optional) – argument for save location path override, defaults to “”

    - threads (int, optional) – argument for number of threads to use, defaults to 0

    - stdout_loglevel (str, optional) – argument for stdout log level, defaults to “WARNING”

    - file_loglevel (str, optional) – argument for stdout log level, defaults to “INFO”

    - verbose (bool, optional) – Flag, toggles progress bar/stdout, defaults to False

  _define_number_of_threads()

    Defines number of threads. By default number of threads is set to half of available threads. If threads value is not available number of threds will be set to 2. If `report` parameter is filled (single report mode) then number of threads will be automatically set to 1.

  _input_report_path_cast(object_kwargs: list[dict[str, Any]]) → list[dict[str, str | os.PathLike]]

    Casts value of `path` into `Path` object.

    Parameters:

      object_kwargs (list[dict[str, Any]]) – colection of report parameters

    Returns:

      collection of object parameters with path casted to `Path` object

    Return type:

      list[dict[str, str | `os.PathLike`]]

  _input_report_single_mode_override() → list[dict[str, str]]

    Creates report’s parameters in single report mode. Returns parsed parameters as object kwargs.

    Returns:

      collection of report’s parameters in single report mode

    Return type:

      list[dict[str, str]]

  _input_report_csv_standard_file_mode() → list[dict[str, str]]

    Reads parameteres taken from input csv . Returns parsed parameters as objects kwargs.

    Returns:

      collection of report’s parameters in standard mode

    Return type:

      list[dict[str, str]]

  _input_report_path_override(object_kwargs: list[dict[str, str]]) → list[dict[str, str]]

    Replaces value of path key in object_kwargs dict with path parameter value. path override.

    Parameters:

      object_kwargs (list[dict[str, str]]) – Colection of report parameters

    Returns:

      Collection of report’s parameters with replaced path

    Return type:

      list[dict[str, str]]

  _parse_input_report() → list[dict[str, Any]]

    Orchestrating function which parses parameters for `ReportProtocol` .

    Returns:

      Collection of ready to use report kwargs.

    Return type:

      list[dict[str, Any]]

## Connectors

class components.connectors.ConnectorProtocol(*args, **kwargs)

  Protocol class for connector object.

  Parameters:

    - queue (`Queue`) – shared queue object

    - timeout (int) – request’s timeout value in seconds

    - headers (dict[str, str]) – headers required to establish the connection

  check_connection() → bool

    Checks connection with given domain.

    Returns:

      flag, `True` if connection is established, `False` otherwise.

    Return type:

      bool

  async report_gathering(reports: list[components.containers.ReportProtocol], session: ClientSession) → None

    Collects asynchronous responses from the servers.

    Parameters:

      - reports (list[`ReportProtocol`]) – collection of `ReportProtocol` objects.

      - session (`ClientSession`) – HTTP client session object to handle request in transaction.

class components.connectors.SfdcConnector(*, queue: Queue, domain: str, verbose: bool = False, timeout: int = 900, headers: dict[str, str] = {'Content-Type': 'application/csv', 'X-PrettyPrint': '1'})

  Concrete class representing SFDC Connector object.

  Parameters:

    - queue (`Queue`) – shared queue object

    - domain (str) – SalesForce domain of your organization -> “https://corp.my.salesforce.com/”

    - verbose – flag, parameter used as switch between progress bar and logging to stdout, defaults to False

    - timeout (int, optional) – Request’s timeout value in seconds, defaults to 900

    - headers (dict[str, str], optional) – headers required connection, defaults to {‘Content-Type’: ‘application/csv’, ‘X-PrettyPrint’: ‘1’}

    - export_params (str, optional) – GET request parameters required by SFDC, defaults to ‘?export=csv&enc=UTF-8&isdtp=p1’

  _convert_domain_for_cookies_lookup() → str

    Converts domain string according to `sid` CookieJar lookup requirements.

    Returns:

      converted url complaiant with CookieJar schema

    Return type:

      str

  _parse_headers() → None

    Parses headers for request.

  _intercept_sid() → str

    Intercepts sid from MS Edge’s CookieJar.

    Returns:

      Intercepted `sid` or empty string if `sid` doesn’t exist.

    Return type:

      str

  _open_sfdc_site() → None

    Opens SFDC website on given domain url if `sid` is not present or not valid.

  _sid_check() → bool

    Checks `sid` valididty for given SFDC domain.

    Returns:

      Flag, `True` if `sid` was valid, `False` when wasn’t.

    Return type:

      bool

  check_connection() → bool

    Checks the connection with given domain and `sid` validity.

    Returns:

      flag, `True` if connection was successful, `False` if wasn’t.

    Return type:

      bool

  _parse_report_url(report: ReportProtocol) → str

    Parses report object url.

    Parameters:

      report (`ReportProtocol`) – instance of `ReportProtocol`

    Returns:

      parsed url

    Return type:

      str

  async _request_report(report: ReportProtocol, session: ClientSession) → None
    Sends asynchronous request to given domain with given parameters within shared session. Checks response status:
      200: response is saved in `ReportProtocol.response()` , `ReportProtocol.valid()` set to `True` , `ReportProtocol` is being put to the `queue` .
      404: error in response, `ReportProtocol.valid()` set to `False` , no retries.
      500: request timeour, `ReportProtocol.valid()` set to `False` , another attempt.
      *: unknown error, `ReportProtocol.valid()` set to `False`, another attempt.
    Parameters:

      - report (`ReportProtocol`) – instance of `ReportProtocol`

      - session (`aiohttp.ClientSession`) – shared session object

  async _toggle_progress_bar(tasks: list[_asyncio.Task]) → None

    Toggles between showing progress bar or stdout logging.

    Parameters:

      tasks (list[`asyncio.Task`]) – collection of asynchronous request tasks

  _create_async_tasks(reports: list[components.containers.ReportProtocol], session: ClientSession) → list[_asyncio.Task]

    Creates collection of asynchronous request tasks.

    Parameters:

      - reports (list[`ReportProtocol`]) – collection of `ReportProtocol` instances

      - session (`aiohttp.ClientSession`) – shared, asynchronous session

    Returns:

      collection of asynchronous request tasks

    Return type:

      list[`asyncio.Task`]

  async _report_request_all(reports: list[components.containers.ReportProtocol], session: ClientSession) → None

    Orchestrates entire process of processing tasks.

    Parameters:

      - reports (list[`ReportProtocol`]) – collection of `ReportProtocol` instances

      - session (`aiohttp.ClientSession`) – shared, asynchronous session

  async handle_requests(reports: list[components.containers.ReportProtocol]) → None

    Creates session and processes asynchronous tasks.

    Parameters:

      reports (list[`ReportProtocol`]) – collection of `ReportProtocol` instances

## Containers

class components.containers.ReportProtocol(*args, **kwargs)

  Protocol class for report object.

  Parameters:

    - name (str) – report name, propagated to report file name

    - id (str) – report id

    - path (`os.PathLike`) – report path, save location for the report in form of `pathlib.Path` object

    - type (str) – report type, drives connector selection

    - export_params (str) – default GET parameters

    - downloaded (bool) – flag indicating whether the reports has been succesfully downloaded or not

    - valid (bool) – flag indicating whether the response has been succesfully retrieved or not

    - created_date (datetime) – report save completition date

    - pull_date (timedelta) – report response completition date

    - processing_time – the time it took to process the report in seconds

    - attempt_count (int) – number of attempts to process the report

    - size (float) – size of saved report file in Mb

    - response (str) – container for request response

    - content (`pd.DataFrame`) – Pandas `pd.DataFrame` based on response

class components.containers.ReportsContainerProtocol(*args, **kwargs)

  Protocol class for report container object.

  Parameters:

    - report_params_list (list[dict[str, Any]]) – collection of dictionaries with parameters for object crafting

    - summary_report_path (`os.PathLike`) – path to save location of summary report

  create_reports() → list[components.containers.ReportProtocol]

    Orchestrating method to handle report objects factory

    Returns:

      collection of `ReportProtocol` objects

    Return type:

      list[`ReportProtocol`]

  create_summary_report() → None

    Creates summary report which consist of all important details regarding `ReportProtocol` objects. Summary report is generated once all the reports are completed.

class components.containers.SfdcReport(name: str, id: str, path: ~os.PathLike, type: str = 'SFDC', export_params: str = '?export=csv&enc=UTF-8&isdtp=p1', downloaded: bool = False, valid: bool = False, created_date: ~datetime.datetime = datetime.datetime(2023, 4, 7, 3, 40, 58, 22998), pull_date: ~datetime.datetime = datetime.datetime(2023, 4, 7, 3, 40, 58, 22998), processing_time: ~datetime.timedelta = datetime.timedelta(0), attempt_count: int = 0, size: float = 0.0, response: str = '', content: ~pandas.core.frame.DataFrame = <factory>)

  Concrete class representing SFDC Report object.

  Parameters:

    - name (str) – report name, propagated to report file name

    - id (str) – report id, identification number of the report in SFDC

    - path (`os.PathLike`) – report path, save location for the report in form of `pathlib.Path` object

    - type (str, optional) – report type, type drives connector and report objects selectiond, defaults to ‘SFDC’

    - export_params (str, optional) – default GET parameters required by SFDC, defaults to ‘?export=csv&enc=UTF-8&isdtp=p1’

    - downloaded (bool, optional) – flag indicating whether the reports has been succesfully downloaded or not, defaults to False

    - valid (bool, optional) – flag indicating whether the response has been succesfully retrieved or not, defaults to False

    - created_date (datetime, optional) – report save completition date, defaults to current datetime

    - pull_date (timedelta, optional) – report response completition date, defaults to current datetime

    - processing_time – the time it took to process the report in seconds, defaults to 0 microseconds

    - attempt_count (int, optional) – number of attempts to process the report, defaults to 0

    - size (float, optional) – size of saved report file in Mb, defaults to 0.0

    - response (str, optional) – container for request response, defaults “”

    - content (`pd.DataFrame` , optional) – Pandas DataFrame based on response, defaults to empty `pd.DataFrame`

class components.containers.ReportsContainer(reports_params_list: list[dict[str, Any]], summary_path: PathLike | None)

  Concrete class representing ReportContainer object.

  _create_sfdc_reports() → Generator[SfdcReport, None, None]

    Creates `SfdcReport` objects.

    Returns:

      generator with `SfdcReport` objects

    Return type:

      Generator[`SfdcReport`, None, None]

    Yield:

      `SfdcReport` instance based on parsed report parameters

    Return type:

      `SfdcReport`

  create_reports() → list[components.containers.ReportProtocol]

    Orchestrating method to handle report objects crafting

    Returns:

      collection of `ReportProtocol` objects

    Return type:

      list[`ReportProtocol`]

  _create_summary_folder_if_not_exist()

    Creates folder for summary report if doesn’t exist.

  create_summary_report() → None

    Creates summary report which consist of all important details regarding `ReportProtocol` objects. Summary is generated once all the `ReportProtocol` are completed.

  print_summary_table() → None

    Prints summary table which consist of all important details regarding `ReportProtocol` objects. Table is generated once all the `ReportProtocol` are completed.

## Handlers

class components.handlers.WorkerFactoryProtocol(*args, **kwargs)

  Protocol class for worker factory objects.

  Parameters:

    - queue (`Queue`) – shared, thread-safe queue object

    - threads (int) – number of threads, equal to number of `WorkerProtocol` to be deployed

  create_workers() → None

    Creates `WorkerProtocol` on independent threads

  static active_workers() → int

    Counts active `WorkerProtocol` in current time.

    Returns:

      number of active `WorkerProtocol` objects

    Return type:

      int

class components.handlers.WorkerProtocol(*args, **kwargs)

  Protocol class for worker objects.

  Parameters:

    queue (`Queue`) – shared, thread-safe queue object

  _read_stream(report: ReportProtocol) → None

    Reads the stream of data kept in `ReportProtocol` object via Pandas read method. Deletes response content from the object.

    Parameters:

      report (`ReportProtocol`) – instance of the `ReportProtocol` object

  _save_to_csv(report: ReportProtocol) → None

    Saves readed data to `csv` file using Pandas save method.

    Parameters:

      report (`ReportProtocol`) – instance of the `ReportProtocol` object

  _erase_report(report: ReportProtocol) → None

    Erases the report data.

    Parameters:

      report (`ReportProtocol`) – instance of the `ReportProtocol` object

  report_processing(report: ReportProtocol) → None

    Orchiestrates the `ReportProtocol` processing.

    Parameters:

      report (`ReportProtocol`) – instance of the `ReportProtocol` object

  run() → NoReturn

    Starts listner process on separate thread, awaits objects in the queue.

    Returns:

      method never returns

    Return type:

      NoReturn

class components.handlers.WorkerFactory(*, queue: Queue, threads: int = 1)

  Concrete class representing WorkerFactory object.

  Parameters:

    - queue (`Queue`) – shared, thread-safe queue object

    - threads (int, optional) – number of threads, equal to number of `WorkerProtocol` to be deployed, defaults to 1

  create_workers() → None

    Deploys given number of `WorkerProtocol`.

  static active_workers() → int

    Returns number of currently active `WorkerProtocol`.

    Returns:

      number of `WorkerProtocol`

    Return type:

      int

class components.handlers.Worker(queue: Queue)

  Concrete class representing Worker object.

  _read_stream(report: ReportProtocol) → None

    Reads `ReportProtocol` `response` and save it as `content` atribute. Erases saved `response`.

    Parameters:

      report (`ReportProtocol`) – instance of the `ReportProtocol` object

  _parse_save_path(report: ReportProtocol) → PathLike

    Parses path to save location.

    Parameters:

      report (`ReportProtocol`) – instance of the `ReportProtocol` object.

    Returns:

      Path to save location

    Return type:

      `os.PathLike`

  _save_to_csv(report: ReportProtocol) → None

    Saves `ReportProtocol` `content` to `csv` file. Sets object flags.

    Parameters:

      report (`ReportProtocol`) – instance of the `ReportProtocol` object

  _erase_report(report: ReportProtocol) → None

    Deletes `ReportProtocol` object `content` .

    Parameters:

      report (`ReportProtocol`) – instance of the `ReportProtocol` object

  process_report(report: ReportProtocol) → None

    Orchiestrates entire process of downloading the report.

    Parameters:

      report (`ReportProtocol`) – instance of the `ReportProtocol` object

  run() → NoReturn

    Starts to listen to the queue. Starts processing once will get item from the queue. Sends signal to the queue once task is done.

    Returns:

      function never returns

    Return type:

      NoReturn

## Excpetions

exception components.exceptions.OutdatedSIDError(message: str = 'Your SID is outdate, please provide recent SID')

  Exception raised for errors in the SID value.

  …

  message

    explanation of the error

    Type:

      str

exception components.exceptions.EnvFileNotPresent(message: str = '.env file not present in main directory')

  Exception raised if the .env file is not present in main directory.

  …

  message

    explanation of the error

    Type:

      str