API — Sneakpeek 0.2.2 documentation

# API

class sneakpeek.server.SneakpeekServer(consumer: Optional[sneakpeek.queue.consumer.Consumer] = None, scheduler: Optional[sneakpeek.scheduler.model.SchedulerABC] = None, web_server: Optional[fastapi_jsonrpc.API] = None, web_server_port: int = 8080)

  Bases: `object`

  Sneakpeek server. It can run multiple services at once:

  - API - allows interactions with scrapers storage and scrapers via JsonRPC or UI

  - Worker - executes scheduled scrapers

  - Scheduler - automatically schedules scrapers that are stored in the storage

  Parameters

    - consumer (Consumer | None, optional) – Worker that consumes tasks queue. Defaults to None.

    - scheduler (SchedulerABC | None, optional) – Scrapers scheduler. Defaults to None.

    - web_server (jsonrpc.API | None, optional) – Web Server that implements API and exposes UI to interact with the system. Defaults to None.

    - web_server_port (int, optional) – Port which is used for Web Server (API, UI and metrics). Defaults to 8080.

  Return type

    None

  static create(handlers: list[sneakpeek.scraper.model.ScraperHandler], scraper_storage: sneakpeek.scraper.model.ScraperStorageABC, queue_storage: sneakpeek.queue.model.QueueStorageABC, lease_storage: sneakpeek.scheduler.model.LeaseStorageABC, with_web_server: bool = True, with_worker: bool = True, with_scheduler: bool = True, worker_max_concurrency: int = 50, web_server_port: int = 8080, scheduler_storage_poll_delay: datetime.timedelta = datetime.timedelta(seconds=5), scheduler_lease_duration: datetime.timedelta = datetime.timedelta(seconds=60), middlewares: Optional[list[sneakpeek.scraper.model.Middleware]] = None, add_dynamic_scraper_handler: bool = False, session_logger_handler: Optional[logging.Handler] = None)

    Create Sneakpeek server using default API, worker and scheduler implementations

    Parameters

      - handlers (list[ScraperHandler]) – List of handlers that implement scraper logic

      - scraper_storage (ScrapersStorage) – Scrapers storage

      - jobs_storage (ScraperJobsStorage) – Jobs storage

      - lease_storage (LeaseStorage) – Lease storage

      - with_web_server (bool, optional) – Whether to run API service. Defaults to True.

      - with_worker (bool, optional) – Whether to run worker service. Defaults to True.

      - with_scheduler (bool, optional) – Whether to run scheduler service. Defaults to True.

      - worker_max_concurrency (int, optional) – Maximum number of concurrently executed scrapers. Defaults to 50.

      - web_server_port (int, optional) – Port which is used for Web Server (API, UI and metrics). Defaults to 8080.

      - scheduler_storage_poll_delay (timedelta, optional) – How much scheduler wait before polling storage for scrapers updates. Defaults to 5 seconds.

      - scheduler_lease_duration (timedelta, optional) – How long scheduler lease lasts. Lease is required for scheduler to be able to create new scraper jobs. This is needed so at any point of time there’s only one active scheduler instance. Defaults to 1 minute.

      - plugins (list[Plugin] | None, optional) – List of plugins that will be used by scraper runner. Can be omitted if run_worker is False. Defaults to None.

      - add_dynamic_scraper_handler (bool, optional) – Whether to add dynamic scraper handler which can execute arbitrary user scripts. Defaults to False.

      - queue_storage (sneakpeek.queue.model.QueueStorageABC) –

      - middlewares (Optional[list[sneakpeek.scraper.model.Middleware]]) –

      - session_logger_handler (Optional[logging.Handler]) –

  serve(loop: Optional[asyncio.events.AbstractEventLoop] = None, blocking: bool = True) → None

    Start Sneakpeek server

    Parameters

      - loop (asyncio.AbstractEventLoop | None, optional) – AsyncIO loop to use. In case it’s None result of asyncio.get_event_loop() will be used. Defaults to None.

      - blocking (bool, optional) – Whether to block thread while server is running. Defaults to True.

    Return type

      None

  stop(loop: Optional[asyncio.events.AbstractEventLoop] = None) → None

    Stop Sneakpeek server

    Parameters

      loop (asyncio.AbstractEventLoop | None, optional) – AsyncIO loop to use. In case it’s None result of asyncio.get_event_loop() will be used. Defaults to None.

    Return type

      None

class sneakpeek.queue.model.EnqueueTaskRequest(*, task_name: str, task_handler: str, priority: sneakpeek.queue.model.TaskPriority, payload: str, timeout: datetime.timedelta | None = None)

  Bases: `pydantic.main.BaseModel`

  Enqueue request

  Create a new model by parsing and validating input data from keyword arguments.

  Raises ValidationError if the input data cannot be parsed to form a valid model.

  Parameters

    - task_name (str) –

    - task_handler (str) –

    - priority (sneakpeek.queue.model.TaskPriority) –

    - payload (str) –

    - timeout (datetime.timedelta | None) –

  Return type

    None

  payload: str

    Serialized task payload

  priority: sneakpeek.queue.model.TaskPriority

    Task priority

  task_handler: str

    Name of the task handler

  task_name: str

    Task name (used to disallow concurrent execution of the task)

  timeout: datetime.timedelta | None

    Task timeout

class sneakpeek.queue.model.QueueABC

  Bases: `abc.ABC`

  Task priority queue

  abstract async delete_old_tasks(keep_last: int = 50) → None

    Delete old historical tasks

    Parameters

      keep_last (int, optional) – How many tasks to keep. Defaults to 50.

    Return type

      None

  abstract async dequeue() → sneakpeek.queue.model.Task | None

    Try to dequeue a task from the queue.

    Returns

      Dequeued task metadata

    Return type

      Task

  abstract async enqueue(request: sneakpeek.queue.model.EnqueueTaskRequest) → sneakpeek.queue.model.Task

    Enqueue task

    Parameters

      request (EnqueueTaskRequest) – metadata of task to enqueue

    Returns

      Enqueued task metadata

    Return type

      Task

    Raises

      TaskHasActiveRunError – Error when there are other tasks with the same name in `PENDING` or `STARTED` state

  abstract async get_queue_len() → int

    Get number of pending items in the queue

    Return type

      int

  abstract async get_task_instance(task_id: int) → sneakpeek.queue.model.Task

    Get task instance by ID

    Parameters

      task_id (int) – Task ID

    Returns

      Task instance

    Return type

      Task

  abstract async get_task_instances(task_name: str) → list[sneakpeek.queue.model.Task]

    Get task instances by task name

    Parameters

      task_name (str) – Task name

    Returns

      List of task instances

    Return type

      list[Task]

  abstract async kill_dead_tasks() → list[sneakpeek.queue.model.Task]

    Kill dead tasks

    Returns

      List of killed dead tasks

    Return type

      list[Task]

  abstract async ping_task(id: int) → sneakpeek.queue.model.Task

    Send a heartbeat for the task

    Parameters

      id (int) – Task ID

    Returns

      Updated task metadata

    Return type

      Task

    Raises

      - TaskNotFoundError – Raised if task doesn’t exist

      - TaskNotStartedError – Raised if task is still in the `PENDING` state

      - TaskPingFinishedError – Raised if task is in finished state (e.g. `DEAD`)

  abstract async update_task(task: sneakpeek.queue.model.Task) → sneakpeek.queue.model.Task

    Update task metadata

    Parameters

      task (Task) – updated task metadata to save

    Returns

      Updated task metadata

    Return type

      Task

    Raises

      TaskNotFoundError – Raised when task doesn’t exist

class sneakpeek.queue.model.QueueStorageABC

  Bases: `abc.ABC`

  Priority queue storage

  abstract async delete_old_tasks(keep_last: int = 50) → None

    Delete old historical tasks

    Parameters

      keep_last (int, optional) – How many tasks to keep. Defaults to 50.

    Return type

      None

  abstract async dequeue_task() → sneakpeek.queue.model.Task | None

    Try to dequeue pending task

    Returns

      First pending task or None if the queue is empty

    Return type

      Task | None

  abstract async enqueue_task(task: sneakpeek.queue.model.Task) → sneakpeek.queue.model.Task

    Add a new task instance and put it into the queue

    Parameters

      task (Task) – task to add

    Returns

      Created task metadata

    Return type

      Task

  abstract async get_queue_len() → int

    Get number of pending items in the queue

    Return type

      int

  abstract async get_task_instance(id: int) → sneakpeek.queue.model.Task

    Get task instance by ID

    Parameters

      id (int) – Task ID

    Returns

      Found task metadata

    Return type

      Task

    Raises

      TaskNotFoundError – Raised when task doesn’t exist

  abstract async get_task_instances(task_name: str) → list[sneakpeek.queue.model.Task]

    Get task instances by task name

    Parameters

      task_name (str) – Task name

    Returns

      List of task instances

    Return type

      list[Task]

  abstract async get_tasks() → list[sneakpeek.queue.model.Task]

    Get all task instances

    Returns

      List of task instances

    Return type

      list[Task]

  abstract async update_task(task: sneakpeek.queue.model.Task) → sneakpeek.queue.model.Task

    Update task metadata

    Parameters

      task (Task) – updated task metadata to save

    Returns

      Updated task metadata

    Return type

      Task

    Raises

      TaskNotFoundError – Raised when task doesn’t exist

class sneakpeek.queue.model.Task(*, id: int, task_name: str, task_handler: str, status: sneakpeek.queue.model.TaskStatus, priority: sneakpeek.queue.model.TaskPriority, created_at: datetime.datetime, started_at: datetime.datetime | None = None, last_active_at: datetime.datetime | None = None, finished_at: datetime.datetime | None = None, payload: str | None = None, result: str | None = None, timeout: datetime.timedelta | None = None)

  Bases: `pydantic.main.BaseModel`

  Queue task state

  Create a new model by parsing and validating input data from keyword arguments.

  Raises ValidationError if the input data cannot be parsed to form a valid model.

  Parameters

    - id (int) –

    - task_name (str) –

    - task_handler (str) –

    - status (sneakpeek.queue.model.TaskStatus) –

    - priority (sneakpeek.queue.model.TaskPriority) –

    - created_at (datetime.datetime) –

    - started_at (datetime.datetime | None) –

    - last_active_at (datetime.datetime | None) –

    - finished_at (datetime.datetime | None) –

    - payload (str | None) –

    - result (str | None) –

    - timeout (datetime.timedelta | None) –

  Return type

    None

  created_at: datetime.datetime

    When the task was created and enqueued

  finished_at: datetime.datetime | None

    When the task finished

  id: int

    Task unique identifier

  last_active_at: datetime.datetime | None

    When the task last sent heartbeat

  payload: str | None

    Serialized task payload

  priority: sneakpeek.queue.model.TaskPriority

    Task priority

  result: str | None

    Serialised task result

  started_at: datetime.datetime | None

    When the task was dequeued and started being processed by the worker

  status: sneakpeek.queue.model.TaskStatus

    Task status

  task_handler: str

    Name of the task handler

  task_name: str

    Task name (used to disallow concurrent execution of the task)

  timeout: datetime.timedelta | None

    Task timeout

exception sneakpeek.queue.model.TaskHasActiveRunError(data=None)

  Bases: `fastapi_jsonrpc.BaseError`

exception sneakpeek.queue.model.TaskNotFoundError(data=None)

  Bases: `fastapi_jsonrpc.BaseError`

exception sneakpeek.queue.model.TaskPingFinishedError(data=None)

  Bases: `fastapi_jsonrpc.BaseError`

exception sneakpeek.queue.model.TaskPingNotStartedError(data=None)

  Bases: `fastapi_jsonrpc.BaseError`

class sneakpeek.queue.model.TaskPriority(value)

  Bases: `enum.Enum`

  Priority of the scraper job

  HIGH = 1
  NORMAL = 2
  UTMOST = 0

class sneakpeek.queue.model.TaskStatus(value)

  Bases: `str`, `enum.Enum`

  Scraper job status

  DEAD = 'dead'

    Task was inactive for a while, so scheduler marked it as dead and scheduler can schedule the task again

  FAILED = 'failed'

    Task failed

  KILLED = 'killed'

    Task was killed by the user

  PENDING = 'pending'

    Task is in the queue

  STARTED = 'started'

    Task dequeued by the worker and is being processed

  SUCCEEDED = 'succeeded'

    Task succeeded

exception sneakpeek.queue.model.TaskTimedOut(data=None)

  Bases: `fastapi_jsonrpc.BaseError`

exception sneakpeek.queue.model.UnknownTaskHandlerError(data=None)

  Bases: `fastapi_jsonrpc.BaseError`

class sneakpeek.scheduler.model.Lease(*, name: str, owner_id: str, acquired: datetime.datetime, acquired_until: datetime.datetime)

  Bases: `pydantic.main.BaseModel`

  Global lease metadata

  Create a new model by parsing and validating input data from keyword arguments.

  Raises ValidationError if the input data cannot be parsed to form a valid model.

  Parameters

    - name (str) –

    - owner_id (str) –

    - acquired (datetime.datetime) –

    - acquired_until (datetime.datetime) –

  Return type

    None

  acquired: datetime.datetime

    Time when the lease was acquired

  acquired_until: datetime.datetime

    Time until the lease is acquired

  name: str

    Lease name (resource name to be locked)

  owner_id: str

    ID of the acquirer (should be the same if you already have the lease and want to prolong it)

class sneakpeek.scheduler.model.LeaseStorageABC

  Bases: `abc.ABC`

  Global lease storage abstract class

  abstract async maybe_acquire_lease(lease_name: str, owner_id: str, acquire_for: datetime.timedelta) → sneakpeek.scheduler.model.Lease | None

    Try to acquire lease (global lock).

    Parameters

      - lease_name (str) – Lease name (resource name to be locked)

      - owner_id (str) – ID of the acquirer (should be the same if you already have the lease and want to prolong it)

      - acquire_for (timedelta) – For how long lease will be acquired

    Returns

      Lease metadata if it was acquired, None otherwise

    Return type

      Lease | None

  abstract async release_lease(lease_name: str, owner_id: str) → None

    Release lease (global lock)

    Parameters

      - lease_name (str) – Lease name (resource name to be unlocked)

      - owner_id (str) – ID of the acquirer

    Return type

      None

class sneakpeek.scheduler.model.MultiPeriodicTasksStorage(storages: list[sneakpeek.scheduler.model.PeriodicTasksStorageABC])

  Bases: `sneakpeek.scheduler.model.PeriodicTasksStorageABC`

  Parameters

    storages (list[sneakpeek.scheduler.model.PeriodicTasksStorageABC]) –

  Return type

    None

class sneakpeek.scheduler.model.PeriodicTasksStorageABC

  Bases: `abc.ABC`

class sneakpeek.scheduler.model.SchedulerABC

  Bases: `abc.ABC`

class sneakpeek.scheduler.model.StaticPeriodicTasksStorage(tasks: list[sneakpeek.scheduler.model.PeriodicTask])

  Bases: `sneakpeek.scheduler.model.PeriodicTasksStorageABC`

  Parameters

    tasks (list[sneakpeek.scheduler.model.PeriodicTask]) –

  Return type

    None

class sneakpeek.scheduler.model.TaskSchedule(value)

  Bases: `str`, `enum.Enum`

  Periodic task schedule options. Note that it’s disallowed to have 2 concurrent task, so if there’s an active task new one won’t be scheduled

  CRONTAB = 'crontab'

    Specify crontab when scraper should be scheduled

  EVERY_DAY = 'every_day'

    Scraper will be scheduled every day

  EVERY_HOUR = 'every_hour'

    Scraper will be scheduled every hour

  EVERY_MINUTE = 'every_minute'

    Scraper will be scheduled every minute

  EVERY_MONTH = 'every_month'

    Scraper will be scheduled every month

  EVERY_SECOND = 'every_second'

    Scraper will be scheduled every second

  EVERY_WEEK = 'every_week'

    Scraper will be scheduled every week

  INACTIVE = 'inactive'

    Scraper won’t be automatically scheduled

class sneakpeek.scraper.model.HttpMethod(value)

  Bases: `str`, `enum.Enum`

  HTTP method

class sneakpeek.scraper.model.Middleware

  Bases: `abc.ABC`

  Abstract class for the middleware which is called before each request and response

  abstract property name: str

    Name of the middleware

  abstract async on_request(request: sneakpeek.scraper.model.Request, config: Optional[Any] = None) → sneakpeek.scraper.model.Request

    Function that is called on each (HTTP) request before its dispatched.

    Parameters

      - request (Request) – Request metadata

      - config (Any | None, optional) – Middleware configuration. Defaults to None.

    Returns

      Request metadata

    Return type

      Request

  abstract async on_response(request: sneakpeek.scraper.model.Request, response: aiohttp.client_reqrep.ClientResponse, config: Optional[Any] = None) → aiohttp.client_reqrep.ClientResponse

    Function that is called on each (HTTP) response before its result returned to the caller.

    Parameters

      - request (Request) – Request metadata

      - response (aiohttp.ClientResponse) – HTTP Response

      - config (Any | None, optional) – Middleware configuration. Defaults to None.

    Returns

      HTTP Response

    Return type

      aiohttp.ClientResponse

class sneakpeek.scraper.model.Request(method: sneakpeek.scraper.model.HttpMethod, url: str, headers: Optional[dict[str, str]] = None, kwargs: Optional[dict[str, typing.Any]] = None)

  Bases: `object`

  HTTP Request metadata

  Parameters

    - method (sneakpeek.scraper.model.HttpMethod) –

    - url (str) –

    - headers (dict[str, str] | None) –

    - kwargs (dict[str, typing.Any] | None) –

  Return type

    None

class sneakpeek.scraper.model.Scraper(*, id: str, name: str, handler: str, schedule: sneakpeek.scheduler.model.TaskSchedule, schedule_crontab: str | None = None, config: sneakpeek.scraper.model.ScraperConfig, priority: sneakpeek.queue.model.TaskPriority = TaskPriority.NORMAL, state: str | None = None, timeout: datetime.timedelta | None = None)

  Bases: `pydantic.main.BaseModel`

  Scraper metadata

  Create a new model by parsing and validating input data from keyword arguments.

  Raises ValidationError if the input data cannot be parsed to form a valid model.

  Parameters

    - id (str) –

    - name (str) –

    - handler (str) –

    - schedule (sneakpeek.scheduler.model.TaskSchedule) –

    - schedule_crontab (str | None) –

    - config (sneakpeek.scraper.model.ScraperConfig) –

    - priority (sneakpeek.queue.model.TaskPriority) –

    - state (str | None) –

    - timeout (datetime.timedelta | None) –

  Return type

    None

  config: sneakpeek.scraper.model.ScraperConfig

    Scraper configuration that is passed to the handler

  handler: str

    Name of the scraper handler that implements scraping logic

  id: str

    Scraper unique identifier

  name: str

    Scraper name

  priority: sneakpeek.queue.model.TaskPriority

    Default priority to enqueue scraper jobs with

  schedule: sneakpeek.scheduler.model.TaskSchedule

    Scraper schedule configuration

  schedule_crontab: str | None

    Must be defined if schedule equals to `CRONTAB`

  state: str | None

    Scraper state (might be useful to optimise scraping, e.g. only process pages that weren’t processed in the last jobs)

  timeout: datetime.timedelta | None

    Timeout for the single scraper job

class sneakpeek.scraper.model.ScraperContextABC

  Bases: `abc.ABC`

  Scraper context - helper class that implements basic HTTP client which logic can be extended by middleware that can preprocess request (e.g. Rate Limiter) and postprocess response (e.g. Response logger).

  async delete(url: str | list[str], *, headers: Optional[dict[str, str]] = None, max_concurrency: int = 0, return_exceptions: bool = False, **kwargs) → aiohttp.client_reqrep.ClientResponse | list[aiohttp.client_reqrep.ClientResponse | Exception]

    Make DELETE request to the given URL(s)

    Parameters

      - url (str | list[str]) – URL(s) to send DELETE request to

      - headers (HttpHeaders | None, optional) – HTTP headers. Defaults to None.

      - max_concurrency (int, optional) – Maximum number of concurrent requests. If set to 0 no limit is applied. Defaults to 0.

      - return_exceptions (bool, optional) – Whether to return exceptions instead of raising if there are multiple URLs provided. Defaults to False,

      - **kwargs – See aiohttp.delete() for the full list of arguments

    Returns

      HTTP response(s)

    Return type

      Response

  abstract async download_file(method: sneakpeek.scraper.model.HttpMethod, url: str, *, file_path: Optional[str] = None, file_process_fn: Optional[Callable[[str], Awaitable[Any]]] = None, headers: Optional[dict[str, str]] = None, **kwargs) → Union[str, Any]

    Perform HTTP request and save it to the specified file

    Parameters

      - method (HttpMethod) – HTTP request method to perform

      - url (str) – URL to send HTTP request to

      - file_path (str, optional) – Path of the file to save request to. If not specified temporary file name will be generated. Defaults to None.

      - file_process_fn (Callable[[str], Any], optional) – Function to process the file. If specified then function will be applied to the file and its result will be returned, the file will be removed after the function call. Defaults to None.

      - headers (HttpHeaders | None, optional) – HTTP headers. Defaults to None.

      - **kwargs – See aiohttp.request() for the full list of arguments

    Returns

      File path if file process function is not defined or file process function result otherwise

    Return type

      str | Any

  abstract async download_files(method: sneakpeek.scraper.model.HttpMethod, urls: list[str], *, file_paths: Optional[list[str]] = None, file_process_fn: Optional[Callable[[str], Awaitable[Any]]] = None, headers: Optional[dict[str, str]] = None, max_concurrency: int = 0, return_exceptions: bool = False, **kwargs) → list[typing.Union[str, typing.Any, Exception]]

    Perform HTTP requests and save them to the specified files

    Parameters

      - method (HttpMethod) – HTTP request method to perform

      - urls (list[str]) – URLs to send HTTP request to

      - file_paths (list[str], optional) – Path of the files to save requests to. If not specified temporary file names will be generated. Defaults to None.

      - file_process_fn (Callable[[str], Any], optional) – Function to process the file. If specified then function will be applied to the file and its result will be returned, the file will be removed after the function call. Defaults to None.

      - headers (HttpHeaders | None, optional) – HTTP headers. Defaults to None.

      - max_concurrency (int, optional) – Maximum number of concurrent requests. If set to 0 no limit is applied. Defaults to 0.

      - return_exceptions (bool, optional) – Whether to return exceptions instead of raising if there are multiple URLs provided. Defaults to False,

      - **kwargs – See aiohttp.request() for the full list of arguments

    Returns

      For each URL: file path if file process function is not defined or file process function result otherwise

    Return type

      list[str | Any | Exception]

  async get(url: str | list[str], *, headers: Optional[dict[str, str]] = None, max_concurrency: int = 0, return_exceptions: bool = False, **kwargs) → aiohttp.client_reqrep.ClientResponse | list[aiohttp.client_reqrep.ClientResponse | Exception]

    Make GET request to the given URL(s)

    Parameters

      - url (str | list[str]) – URL(s) to send GET request to

      - headers (HttpHeaders | None, optional) – HTTP headers. Defaults to None.

      - max_concurrency (int, optional) – Maximum number of concurrent requests. If set to 0 no limit is applied. Defaults to 0.

      - return_exceptions (bool, optional) – Whether to return exceptions instead of raising if there are multiple URLs provided. Defaults to False,

      - **kwargs – See aiohttp.get() for the full list of arguments

    Returns

      HTTP response(s)

    Return type

      Response

  async head(url: str | list[str], *, headers: Optional[dict[str, str]] = None, max_concurrency: int = 0, return_exceptions: bool = False, **kwargs) → aiohttp.client_reqrep.ClientResponse | list[aiohttp.client_reqrep.ClientResponse | Exception]

    Make HEAD request to the given URL(s)

    Parameters

      - url (str | list[str]) – URL(s) to send HEAD request to

      - headers (HttpHeaders | None, optional) – HTTP headers. Defaults to None.

      - max_concurrency (int, optional) – Maximum number of concurrent requests. If set to 0 no limit is applied. Defaults to 0.

      - return_exceptions (bool, optional) – Whether to return exceptions instead of raising if there are multiple URLs provided. Defaults to False,

      - **kwargs – See aiohttp.head() for the full list of arguments

    Returns

      HTTP response(s)

    Return type

      Response

  async options(url: str | list[str], *, headers: Optional[dict[str, str]] = None, max_concurrency: int = 0, return_exceptions: bool = False, **kwargs) → aiohttp.client_reqrep.ClientResponse | list[aiohttp.client_reqrep.ClientResponse | Exception]

    Make OPTIONS request to the given URL(s)

    Parameters

      - url (str | list[str]) – URL(s) to send OPTIONS request to

      - headers (HttpHeaders | None, optional) – HTTP headers. Defaults to None.

      - max_concurrency (int, optional) – Maximum number of concurrent requests. If set to 0 no limit is applied. Defaults to 0.

      - return_exceptions (bool, optional) – Whether to return exceptions instead of raising if there are multiple URLs provided. Defaults to False,

      - **kwargs – See aiohttp.options() for the full list of arguments

    Returns

      HTTP response(s)

    Return type

      Response

  async post(url: str | list[str], *, headers: Optional[dict[str, str]] = None, max_concurrency: int = 0, return_exceptions: bool = False, **kwargs) → aiohttp.client_reqrep.ClientResponse | list[aiohttp.client_reqrep.ClientResponse | Exception]

    Make POST request to the given URL(s)

    Parameters

      - url (str | list[str]) – URL(s) to send POST request to

      - headers (HttpHeaders | None, optional) – HTTP headers. Defaults to None.

      - max_concurrency (int, optional) – Maximum number of concurrent requests. If set to 0 no limit is applied. Defaults to 0.

      - return_exceptions (bool, optional) – Whether to return exceptions instead of raising if there are multiple URLs provided. Defaults to False,

      - **kwargs – See aiohttp.post() for the full list of arguments

    Returns

      HTTP response(s)

    Return type

      Response

  async put(url: str | list[str], *, headers: Optional[dict[str, str]] = None, max_concurrency: int = 0, return_exceptions: bool = False, **kwargs) → aiohttp.client_reqrep.ClientResponse | list[aiohttp.client_reqrep.ClientResponse | Exception]

    Make PUT request to the given URL(s)

    Parameters

      - url (str | list[str]) – URL(s) to send PUT request to

      - headers (HttpHeaders | None, optional) – HTTP headers. Defaults to None.

      - max_concurrency (int, optional) – Maximum number of concurrent requests. If set to 0 no limit is applied. Defaults to 0.

      - return_exceptions (bool, optional) – Whether to return exceptions instead of raising if there are multiple URLs provided. Defaults to False,

      - **kwargs – See aiohttp.put() for the full list of arguments

    Returns

      HTTP response(s)

    Return type

      Response

  abstract async request(method: sneakpeek.scraper.model.HttpMethod, url: str | list[str], *, headers: Optional[dict[str, str]] = None, max_concurrency: int = 0, return_exceptions: bool = False, **kwargs) → aiohttp.client_reqrep.ClientResponse | list[aiohttp.client_reqrep.ClientResponse | Exception]

    Perform HTTP request to the given URL(s)

    Parameters

      - method (HttpMethod) – HTTP request method to perform

      - url (str | list[str]) – URL(s) to send HTTP request to

      - headers (HttpHeaders | None, optional) – HTTP headers. Defaults to None.

      - max_concurrency (int, optional) – Maximum number of concurrent requests. If set to 0 no limit is applied. Defaults to 0.

      - return_exceptions (bool, optional) – Whether to return exceptions instead of raising if there are multiple URLs provided. Defaults to False,

      - **kwargs – See aiohttp.request() for the full list of arguments

    Returns

      HTTP response(s)

    Return type

      Response

  abstract async update_scraper_state(state: str) → sneakpeek.scraper.model.Scraper

    Update scraper state

    Parameters

      state (str) – State to persist

    Returns

      Updated scraper metadata

    Return type

      Scraper

class sneakpeek.scraper.model.ScraperHandler

  Bases: `abc.ABC`

  Abstract class that scraper logic handler must implement

  abstract property name: str

    Name of the handler

  abstract async run(context: sneakpeek.scraper.model.ScraperContextABC) → str

    Execute scraper logic

    Parameters

      context (ScraperContext) – Scraper context

    Returns

      scraper result that will be persisted in the storage (should be relatively small information to give sense on job result)

    Return type

      str

exception sneakpeek.scraper.model.ScraperNotFoundError(data=None)

  Bases: `fastapi_jsonrpc.BaseError`

class sneakpeek.scraper.model.ScraperRunnerABC

  Bases: `abc.ABC`

  abstract async run(handler: sneakpeek.scraper.model.ScraperHandler, scraper: sneakpeek.scraper.model.Scraper) → str
    Parameters

      - handler (ScraperHandler) – Scraper logic implementation

      - scraper (Scraper) – Scraper metadata

    Return type

      str

class sneakpeek.scraper.model.ScraperStorageABC

  Bases: `sneakpeek.scheduler.model.PeriodicTasksStorageABC`

exception sneakpeek.scraper.model.StorageIsReadOnlyError(data=None)

  Bases: `fastapi_jsonrpc.BaseError`

exception sneakpeek.scraper.model.UnknownScraperHandlerError(data=None)

  Bases: `fastapi_jsonrpc.BaseError`

class sneakpeek.queue.queue.Queue(storage: sneakpeek.queue.model.QueueStorageABC, dead_task_timeout: datetime.timedelta = datetime.timedelta(seconds=300))

  Bases: `sneakpeek.queue.model.QueueABC`

  Parameters

    - storage (sneakpeek.queue.model.QueueStorageABC) –

    - dead_task_timeout (datetime.timedelta) –

  Return type

    None

  async delete_old_tasks(keep_last: int = 50) → None

    Delete old historical tasks

    Parameters

      keep_last (int, optional) – How many tasks to keep. Defaults to 50.

    Return type

      None

  async dequeue() → sneakpeek.queue.model.Task | None

    Try to dequeue a task from the queue.

    Returns

      Dequeued task metadata

    Return type

      Task

  async enqueue(request: sneakpeek.queue.model.EnqueueTaskRequest) → sneakpeek.queue.model.Task

    Enqueue task

    Parameters

      request (EnqueueTaskRequest) – metadata of task to enqueue

    Returns

      Enqueued task metadata

    Return type

      Task

    Raises

      TaskHasActiveRunError – Error when there are other tasks with the same name in `PENDING` or `STARTED` state

  async get_queue_len() → int

    Get number of pending items in the queue

    Return type

      int

  async get_task_instance(task_id: int) → sneakpeek.queue.model.Task

    Get task instance by ID

    Parameters

      task_id (int) – Task ID

    Returns

      Task instance

    Return type

      Task

  async get_task_instances(task_name: str) → list[sneakpeek.queue.model.Task]

    Get task instances by task name

    Parameters

      task_name (str) – Task name

    Returns

      List of task instances

    Return type

      list[Task]

  async kill_dead_tasks() → list[sneakpeek.queue.model.Task]

    Kill dead tasks

    Returns

      List of killed dead tasks

    Return type

      list[Task]

  async ping_task(id: int) → sneakpeek.queue.model.Task

    Send a heartbeat for the task

    Parameters

      id (int) – Task ID

    Returns

      Updated task metadata

    Return type

      Task

    Raises

      - TaskNotFoundError – Raised if task doesn’t exist

      - TaskNotStartedError – Raised if task is still in the `PENDING` state

      - TaskPingFinishedError – Raised if task is in finished state (e.g. `DEAD`)

  async update_task(task: sneakpeek.queue.model.Task) → sneakpeek.queue.model.Task

    Update task metadata

    Parameters

      task (Task) – updated task metadata to save

    Returns

      Updated task metadata

    Return type

      Task

    Raises

      TaskNotFoundError – Raised when task doesn’t exist

class sneakpeek.queue.in_memory_storage.InMemoryQueueStorage

  Bases: `sneakpeek.queue.model.QueueStorageABC`

  In memory queue storage (should only be used for development purposes)

  Parameters

    redis (Redis) – Async redis client

  Return type

    None

  async delete_old_tasks(keep_last: int = 50) → None

    Delete old historical tasks

    Parameters

      keep_last (int, optional) – How many tasks to keep. Defaults to 50.

    Return type

      None

  async dequeue_task() → sneakpeek.queue.model.Task | None

    Try to dequeue pending task

    Returns

      First pending task or None if the queue is empty

    Return type

      Task | None

  async enqueue_task(task: sneakpeek.queue.model.Task) → sneakpeek.queue.model.Task

    Add a new task instance and put it into the queue

    Parameters

      task (Task) – task to add

    Returns

      Created task metadata

    Return type

      Task

  async get_queue_len() → int

    Get number of pending items in the queue

    Return type

      int

  async get_task_instance(id: int) → sneakpeek.queue.model.Task

    Get task instance by ID

    Parameters

      id (int) – Task ID

    Returns

      Found task metadata

    Return type

      Task

    Raises

      TaskNotFoundError – Raised when task doesn’t exist

  async get_task_instances(task_name: str) → list[sneakpeek.queue.model.Task]

    Get task instances by task name

    Parameters

      task_name (str) – Task name

    Returns

      List of task instances

    Return type

      list[Task]

  async get_tasks() → list[sneakpeek.queue.model.Task]

    Get all task instances

    Returns

      List of task instances

    Return type

      list[Task]

  async update_task(task: sneakpeek.queue.model.Task) → sneakpeek.queue.model.Task

    Update task metadata

    Parameters

      task (Task) – updated task metadata to save

    Returns

      Updated task metadata

    Return type

      Task

    Raises

      TaskNotFoundError – Raised when task doesn’t exist

class sneakpeek.queue.redis_storage.RedisQueueStorage(redis: redis.asyncio.client.Redis, task_ttl: datetime.timedelta = datetime.timedelta(days=7))

  Bases: `sneakpeek.queue.model.QueueStorageABC`

  Redis queue storage. Queue has two components: priority queue implemented by sorted set (ZADD and ZPOPMIN) and key (task name) values (set of task instances) set

  Parameters

    - redis (Redis) – Async redis client

    - task_ttl (timedelta) – TTL of the task record in the redis. Defaults to 7 days.

  Return type

    None

  async delete_old_tasks(keep_last: int = 50) → None

    Delete old historical tasks

    Parameters

      keep_last (int, optional) – How many tasks to keep. Defaults to 50.

    Return type

      None

  async dequeue_task() → sneakpeek.queue.model.Task | None

    Try to dequeue pending task

    Returns

      First pending task or None if the queue is empty

    Return type

      Task | None

  async enqueue_task(task: sneakpeek.queue.model.Task) → sneakpeek.queue.model.Task

    Add a new task instance and put it into the queue

    Parameters

      task (Task) – task to add

    Returns

      Created task metadata

    Return type

      Task

  async get_queue_len() → int

    Get number of pending items in the queue

    Return type

      int

  async get_task_instance(id: int) → sneakpeek.queue.model.Task

    Get task instance by ID

    Parameters

      id (int) – Task ID

    Returns

      Found task metadata

    Return type

      Task

    Raises

      TaskNotFoundError – Raised when task doesn’t exist

  async get_task_instances(task_name: str) → list[sneakpeek.queue.model.Task]

    Get task instances by task name

    Parameters

      task_name (str) – Task name

    Returns

      List of task instances

    Return type

      list[Task]

  async get_tasks() → list[sneakpeek.queue.model.Task]

    Get all task instances

    Returns

      List of task instances

    Return type

      list[Task]

  async update_task(task: sneakpeek.queue.model.Task) → sneakpeek.queue.model.Task

    Update task metadata

    Parameters

      task (Task) – updated task metadata to save

    Returns

      Updated task metadata

    Return type

      Task

    Raises

      TaskNotFoundError – Raised when task doesn’t exist

class sneakpeek.scheduler.scheduler.Scheduler(tasks_storage: sneakpeek.scheduler.model.PeriodicTasksStorageABC, lease_storage: sneakpeek.scheduler.model.LeaseStorageABC, queue: sneakpeek.queue.model.QueueABC, loop: Optional[asyncio.events.AbstractEventLoop] = None, tasks_poll_delay: datetime.timedelta = datetime.timedelta(seconds=5), lease_duration: datetime.timedelta = datetime.timedelta(seconds=60))

  Bases: `sneakpeek.scheduler.model.SchedulerABC`

  Parameters

    - tasks_storage (sneakpeek.scheduler.model.PeriodicTasksStorageABC) –

    - lease_storage (sneakpeek.scheduler.model.LeaseStorageABC) –

    - queue (sneakpeek.queue.model.QueueABC) –

    - loop (asyncio.events.AbstractEventLoop | None) –

    - tasks_poll_delay (datetime.timedelta) –

    - lease_duration (datetime.timedelta) –

  Return type

    None

  async update_tasks() → None

    Poll storage for all existing periodic tasks and update corresponding scheduler jobs

    Return type

      None

class sneakpeek.scheduler.in_memory_lease_storage.InMemoryLeaseStorage

  Bases: `sneakpeek.scheduler.model.LeaseStorageABC`

  In memory storage for leases. Should only be used for development purposes

  Return type

    None

  async maybe_acquire_lease(lease_name: str, owner_id: str, acquire_for: datetime.timedelta) → sneakpeek.scheduler.model.Lease | None

    Try to acquire lease (global lock).

    Parameters

      - lease_name (str) – Lease name (resource name to be locked)

      - owner_id (str) – ID of the acquirer (should be the same if you already have the lease and want to prolong it)

      - acquire_for (timedelta) – For how long lease will be acquired

    Returns

      Lease metadata if it was acquired, None otherwise

    Return type

      Lease | None

  async release_lease(lease_name: str, owner_id: str) → None

    Release lease (global lock)

    Parameters

      - lease_name (str) – Lease name (resource name to be unlocked)

      - owner_id (str) – ID of the acquirer

    Return type

      None

class sneakpeek.scheduler.redis_lease_storage.RedisLeaseStorage(redis: redis.asyncio.client.Redis)

  Bases: `sneakpeek.scheduler.model.LeaseStorageABC`

  Redis storage for leases. Should only be used for development purposes

  Parameters

    redis (Redis) – Async redis client

  Return type

    None

  async maybe_acquire_lease(lease_name: str, owner_id: str, acquire_for: datetime.timedelta) → sneakpeek.scheduler.model.Lease | None

    Try to acquire lease (global lock).

    Parameters

      - lease_name (str) – Lease name (resource name to be locked)

      - owner_id (str) – ID of the acquirer (should be the same if you already have the lease and want to prolong it)

      - acquire_for (timedelta) – For how long lease will be acquired

    Returns

      Lease metadata if it was acquired, None otherwise

    Return type

      Lease | None

  async release_lease(lease_name: str, owner_id: str) → None

    Release lease (global lock)

    Parameters

      - lease_name (str) – Lease name (resource name to be unlocked)

      - owner_id (str) – ID of the acquirer

    Return type

      None

class sneakpeek.scraper.context.ScraperContext(config: sneakpeek.scraper.model.ScraperConfig, middlewares: Optional[list[sneakpeek.scraper.model.Middleware]] = None, scraper_state: Optional[str] = None, update_scraper_state_func: Optional[Callable] = None)

  Bases: `sneakpeek.scraper.model.ScraperContextABC`

  Scraper context - helper class that implements basic HTTP client which logic can be extended by plugins that can preprocess request (e.g. Rate Limiter) and postprocess response (e.g. Response logger).

  Parameters

    - config (ScraperConfig) – Scraper configuration

    - middleware (list[Middleware] | None, optional) – List of available middleware. Defaults to None.

    - scraper_state (str | None, optional) – Scraper state. Defaults to None.

    - update_scraper_state_func (Callable | None, optional) – Function that update scraper state. Defaults to None.

    - middlewares (list[sneakpeek.scraper.model.Middleware] | None) –

  Return type

    None

  async download_file(method: sneakpeek.scraper.model.HttpMethod, url: str, *, file_path: Optional[str] = None, file_process_fn: Optional[Callable[[str], Awaitable[Any]]] = None, headers: Optional[dict[str, str]] = None, **kwargs) → Union[str, Any]

    Perform HTTP request and save it to the specified file

    Parameters

      - method (HttpMethod) – HTTP request method to perform

      - url (str) – URL to send HTTP request to

      - file_path (str, optional) – Path of the file to save request to. If not specified temporary file name will be generated. Defaults to None.

      - file_process_fn (Callable[[str], Any], optional) – Function to process the file. If specified then function will be applied to the file and its result will be returned, the file will be removed after the function call. Defaults to None.

      - headers (HttpHeaders | None, optional) – HTTP headers. Defaults to None.

      - **kwargs – See aiohttp.request() for the full list of arguments

    Returns

      File path if file process function is not defined or file process function result otherwise

    Return type

      str | Any

  async download_files(method: sneakpeek.scraper.model.HttpMethod, urls: list[str], *, file_paths: Optional[list[str]] = None, file_process_fn: Optional[Callable[[str], Awaitable[Any]]] = None, headers: Optional[dict[str, str]] = None, max_concurrency: int = 0, return_exceptions: bool = False, **kwargs) → list[typing.Union[str, typing.Any, Exception]]

    Perform HTTP requests and save them to the specified files

    Parameters

      - method (HttpMethod) – HTTP request method to perform

      - urls (list[str]) – URLs to send HTTP request to

      - file_paths (list[str], optional) – Path of the files to save requests to. If not specified temporary file names will be generated. Defaults to None.

      - file_process_fn (Callable[[str], Any], optional) – Function to process the file. If specified then function will be applied to the file and its result will be returned, the file will be removed after the function call. Defaults to None.

      - headers (HttpHeaders | None, optional) – HTTP headers. Defaults to None.

      - max_concurrency (int, optional) – Maximum number of concurrent requests. If set to 0 no limit is applied. Defaults to 0.

      - return_exceptions (bool, optional) – Whether to return exceptions instead of raising if there are multiple URLs provided. Defaults to False,

      - **kwargs – See aiohttp.request() for the full list of arguments

    Returns

      For each URL: file path if file process function is not defined or file process function result otherwise

    Return type

      list[str | Any | Exception]

  async request(method: sneakpeek.scraper.model.HttpMethod, url: str | list[str], *, headers: Optional[dict[str, str]] = None, max_concurrency: int = 0, return_exceptions: bool = False, **kwargs) → aiohttp.client_reqrep.ClientResponse | list[aiohttp.client_reqrep.ClientResponse | Exception]

    Perform HTTP request to the given URL(s)

    Parameters

      - method (HttpMethod) – HTTP request method to perform

      - url (str | list[str]) – URL(s) to send HTTP request to

      - headers (HttpHeaders | None, optional) – HTTP headers. Defaults to None.

      - max_concurrency (int, optional) – Maximum number of concurrent requests. If set to 0 no limit is applied. Defaults to 0.

      - return_exceptions (bool, optional) – Whether to return exceptions instead of raising if there are multiple URLs provided. Defaults to False,

      - **kwargs – See aiohttp.request() for the full list of arguments

    Returns

      HTTP response(s)

    Return type

      Response

  async update_scraper_state(state: str) → sneakpeek.scraper.model.Scraper

    Update scraper state

    Parameters

      state (str) – State to persist

    Returns

      Updated scraper metadata

    Return type

      Scraper

class sneakpeek.scraper.runner.ScraperRunner(scraper_storage: sneakpeek.scraper.model.ScraperStorageABC, middlewares: Optional[list[sneakpeek.scraper.model.Middleware]] = None, loop: Optional[asyncio.events.AbstractEventLoop] = None)

  Bases: `sneakpeek.scraper.model.ScraperRunnerABC`

  Default scraper runner implementation that is meant to be used in the Sneakpeek server

  Parameters

    - handlers (list[ScraperHandler]) – List of handlers that implement scraper logic

    - scrapers_storage (ScrapersStorage) – Sneakpeek scrapers storage implementation

    - jobs_storage (ScraperJobsStorage) – Sneakpeek jobs storage implementation

    - middlewares (list[Middleware] | None, optional) – List of middleware that will be used by scraper runner. Defaults to None.

    - scraper_storage (sneakpeek.scraper.model.ScraperStorageABC) –

    - loop (asyncio.events.AbstractEventLoop | None) –

  Return type

    None

  async run(handler: sneakpeek.scraper.model.ScraperHandler, scraper: sneakpeek.scraper.model.Scraper) → str
    Parameters

      - handler (ScraperHandler) – Scraper logic implementation

      - scraper (Scraper) – Scraper metadata

    Return type

      str

class sneakpeek.scraper.redis_storage.RedisScraperStorage(redis: redis.asyncio.client.Redis, is_read_only: bool = False)

  Bases: `sneakpeek.scraper.model.ScraperStorageABC`

  Parameters

    - redis (redis.asyncio.client.Redis) –

    - is_read_only (bool) –

  Return type

    None

class sneakpeek.scraper.in_memory_storage.InMemoryScraperStorage(initial_scrapers: Optional[list[sneakpeek.scraper.model.Scraper]] = None, is_read_only: bool = False)

  Bases: `sneakpeek.scraper.model.ScraperStorageABC`

  Parameters

    - initial_scrapers (list[sneakpeek.scraper.model.Scraper] | None) –

    - is_read_only (bool) –

  Return type

    None

class sneakpeek.scraper.dynamic_scraper_handler.DynamicScraperHandler

  Bases: `sneakpeek.scraper.model.ScraperHandler`

  property name: str

    Name of the handler

  async run(context: sneakpeek.scraper.model.ScraperContextABC) → str

    Execute scraper logic

    Parameters

      context (ScraperContext) – Scraper context

    Returns

      scraper result that will be persisted in the storage (should be relatively small information to give sense on job result)

    Return type

      str

class sneakpeek.middleware.base.BaseMiddleware

  Bases: `sneakpeek.scraper.model.Middleware`

  property name: str

    Name of the middleware

  async on_request(request: sneakpeek.scraper.model.Request, config: Optional[Any]) → sneakpeek.scraper.model.Request

    Function that is called on each (HTTP) request before its dispatched.

    Parameters

      - request (Request) – Request metadata

      - config (Any | None, optional) – Middleware configuration. Defaults to None.

    Returns

      Request metadata

    Return type

      Request

  async on_response(request: sneakpeek.scraper.model.Request, response: aiohttp.client_reqrep.ClientResponse, config: Optional[Any] = None) → Coroutine[Any, Any, aiohttp.client_reqrep.ClientResponse]

    Function that is called on each (HTTP) response before its result returned to the caller.

    Parameters

      - request (Request) – Request metadata

      - response (aiohttp.ClientResponse) – HTTP Response

      - config (Any | None, optional) – Middleware configuration. Defaults to None.

    Returns

      HTTP Response

    Return type

      aiohttp.ClientResponse

class sneakpeek.middleware.parser.ParserMiddleware

  Bases: `sneakpeek.middleware.base.BaseMiddleware`

  Parser middleware provides parsing utilities

  property name: str

    Name of the middleware

  regex(text: str, pattern: str, flags: re.RegexFlag = RegexFlag.None) → list[sneakpeek.middleware.parser.RegexMatch]

    Find matches in the text using regular expression

    Parameters

      - text (str) – Text to search in

      - pattern (str) – Regular expression

      - flags (re.RegexFlag, optional) – Regular expression flags. Defaults to re.UNICODE | re.MULTILINE | re.IGNORECASE.

    Returns

      Matches found in the text

    Return type

      list[RegexMatch]

class sneakpeek.middleware.parser.RegexMatch(full_match: str, groups: dict[str, str])

  Bases: `object`

  Regex match

  Parameters

    - full_match (str) –

    - groups (dict[str, str]) –

  Return type

    None

  full_match: str

    Full regular expression match

  groups: dict[str, str]

    Regular expression group matches

class sneakpeek.middleware.proxy_middleware.ProxyMiddleware(default_config: Optional[sneakpeek.middleware.proxy_middleware.ProxyMiddlewareConfig] = None)

  Bases: `sneakpeek.middleware.base.BaseMiddleware`

  Proxy middleware automatically sets proxy arguments for all HTTP requests.

  Parameters

    default_config (sneakpeek.middleware.proxy_middleware.ProxyMiddlewareConfig | None) –

  Return type

    None

  property name: str

    Name of the middleware

  async on_request(request: sneakpeek.scraper.model.Request, config: Optional[Any]) → sneakpeek.scraper.model.Request

    Function that is called on each (HTTP) request before its dispatched.

    Parameters

      - request (Request) – Request metadata

      - config (Any | None, optional) – Middleware configuration. Defaults to None.

    Returns

      Request metadata

    Return type

      Request

class sneakpeek.middleware.proxy_middleware.ProxyMiddlewareConfig(*, proxy: str | yarl.URL | None = None, proxy_auth: aiohttp.helpers.BasicAuth | None = None)

  Bases: `pydantic.main.BaseModel`

  Proxy middleware config

  Create a new model by parsing and validating input data from keyword arguments.

  Raises ValidationError if the input data cannot be parsed to form a valid model.

  Parameters

    - proxy (str | yarl.URL | None) –

    - proxy_auth (aiohttp.helpers.BasicAuth | None) –

  Return type

    None

  proxy: str | yarl.URL | None

    Proxy URL

  proxy_auth: aiohttp.helpers.BasicAuth | None

    Proxy authentication info to use

exception sneakpeek.middleware.rate_limiter_middleware.RateLimitedException

  Bases: `Exception`

  Request is rate limited because too many requests were made to the host

class sneakpeek.middleware.rate_limiter_middleware.RateLimitedStrategy(value)

  Bases: `enum.Enum`

  What to do if the request is rate limited

  THROW = 1

    Throw an exception

  WAIT = 2

    Wait until request is no longer rate limited

class sneakpeek.middleware.rate_limiter_middleware.RateLimiterMiddleware(default_config: Optional[sneakpeek.middleware.rate_limiter_middleware.RateLimiterMiddlewareConfig] = None)

  Bases: `sneakpeek.middleware.base.BaseMiddleware`

  Rate limiter implements leaky bucket algorithm to limit number of requests made to the hosts. If the request is rate limited it can either raise an exception or wait until the request won’t be limited anymore.

  Parameters

    default_config (sneakpeek.middleware.rate_limiter_middleware.RateLimiterMiddlewareConfig | None) –

  Return type

    None

  property name: str

    Name of the middleware

  async on_request(request: sneakpeek.scraper.model.Request, config: Optional[Any]) → sneakpeek.scraper.model.Request

    Function that is called on each (HTTP) request before its dispatched.

    Parameters

      - request (Request) – Request metadata

      - config (Any | None, optional) – Middleware configuration. Defaults to None.

    Returns

      Request metadata

    Return type

      Request

class sneakpeek.middleware.rate_limiter_middleware.RateLimiterMiddlewareConfig(*, max_requests: int = 60, rate_limited_strategy: sneakpeek.middleware.rate_limiter_middleware.RateLimitedStrategy = RateLimitedStrategy.WAIT, time_window: datetime.timedelta = datetime.timedelta(seconds=60))

  Bases: `pydantic.main.BaseModel`

  Rate limiter middleware configuration

  Create a new model by parsing and validating input data from keyword arguments.

  Raises ValidationError if the input data cannot be parsed to form a valid model.

  Parameters

    - max_requests (int) –

    - rate_limited_strategy (sneakpeek.middleware.rate_limiter_middleware.RateLimitedStrategy) –

    - time_window (datetime.timedelta) –

  Return type

    None

  max_requests: int

    Maximum number of allowed requests per host within time window

  rate_limited_strategy: sneakpeek.middleware.rate_limiter_middleware.RateLimitedStrategy

    What to do if the request is rate limited

  time_window: datetime.timedelta

    Time window to aggregate requests

class sneakpeek.middleware.requests_logging_middleware.RequestsLoggingMiddleware(default_config: Optional[sneakpeek.middleware.requests_logging_middleware.RequestsLoggingMiddlewareConfig] = None)

  Bases: `sneakpeek.scraper.model.Middleware`

  Requests logging middleware logs all requests being made and received responses.

  Parameters

    default_config (sneakpeek.middleware.requests_logging_middleware.RequestsLoggingMiddlewareConfig | None) –

  Return type

    None

  property name: str

    Name of the middleware

  async on_request(request: sneakpeek.scraper.model.Request, config: Optional[Any]) → sneakpeek.scraper.model.Request

    Function that is called on each (HTTP) request before its dispatched.

    Parameters

      - request (Request) – Request metadata

      - config (Any | None, optional) – Middleware configuration. Defaults to None.

    Returns

      Request metadata

    Return type

      Request

  async on_response(request: sneakpeek.scraper.model.Request, response: aiohttp.client_reqrep.ClientResponse, config: Optional[Any]) → aiohttp.client_reqrep.ClientResponse

    Function that is called on each (HTTP) response before its result returned to the caller.

    Parameters

      - request (Request) – Request metadata

      - response (aiohttp.ClientResponse) – HTTP Response

      - config (Any | None, optional) – Middleware configuration. Defaults to None.

    Returns

      HTTP Response

    Return type

      aiohttp.ClientResponse

class sneakpeek.middleware.requests_logging_middleware.RequestsLoggingMiddlewareConfig(*, log_request: bool = True, log_response: bool = True)

  Bases: `pydantic.main.BaseModel`

  Requests logging middleware config

  Create a new model by parsing and validating input data from keyword arguments.

  Raises ValidationError if the input data cannot be parsed to form a valid model.

  Parameters

    - log_request (bool) –

    - log_response (bool) –

  Return type

    None

  log_request: bool

    Whether to log request being made

  log_response: bool

    Whether to log response being made

class sneakpeek.middleware.robots_txt_middleware.RobotsTxtMiddleware(default_config: Optional[sneakpeek.middleware.robots_txt_middleware.RobotsTxtMiddlewareConfig] = None)

  Bases: `sneakpeek.middleware.base.BaseMiddleware`

  Robots.txt middleware can log and optionally block requests if they are disallowed by website robots.txt.

  Parameters

    default_config (sneakpeek.middleware.robots_txt_middleware.RobotsTxtMiddlewareConfig | None) –

  Return type

    None

  property name: str

    Name of the middleware

  async on_request(request: sneakpeek.scraper.model.Request, config: Optional[Any]) → sneakpeek.scraper.model.Request

    Function that is called on each (HTTP) request before its dispatched.

    Parameters

      - request (Request) – Request metadata

      - config (Any | None, optional) – Middleware configuration. Defaults to None.

    Returns

      Request metadata

    Return type

      Request

class sneakpeek.middleware.robots_txt_middleware.RobotsTxtMiddlewareConfig(*, violation_strategy: sneakpeek.middleware.robots_txt_middleware.RobotsTxtViolationStrategy = RobotsTxtViolationStrategy.LOG)

  Bases: `pydantic.main.BaseModel`

  robots.txt middleware configuration

  Create a new model by parsing and validating input data from keyword arguments.

  Raises ValidationError if the input data cannot be parsed to form a valid model.

  Parameters

    violation_strategy (sneakpeek.middleware.robots_txt_middleware.RobotsTxtViolationStrategy) –

  Return type

    None

exception sneakpeek.middleware.robots_txt_middleware.RobotsTxtViolationException

  Bases: `Exception`

  Exception which is raised if request is disallowed by website robots.txt

class sneakpeek.middleware.robots_txt_middleware.RobotsTxtViolationStrategy(value)

  Bases: `enum.Enum`

  What to do if the request is disallowed by website robots.txt

  LOG = 1

    Only log violation

  THROW = 2

    Raise an exception on vioalation

class sneakpeek.middleware.user_agent_injecter_middleware.UserAgentInjecterMiddleware(default_config: Optional[sneakpeek.middleware.user_agent_injecter_middleware.UserAgentInjecterMiddlewareConfig] = None)

  Bases: `sneakpeek.middleware.base.BaseMiddleware`

  This middleware automatically adds `User-Agent` header if it’s not present. It uses fake-useragent in order to generate fake real world user agents.

  Parameters

    default_config (sneakpeek.middleware.user_agent_injecter_middleware.UserAgentInjecterMiddlewareConfig | None) –

  Return type

    None

  property name: str

    Name of the middleware

  async on_request(request: sneakpeek.scraper.model.Request, config: Optional[Any]) → sneakpeek.scraper.model.Request

    Function that is called on each (HTTP) request before its dispatched.

    Parameters

      - request (Request) – Request metadata

      - config (Any | None, optional) – Middleware configuration. Defaults to None.

    Returns

      Request metadata

    Return type

      Request

class sneakpeek.middleware.user_agent_injecter_middleware.UserAgentInjecterMiddlewareConfig(*, use_external_data: bool = True, browsers: list[str] = ['chrome', 'edge', 'firefox', 'safari', 'opera'])

  Bases: `pydantic.main.BaseModel`

  Middleware configuration

  Create a new model by parsing and validating input data from keyword arguments.

  Raises ValidationError if the input data cannot be parsed to form a valid model.

  Parameters

    - use_external_data (bool) –

    - browsers (list[str]) –

  Return type

    None

  browsers: list[str]

    List of browsers which are used to generate user agents

  use_external_data: bool

    Whether to use external data as a fallback

sneakpeek.api.create_api(scraper_storage: sneakpeek.scraper.model.ScraperStorageABC, queue: sneakpeek.queue.model.QueueABC, handlers: list[sneakpeek.scraper.model.ScraperHandler], session_logger_handler: Optional[logging.Handler] = None) → fastapi_jsonrpc.API

  Create JsonRPC API (FastAPI is used under the hood)

  Parameters

    - storage (Storage) – Sneakpeek storage implementation

    - queue (Queue) – Sneakpeek queue implementation

    - handlers (list[ScraperHandler]) – List of handlers that implement scraper logic

    - scraper_storage (sneakpeek.scraper.model.ScraperStorageABC) –

    - session_logger_handler (Optional[logging.Handler]) –

  Return type

    fastapi_jsonrpc.API

sneakpeek.api.get_api_entrypoint(scraper_storage: sneakpeek.scraper.model.ScraperStorageABC, queue: sneakpeek.queue.model.QueueABC, handlers: list[sneakpeek.scraper.model.ScraperHandler], session_logger_handler: Optional[sneakpeek.session_loggers.base.SessionLogger] = None) → fastapi_jsonrpc.Entrypoint

  Create public JsonRPC API entrypoint (mostly mimics storage and queue API)

  Parameters

    - storage (Storage) – Sneakpeek storage implementation

    - queue (Queue) – Sneakpeek queue implementation

    - handlers (list[ScraperHandler]) – List of handlers that implement scraper logic

    - scraper_storage (sneakpeek.scraper.model.ScraperStorageABC) –

    - session_logger_handler (Optional[sneakpeek.session_loggers.base.SessionLogger]) –

  Returns

    FastAPI JsonRPC entrypoint

  Return type

    jsonrpc.Entrypoint

class sneakpeek.logging.TaskContextInjectingFilter(name='')

  Bases: `logging.Filter`

  Scraper context filter which automatically injects scraper and scraper job IDs to the logging metadata.

  Example of usage:

      logger = logging.getLogger()
      handler = logging.StreamHandler()
      handler.addFilter(ScraperContextInjectingFilter())
      logger.addHandler(handler)

  Initialize a filter.

  Initialize with the name of the logger which, together with its children, will have its events allowed through the filter. If no name is specified, allow every event.

  filter(record: logging.LogRecord) → bool

    Injects task metadata into log record:

    - `task_id` - Task ID

    - `task_name` - Task name

    - `task_handler` - Task handler

    Parameters

      record (logging.LogRecord) – Log record to inject metadata into

    Returns

      Always True

    Return type

      bool

sneakpeek.logging.configure_logging(level: int = 20, session_logger_handler: Optional[logging.Handler] = None)

  Helper function to configure logging:

  - Adds console logger to the root logger

  - Adds scraper context injector filter to the console logger

  - Configures console formatting to use scraper metadata

  Parameters

    - level (int, optional) – Minimum logging level. Defaults to logging.INFO.

    - session_logger_handler (Optional[logging.Handler]) –

sneakpeek.logging.task_context(task: sneakpeek.queue.model.Task) → None

  Initialize scraper job logging context which automatically adds scraper and scraper job IDs to the logging metadata

  Parameters

    - scraper_job (ScraperJob) – Scraper job definition

    - task (sneakpeek.queue.model.Task) –

  Return type

    None

sneakpeek.metrics.count_invocations(subsystem: str)

  Decorator for measuring number of function invocations (works for both sync and async functions).

      @count_invocations(subsytem="my subsystem")
      def my_awesome_func():
          ...

  This will export following Prometheus counter metrics:

      # Total number of invocations
      sneakpeek_invocations{subsystem="my subsystem", method="my_awesome_func", type="total", error=""}
      # Total number of successful invocations (ones that haven't thrown an exception)
      sneakpeek_invocations{subsystem="my subsystem", method="my_awesome_func", type="success", error=""}
      # Total number of failed invocations (ones that have thrown an exception)
      sneakpeek_invocations{subsystem="my subsystem", method="my_awesome_func", type="error", error="<Exception class name>"}

  Parameters

    subsystem (str) – Subsystem name to be used in the metric annotation

sneakpeek.metrics.measure_latency(subsystem: str)

  Decorator for measuring latency of the function (works for both sync and async functions).

      @measure_latency(subsytem="my subsystem")
      def my_awesome_func():
          ...

  This will export following Prometheus histogram metric:

      sneakpeek_latency{subsystem="my subsystem", method="my_awesome_func"}

  Parameters

    subsystem (str) – Subsystem name to be used in the metric annotation