---
title: "Research Report for Executor Implementation (Issue 17)"
author: "@khive-researcher"
date: "2025-05-20"
status: "draft"
issue: "https://github.com/khive-ai/lionfuncs/issues/17"
---

## 1. Introduction

This report outlines a research-backed approach to implement the `Executor`
class described in GitHub Issue #17 for the `khive-ai/lionfuncs` project. The
goal is to leverage existing `lionfuncs` primitives for concurrency and
rate-limiting, specifically addressing "tokens per interval" and "requests per
interval" limits with a replenishment mechanism, and integrating an
`APICalling`-like event structure.

## 2. Analysis of `lionfuncs` Primitives

Based on a review of
[`src/lionfuncs/concurrency.py`](src/lionfuncs/concurrency.py:0),
[`src/lionfuncs/network/primitives.py`](src/lionfuncs/network/primitives.py:0),
and [`src/lionfuncs/async_utils.py`](src/lionfuncs/async_utils.py:0), the
following primitives are most relevant:

- **[`WorkQueue`](src/lionfuncs/concurrency.py:341) (`lionfuncs.concurrency`):**
  Suitable for managing a queue of incoming API call tasks. It wraps
  `BoundedQueue` and provides a worker processing model.
- **[`CapacityLimiter`](src/lionfuncs/concurrency.py:487)
  (`lionfuncs.concurrency`):** Designed to limit the total number of concurrent
  operations. This can be used for the overall concurrency limit of active API
  calls.
- **[`TokenBucketRateLimiter`](src/lionfuncs/network/primitives.py:256)
  (`lionfuncs.network.primitives`):** This primitive is well-suited for
  implementing "N items per interval" limits. It uses a token bucket algorithm
  with an internal `_refill` mechanism that replenishes tokens based on elapsed
  time, rate, and period. This can be used for both "requests per interval" and
  "API tokens per interval" limits by instantiating it accordingly.
- **[`Throttle`](src/lionfuncs/async_utils.py:45) (`lionfuncs.async_utils`):**
  This primitive ensures a minimum time period between calls. It is simpler than
  `TokenBucketRateLimiter` and does not maintain a "bucket" of calls/tokens. For
  the requirements of Issue #17 (N requests/tokens per interval with
  replenishment), `TokenBucketRateLimiter` is a more direct fit.
- **[`Lock`](src/lionfuncs/concurrency.py:430),
  [`Semaphore`](src/lionfuncs/concurrency.py:459),
  [`Event`](src/lionfuncs/concurrency.py:537),
  [`Condition`](src/lionfuncs/concurrency.py:555) (`lionfuncs.concurrency`):**
  General synchronization primitives. `TokenBucketRateLimiter` already uses
  `Lock`. `Semaphore` could be an alternative for custom bucket implementations,
  but `TokenBucketRateLimiter` is preferred as an existing, specialized
  primitive.

The `APICalling` event structure mentioned in the issue is derived from the
user-provided example code (GitHub Issue #17 comment) and is not currently a
primitive within `lionfuncs`.

## 3. Proposed `Executor` Implementation Strategy

The new `lionfuncs.Executor` can be implemented by orchestrating the above
primitives:

### 3.1. Core Components:

1. **Task Queue:** An instance of
   [`WorkQueue`](src/lionfuncs/concurrency.py:341) to hold tasks. Each task
   would represent an API call to be made, likely encapsulating data similar to
   the example `APICalling` event.
2. **Overall Concurrency Limiter:** An instance of
   [`CapacityLimiter`](src/lionfuncs/concurrency.py:487) to control the maximum
   number of API calls that can be simultaneously in the "calling" phase.
3. **Requests-Per-Interval Limiter:** An instance of
   [`TokenBucketRateLimiter`](src/lionfuncs/network/primitives.py:256)
   configured for the desired requests per interval (e.g., `rate=X`, `period=Y`
   seconds).
4. **Tokens-Per-Interval Limiter:** Another instance of
   [`TokenBucketRateLimiter`](src/lionfuncs/network/primitives.py:256)
   configured for the desired API tokens per interval (e.g., `rate=M_tokens`,
   `period=N` seconds).

### 3.2. Execution Flow for an API Call Task:

A worker processing a task from the `WorkQueue` would:

1. **Acquire Global Concurrency Slot:** Attempt to acquire a slot from the main
   `CapacityLimiter`. This is a blocking operation if the limit is reached.
   ```python
   # Conceptual
   async with global_capacity_limiter:
       # Proceed to next steps
   ```
2. **Acquire Request Rate Limit Token:** Attempt to acquire one "request" token
   from the `requests_rate_limiter`
   ([`TokenBucketRateLimiter`](src/lionfuncs/network/primitives.py:256)). The
   `acquire(tokens=1)` method returns a `wait_time`. If `wait_time > 0`, the
   worker must `await asyncio.sleep(wait_time)`.
   ```python
   # Conceptual
   wait_time = await requests_rate_limiter.acquire(tokens=1)
   if wait_time > 0:
       await asyncio.sleep(wait_time)
   ```
3. **Acquire API Token Rate Limit Tokens:** If the API call consumes specific
   API provider tokens (e.g., OpenAI tokens), attempt to acquire the required
   number of tokens (e.g., `num_api_tokens_needed`) from the
   `api_tokens_rate_limiter`
   ([`TokenBucketRateLimiter`](src/lionfuncs/network/primitives.py:256)).
   Similar to step 2, if `acquire()` returns a `wait_time > 0`, the worker must
   sleep.
   ```python
   # Conceptual
   wait_time = await api_tokens_rate_limiter.acquire(tokens=num_api_tokens_needed)
   if wait_time > 0:
       await asyncio.sleep(wait_time)
   ```
4. **Make the API Call:** Execute the actual HTTP request.
5. **Release Global Concurrency Slot:** The `CapacityLimiter` slot is released
   automatically when exiting the `async with` block. Tokens from
   `TokenBucketRateLimiter` are consumed by its `acquire` logic and replenished
   internally over time by subsequent calls to `_refill` (triggered by
   `acquire`).

### 3.3. Replenishment Mechanism:

The [`TokenBucketRateLimiter`](src/lionfuncs/network/primitives.py:256)
inherently handles replenishment. Its `_refill()` method is called internally
when `acquire()` is invoked. This method calculates and adds tokens based on the
elapsed time since the last refill, the configured `rate`, and `period`. This is
a continuous, gradual replenishment model.

This differs slightly from the `RateLimitedAPIProcessor.start_replenishing`
method in the issue's example code, which appears to perform a full reset of
available tokens/requests to their maximum capacity at fixed intervals using an
`asyncio.Task`.

**Recommendation on Replenishment:** The `TokenBucketRateLimiter`'s gradual
refill is generally a smoother and more standard approach for token buckets,
preventing thundering herd issues that can arise from periodic full resets. It's
recommended to use the existing `TokenBucketRateLimiter` behavior.

### 3.4. `APICalling` Event Structure Integration:

The new `Executor` should manage an event structure similar to the `APICalling`
model provided in the issue's example code.

- When a task is submitted, an `APICalling`-like event object can be created
  with an initial status (e.g., `QUEUED`).
- As the task progresses through the `Executor`'s pipeline (waiting for
  capacity, rate limits, making the call), its status and other relevant
  information (e.g., wait times, token consumption if applicable, response data,
  errors) should be updated on this event object.
- The `Executor` could provide ways to subscribe to these events or query their
  status.
- The `APICalling` object itself would need to be defined, potentially within
  `lionfuncs.network.events` or alongside the new `Executor`. It should include
  fields for payload, headers, endpoint details, status, timestamps, and
  potentially error information and token usage metrics.

## 4. Comparison with `RateLimitedAPIProcessor` (from Issue Comment)

- **Replenishment:** `RateLimitedAPIProcessor` uses a dedicated `asyncio.Task`
  (`start_replenishing`) to periodically reset `available_request` and
  `available_token` counts to their full limits. The proposed solution uses
  `TokenBucketRateLimiter` which has an integrated, gradual refill mechanism.
- **Permission Check:** `RateLimitedAPIProcessor.request_permission` directly
  decrements available token/request counters. The proposed solution uses
  `TokenBucketRateLimiter.acquire()`, which handles token consumption and
  returns a wait time if necessary.
- **Concurrency:** `RateLimitedAPIProcessor` uses an `asyncio.Semaphore` (via
  its `Processor` superclass and its own `_concurrency_sem`). The proposed
  solution uses `lionfuncs.CapacityLimiter` for overall concurrency, which is a
  more specialized primitive for this purpose.

## 5. Configuration

The new `Executor` should be configurable with parameters similar to those in
the `iModel`'s initialization of `RateLimitedAPIExecutor`:

- `queue_capacity` (for the internal `WorkQueue`)
- `concurrency_limit` (for the `CapacityLimiter`)
- `requests_per_interval_rate`, `requests_per_interval_period` (for the request
  `TokenBucketRateLimiter`)
- `tokens_per_interval_rate`, `tokens_per_interval_period` (for the API token
  `TokenBucketRateLimiter`)
- `max_tokens_bucket_capacity_requests` (optional, for request
  `TokenBucketRateLimiter`)
- `max_tokens_bucket_capacity_api` (optional, for API token
  `TokenBucketRateLimiter`)

## 6. Recommendations and Options

1. **Primary Approach:** Utilize `WorkQueue`, `CapacityLimiter`, and two
   instances of `TokenBucketRateLimiter` as outlined above. This leverages
   existing, tested `lionfuncs` primitives effectively.
2. **`APICalling` Event:** Define a new `APICallingEvent` (or similar) class
   within `lionfuncs`, inspired by the example, to track the lifecycle of API
   requests through the `Executor`.
3. **Replenishment Model:** Stick with the `TokenBucketRateLimiter`'s gradual
   replenishment model. If a "periodic full reset" is a strict requirement,
   `TokenBucketRateLimiter` might need adaptation, or a custom solution closer
   to the original `start_replenishing` (using `Semaphore` and an
   `asyncio.Task`) would be necessary. However, the former is generally
   preferred.
4. **Extensibility:** Consider if the `TokenBucketRateLimiter` could benefit
   from an optional mode or subclass that supports the "periodic full reset" if
   this pattern is common, though it's less standard for token buckets.

## 7. Conclusion

Implementing the `Executor` using `lionfuncs.WorkQueue`,
`lionfuncs.CapacityLimiter`, and `lionfuncs.TokenBucketRateLimiter` provides a
robust and idiomatic solution for managing rate-limited API calls. The
`TokenBucketRateLimiter` is particularly well-suited for handling both "requests
per interval" and "tokens per interval" limits with its built-in replenishment.
The `APICalling` event structure from the example can be adapted and integrated
to provide good observability.

## Appendix A: Search Provenance

- No external searches (Perplexity/Exa) were conducted for this report. Research
  was based on the provided `lionfuncs` source code and GitHub issue details.
- GitHub Issue #17: `(gh:khive-ai/lionfuncs#17)`
- `iModel` and `RateLimitedAPIExecutor` example code:
  `(gh:khive-ai/lionfuncs#17-comment-2895224067)`
- [`lionfuncs.concurrency`](src/lionfuncs/concurrency.py:0)
- [`lionfuncs.network.primitives`](src/lionfuncs/network/primitives.py:0)
- [`lionfuncs.async_utils`](src/lionfuncs/async_utils.py:0)
