This file is a merged representation of a subset of the codebase, containing files not matching ignore patterns, combined into a single document by Repomix. The content has been processed where empty lines have been removed.

================================================================
File Summary
================================================================

Purpose:
--------
This file contains a packed representation of the entire repository's contents.
It is designed to be easily consumable by AI systems for analysis, code review,
or other automated processes.

File Format:
------------
The content is organized as follows:
1. This summary section
2. Repository information
3. Directory structure
4. Multiple file entries, each consisting of:
  a. A separator line (================)
  b. The file path (File: path/to/file)
  c. Another separator line
  d. The full contents of the file
  e. A blank line

Usage Guidelines:
-----------------
- This file should be treated as read-only. Any changes should be made to the
  original repository files, not this packed version.
- When processing this file, use the file path to distinguish
  between different files in the repository.
- Be aware that this file may contain sensitive information. Handle it with
  the same level of security as you would the original repository.

Notes:
------
- Some files may have been excluded based on .gitignore rules and Repomix's configuration
- Binary files are not included in this packed representation. Please refer to the Repository Structure section for a complete list of file paths, including binary files
- Files matching these patterns are excluded: .specstory/**/*.md, .venv/**, _private/**, CLEANUP.txt, **/*.json, *.lock
- Files matching patterns in .gitignore are excluded
- Files matching default ignore patterns are excluded
- Empty lines have been removed from all files

Additional Info:
----------------

================================================================
Directory Structure
================================================================
.cursor/
  rules/
    0project.mdc
    filetree.mdc
.github/
  workflows/
    push.yml
    release.yml
examples/
  basic_usage.py
src/
  opero/
    concurrency/
      __init__.py
      pool.py
    core/
      __init__.py
      cache.py
      fallback.py
      rate_limit.py
      retry.py
    decorators/
      __init__.py
      opero.py
      opmap.py
    utils/
      __init__.py
      async_utils.py
      logging.py
    __init__.py
    exceptions.py
tests/
  test_core.py
  test_decorators.py
  test_package.py
.gitignore
.pre-commit-config.yaml
CHANGELOG.md
cleanup.py
get-pip.py
LICENSE
package.toml
PROGRESS.md
pyproject.toml
README.md
TODO.md
twat-cache.txt
twat-mp.txt

================================================================
Files
================================================================

================
File: .cursor/rules/0project.mdc
================
---
title: Opero Project Rules
glob: "**/*.py"
---

# Opero Project Rules

## Project Overview

Opero is a Python library for orchestrating function execution with retries, fallbacks, and concurrency control. It provides a flexible way to handle errors and ensure reliable execution of functions.

## Code Style

- Follow PEP 8 for code style
- Use type hints for all functions and methods
- Write clear docstrings for all public functions and classes
- Use f-strings for string formatting
- Keep functions and methods small and focused
- Extract complex logic into helper methods
- Use descriptive variable names

## Error Handling

- Use appropriate exception types
- Handle errors gracefully with retries and fallbacks
- Log errors with appropriate context
- Provide helpful error messages

## Testing

- Write unit tests for all functionality
- Test edge cases and error conditions
- Use pytest for testing
- Aim for high test coverage

## Logging

- Use the built-in logging module
- Log at appropriate levels (debug, info, warning, error)
- Include context in log messages
- Configure logging in a way that doesn't interfere with applications using the library

## Performance

- Optimize critical paths
- Minimize overhead in the orchestration layer
- Use async/await for I/O-bound operations
- Use appropriate concurrency mechanisms for CPU-bound operations

================
File: .cursor/rules/filetree.mdc
================
---
description: File tree of the project
globs: 
---
[ 992]  .
├── [  96]  .cursor
│   └── [ 128]  rules
│       ├── [1.3K]  0project.mdc
│       └── [2.9K]  filetree.mdc
├── [  96]  .github
│   └── [ 128]  workflows
│       ├── [2.7K]  push.yml
│       └── [1.4K]  release.yml
├── [4.4K]  .gitignore
├── [ 470]  .pre-commit-config.yaml
├── [  96]  .specstory
│   └── [ 384]  history
│       ├── [2.7K]  .what-is-this.md
│       ├── [574K]  2025-03-04_03-16-comprehensive-plan-for-opero-package-implementation.md
│       ├── [4.1K]  2025-03-04_05-40-cleanup-analysis-and-todo-update.md
│       ├── [157K]  2025-03-04_06-07-implementing-todo-md-phases-1-and-2.md
│       ├── [129K]  2025-03-04_07-28-implementing-todo-md-phases-1-and-2.md
│       ├── [ 67K]  2025-03-04_07-59-project-maintenance-and-documentation-update.md
│       ├── [198K]  2025-03-04_08-40-managing-todo-list-tasks.md
│       ├── [138K]  2025-03-05_12-35-improving-opero-api-with-new-decorators.md
│       ├── [2.2K]  2025-03-05_13-36-repository-analysis-and-todo-update.md
│       └── [574K]  2025-03-05_14-22-implementing-todo-md-for-opero-library.md
├── [2.0K]  CHANGELOG.md
├── [1.5K]  CLEANUP.txt
├── [1.0K]  LICENSE
├── [2.2K]  PROGRESS.md
├── [7.0K]  README.md
├── [349K]  REPO_CONTENT.txt
├── [2.3K]  TODO.md
├── [ 13K]  cleanup.py
├── [ 224]  dist
│   └── [   0]  .gitkeep
├── [  96]  examples
│   └── [9.6K]  basic_usage.py
├── [2.2M]  get-pip.py
├── [ 426]  package.toml
├── [6.0K]  pyproject.toml
├── [ 128]  src
│   └── [ 320]  opero
│       ├── [1.5K]  __init__.py
│       ├── [ 160]  concurrency
│       │   ├── [ 439]  __init__.py
│       │   └── [ 11K]  pool.py
│       ├── [ 256]  core
│       │   ├── [ 765]  __init__.py
│       │   ├── [2.8K]  cache.py
│       │   ├── [6.4K]  fallback.py
│       │   ├── [3.4K]  rate_limit.py
│       │   └── [3.8K]  retry.py
│       ├── [ 192]  decorators
│       │   ├── [ 348]  __init__.py
│       │   ├── [3.8K]  opero.py
│       │   └── [4.7K]  opmap.py
│       ├── [ 563]  exceptions.py
│       └── [ 192]  utils
│           ├── [ 460]  __init__.py
│           ├── [2.1K]  async_utils.py
│           └── [1.6K]  logging.py
├── [ 192]  tests
│   ├── [4.6K]  test_core.py
│   ├── [4.3K]  test_decorators.py
│   └── [ 139]  test_package.py
├── [194K]  twat-cache.txt
├── [ 78K]  twat-mp.txt
└── [ 91K]  uv.lock

16 directories, 50 files

================
File: .github/workflows/push.yml
================
name: Build & Test
on:
  push:
    branches: [main]
    tags-ignore: ["v*"]
  pull_request:
    branches: [main]
  workflow_dispatch:
permissions:
  contents: write
  id-token: write
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
jobs:
  quality:
    name: Code Quality
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Run Ruff lint
        uses: astral-sh/ruff-action@v3
        with:
          version: "latest"
          args: "check --output-format=github"
      - name: Run Ruff Format
        uses: astral-sh/ruff-action@v3
        with:
          version: "latest"
          args: "format --check --respect-gitignore"
  test:
    name: Run Tests
    needs: quality
    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
        os: [ubuntu-latest]
      fail-fast: true
    runs-on: ${{ matrix.os }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install UV
        uses: astral-sh/setup-uv@v5
        with:
          version: "latest"
          python-version: ${{ matrix.python-version }}
          enable-cache: true
          cache-suffix: ${{ matrix.os }}-${{ matrix.python-version }}
      - name: Install test dependencies
        run: |
          uv pip install --system --upgrade pip
          uv pip install --system ".[test]"
      - name: Run tests with Pytest
        run: uv run pytest -n auto --maxfail=1 --disable-warnings --cov-report=xml --cov-config=pyproject.toml --cov=src/opero --cov=tests tests/
      - name: Upload coverage report
        uses: actions/upload-artifact@v4
        with:
          name: coverage-${{ matrix.python-version }}-${{ matrix.os }}
          path: coverage.xml
  build:
    name: Build Distribution
    needs: test
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install UV
        uses: astral-sh/setup-uv@v5
        with:
          version: "latest"
          python-version: "3.12"
          enable-cache: true
      - name: Install build tools
        run: uv pip install build hatchling hatch-vcs
      - name: Build distributions
        run: uv run python -m build --outdir dist
      - name: Upload distribution artifacts
        uses: actions/upload-artifact@v4
        with:
          name: dist-files
          path: dist/
          retention-days: 5

================
File: .github/workflows/release.yml
================
name: Release
on:
  push:
    tags: ["v*"]
permissions:
  contents: write
  id-token: write
jobs:
  release:
    name: Release to PyPI
    runs-on: ubuntu-latest
    environment:
      name: pypi
      url: https://pypi.org/p/opero
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install UV
        uses: astral-sh/setup-uv@v5
        with:
          version: "latest"
          python-version: "3.12"
          enable-cache: true
      - name: Install build tools
        run: uv pip install build hatchling hatch-vcs
      - name: Build distributions
        run: uv run python -m build --outdir dist
      - name: Verify distribution files
        run: |
          ls -la dist/
          test -n "$(find dist -name '*.whl')" || (echo "Wheel file missing" && exit 1)
          test -n "$(find dist -name '*.tar.gz')" || (echo "Source distribution missing" && exit 1)
      - name: Publish to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          password: ${{ secrets.PYPI_TOKEN }}
      - name: Create GitHub Release
        uses: softprops/action-gh-release@v1
        with:
          files: dist/*
          generate_release_notes: true
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

================
File: examples/basic_usage.py
================
class APIResponse:
class ProcessingResult:
def configure_logging(level=logging.INFO, format=None):
    logging.basicConfig(
    return logging.getLogger("opero")
class FallbackError(Exception):
    def __init__(self, message: str, errors: list[Exception]):
        super().__init__(message)
def opero(
    def decorator(func):
        if asyncio.iscoroutinefunction(func):
            async def async_wrapper(*args, **kwargs):
                return await func(*args, **kwargs)
            def wrapper(*args, **kwargs):
                return func(*args, **kwargs)
def opmap(
logger = configure_logging(level=logging.INFO, verbose=True)
@opero(
def fibonacci(n: int) -> int:
    logger.info(f"Calculating Fibonacci({n})")
    return fibonacci(n - 1) + fibonacci(n - 2)
def call_api(data: dict[str, Any], api_key: str = "primary_key") -> dict[str, Any]:
    logger.info(f"Calling API with key {api_key}")
        if time.time() % 2 < 1:  # Random failure
            raise ConnectionError(msg)
@opmap(
def process_item(item: int) -> ProcessingResult:
    start_time = time.time()
    time.sleep(random.uniform(0.1, 0.3))
    return ProcessingResult(
        processing_time=time.time() - start_time,
async def fetch_url(url: str) -> dict[str, Any]:
    logger.info(f"Fetching data from {url}")
    await asyncio.sleep(0.2)
    if time.time() % 3 < 1 and "example.com" in url:
    return {"url": url, "data": f"Data from {url}", "timestamp": time.time()}
async def fetch_multiple_urls(url: str) -> dict[str, Any]:
    logger.info(f"Fetching URL {url}")
    return await fetch_url(url)
async def rate_limited_operation(query: str) -> dict[str, Any]:
    logger.info(f"Processing query: {query}")
    await asyncio.sleep(0.1)  # Simulate processing time
        "timestamp": time.time(),
    return await asyncio.sleep(0, result)  # Make the result awaitable
async def resilient_operation(
    logger.info(f"Attempting operation on endpoint: {current_endpoint}")
            await asyncio.wait_for(asyncio.sleep(0.2), timeout=timeout)
        if current_endpoint == "primary" and random.random() < 0.8:
        result = APIResponse(
        raise TimeoutError(msg)
async def main():
    for n in range(5):
        start = time.time()
        fibonacci(n)
        time.time() - start
        await call_api(data)
    items = list(range(5))
    results = [process_item(item) for item in items]  # Process each item individually
    results = await asyncio.gather(*[rate_limited_operation(q) for q in queries])
        await resilient_operation(
    asyncio.run(main())

================
File: src/opero/concurrency/__init__.py
================


================
File: src/opero/concurrency/pool.py
================
T = TypeVar("T")  # Input type
R = TypeVar("R")  # Return type
logger = logging.getLogger(__name__)
def get_parallel_executor(
        def process_executor(func: Callable[[T], R], items: Iterable[T]) -> list[R]:
            return list(
                pmap(func)(
        def thread_executor(func: Callable[[T], R], items: Iterable[T]) -> list[R]:
            with ThreadPool(nodes=workers) as pool:
                return list(pool.map(func, items))
        def async_executor(func: Callable[[T], R], items: Iterable[T]) -> list[R]:
                loop = asyncio.get_running_loop()
                return loop.run_until_complete(
                    amap(func)(
                return asyncio.run(
        def async_process_executor(
                    apmap(func)(
        raise ValueError(msg)
def get_parallel_map_decorator(
        return lambda func: lambda items, **kw: list(
        def decorator(func: Callable[[T], R]) -> Callable[[Iterable[T]], list[R]]:
            def wrapper(items: Iterable[T], **kw: Any) -> list[R]:
                    if asyncio.iscoroutinefunction(func):
                        async def run_async(item: T) -> R:
                            return await func(item)
                        def async_wrapper(item: T) -> R:
                                return asyncio.run(run_async(item))
                                return loop.run_until_complete(run_async(item))
                        return list(pool.map(async_wrapper, items))
            decorated = amap(func)
                        decorated(
            decorated = apmap(func)

================
File: src/opero/core/__init__.py
================


================
File: src/opero/core/cache.py
================
P = ParamSpec("P")
R = TypeVar("R")
logger = logging.getLogger(__name__)
def get_cache_decorator(
    cache_config.update(kwargs)
    return lambda func: ucache(**cache_config)(func)
def get_cache_context(
    return CacheContext(
def clear_cache(
    context = get_cache_context(
    context.clear()
    logger.info(f"Cleared cache for namespace '{namespace}'")

================
File: src/opero/core/fallback.py
================
P = TypeVar("P")
R = TypeVar("R")
logger = logging.getLogger(__name__)
class FallbackError(Exception):
    def __init__(self, message: str, errors: list[Exception]) -> None:
        super().__init__(f"{message}: {errors}")
def get_fallback_decorator(
    def decorator(func: Callable[..., R]) -> Callable[..., R | Coroutine[Any, Any, R]]:
        sig = inspect.signature(func)
            logger.warning(
        type_hints = get_type_hints(func)
        type_hints.get(arg_fallback, Any)
        if asyncio.iscoroutinefunction(func):
            @functools.wraps(func)
            async def async_wrapper(*args: Any, **kwargs: Any) -> R:
                fallback_values = kwargs.get(arg_fallback, ["fallback"])
                if not isinstance(fallback_values, list | tuple):
                    return await func(*args, **kwargs)
                    logger.warning(f"Original call failed for {func.__name__}: {e!s}")
                            return await func(*new_args, **kwargs)
                            errors.append(e)
                    raise FallbackError(
            def sync_wrapper(*args: Any, **kwargs: Any) -> R:
                    return func(*args, **kwargs)
                            return func(*new_args, **kwargs)

================
File: src/opero/core/rate_limit.py
================
P = TypeVar("P")
R = TypeVar("R")
logger = logging.getLogger(__name__)
class RateLimiter:
    def __init__(self, rate_limit: float) -> None:
        self.async_limiter = asynciolimiter.Limiter(rate_limit)
    def wait(self) -> None:
        current_time = time.time()
            time.sleep(sleep_time)
        self.last_call_time = time.time()
    async def wait_async(self) -> None:
def get_rate_limit_decorator(
    limiter = RateLimiter(rate_limit)
    def decorator(func: Callable[..., R]) -> Callable[..., R]:
        if asyncio.iscoroutinefunction(func):
            @functools.wraps(func)
            async def async_wrapper(*args: Any, **kwargs: Any) -> R:
                await limiter.wait_async()
                return await func(*args, **kwargs)
            def sync_wrapper(*args: Any, **kwargs: Any) -> R:
                limiter.wait()
                return func(*args, **kwargs)

================
File: src/opero/core/retry.py
================
P = TypeVar("P")
R = TypeVar("R")
logger = logging.getLogger(__name__)
def get_retry_decorator(
        "stop": stop_after_attempt(
        "wait": wait_exponential(
        "retry": tenacity.retry_if_exception_type(retry_on),
        retry_config["wait"] = tenacity.wait_random_exponential(
    retry_config.update(kwargs)
    def decorator(func: Callable[..., R]) -> Callable[..., R]:
        if asyncio.iscoroutinefunction(func):
            @functools.wraps(func)
            async def async_wrapper(*args: Any, **kwargs: Any) -> R:
                retryer = AsyncRetrying(**retry_config)
                    return await retryer(func, *args, **kwargs)
                    logger.error(f"All retry attempts failed for {func.__name__}: {e}")
            def sync_wrapper(*args: Any, **kwargs: Any) -> R:
                retryer = Retrying(**retry_config)
                    return retryer(func, *args, **kwargs)

================
File: src/opero/decorators/__init__.py
================


================
File: src/opero/decorators/opero.py
================
R = TypeVar("R")
logger = logging.getLogger(__name__)
def opero(
    def decorator(func: Callable[..., R]) -> Callable[..., R]:
        cache_decorator = get_cache_decorator(
        decorated_func = cache_decorator(decorated_func)
        rate_limit_decorator = get_rate_limit_decorator(rate_limit=rate_limit, **kwargs)
        decorated_func = rate_limit_decorator(decorated_func)
        retry_decorator = get_retry_decorator(
        decorated_func = retry_decorator(decorated_func)
        fallback_decorator = get_fallback_decorator(arg_fallback=arg_fallback, **kwargs)
        decorated_func = fallback_decorator(decorated_func)

================
File: src/opero/decorators/opmap.py
================
T = TypeVar("T")  # Input type
R = TypeVar("R")  # Return type
logger = logging.getLogger(__name__)
def opmap(
    def decorator(func: Callable[[T], R]) -> Callable[[Iterable[T]], list[R]]:
        cache_decorator = get_cache_decorator(
        decorated_func = cache_decorator(decorated_func)
        rate_limit_decorator = get_rate_limit_decorator(rate_limit=rate_limit, **kwargs)
        decorated_func = rate_limit_decorator(decorated_func)
        retry_decorator = get_retry_decorator(
        decorated_func = retry_decorator(decorated_func)
        fallback_decorator = get_fallback_decorator(arg_fallback=arg_fallback, **kwargs)
        decorated_func = fallback_decorator(decorated_func)
        parallel_map_decorator = get_parallel_map_decorator(
        return parallel_map_decorator(decorated_func)

================
File: src/opero/utils/__init__.py
================


================
File: src/opero/utils/async_utils.py
================
T = TypeVar("T")
R = TypeVar("R")
def is_async_function(func: Callable[..., Any]) -> bool:
    return asyncio.iscoroutinefunction(func) or inspect.isawaitable(func)
def ensure_async(func: Callable[..., R]) -> Callable[..., Coroutine[Any, Any, R]]:
    if asyncio.iscoroutinefunction(func):
    @functools.wraps(func)
    async def wrapper(*args: Any, **kwargs: Any) -> R:
        return func(*args, **kwargs)
def run_async(
        loop = asyncio.get_running_loop()
        return loop.run_until_complete(func(*args, **kwargs))
        return asyncio.run(func(*args, **kwargs))

================
File: src/opero/utils/logging.py
================
logger = logging.getLogger("opero")
def configure_logging(
    if isinstance(level, str):
        level = getattr(logging, level.upper())
    handler = logging.StreamHandler(stream or sys.stderr)
    handler.setFormatter(logging.Formatter(format_string))
    logger.setLevel(level)
    logger.addHandler(handler)
def get_logger(name: str | None = None) -> logging.Logger:
    return logging.getLogger(f"opero.{name}")

================
File: src/opero/__init__.py
================


================
File: src/opero/exceptions.py
================
class OperoError(Exception):
class AllFailedError(OperoError):
    def __init__(self, message="All fallback operations failed."):
        super().__init__(self.message)
    def __str__(self) -> str:

================
File: tests/test_core.py
================
async def async_success(value):
async def async_fail(value):
    raise ValueError(error_msg)
async def async_fallback(value):
async def async_process_success(*args):
    if len(args) == 1:
async def async_process_fallback(*args):
def sync_success(value):
def sync_fail(value):
def sync_fallback(value):
async def test_fallback_success():
    fallback_decorator = get_fallback_decorator(arg_fallback="fallback_values")
    async def test_func(value, fallback_values=None):
        return await async_success(value)
    result = await test_func(1)
async def test_fallback_with_fallback():
            return await async_fail(value)
        return await async_fallback(value)
async def test_fallback_all_fail():
    with pytest.raises(FallbackError):
        await test_func(1)
async def test_fallback_sync_function():
    def test_func(value, fallback_values=None):
            return sync_fail(value)
        return sync_success(value)
    result = test_func(1)
async def test_retry_success():
    retry_decorator = get_retry_decorator(retries=2)
    async def test_func(value):
async def test_retry_with_retry():
    mock_func = AsyncMock(side_effect=[ValueError("First attempt"), "Success"])
        return await mock_func(value)
async def test_retry_all_fail():
    with pytest.raises(ValueError):
async def test_retry_sync_function():
    def test_func(value):

================
File: tests/test_decorators.py
================
async def async_success(value):
async def async_fail(value):
    raise ValueError(error_msg)
async def async_fallback(value):
def sync_success(value):
def sync_fail(value):
def sync_fallback(value):
async def test_opero_decorator_basic():
    @opero(cache=False)
    async def decorated_func(value):
        return await async_success(value)
    result = await decorated_func(1)
async def test_opero_decorator_fallback():
    @opero(cache=False, arg_fallback="fallback_values")
    async def decorated_func(value, fallback_values=None):
            return await async_fail(value)
        return await async_fallback(value)
async def test_opero_decorator_retry():
    mock_func = AsyncMock(side_effect=[ValueError("First attempt"), "Success"])
    @opero(cache=False, retries=2)
        return await mock_func(value)
async def test_opero_decorator_with_sync_function():
    def decorated_func(value):
        return sync_success(value)
    result = decorated_func(1)
async def test_opmap_decorator_basic():
    @opmap(cache=False, mode="thread", workers=2)
    def process_item(item):
    results = process_item([1, 2, 3])
async def test_opmap_decorator_with_async_function():
    async def process_item(item):
async def test_opmap_decorator_fallback():
    @opmap(cache=False, mode="thread", workers=2, arg_fallback="fallback_values")
    def process_item(item, fallback_values=None):
            raise ValueError(msg)
    assert all(result == "Fallback: fallback" for result in results)
async def test_opmap_decorator_retry():
        1: [ValueError("First attempt"), "Success: 1"],
        3: [ValueError("First attempt"), "Success: 3"],
    def mock_process(item):
        effect = side_effects[item].pop(0)
        if isinstance(effect, Exception):
    @opmap(cache=False, mode="thread", workers=2, retries=2)
        return mock_process(item)

================
File: tests/test_package.py
================
def test_version():

================
File: .gitignore
================
*_autogen/
.DS_Store
__version__.py
__pycache__/
_Chutzpah*
_deps
_NCrunch_*
_pkginfo.txt
_Pvt_Extensions
_ReSharper*/
_TeamCity*
_UpgradeReport_Files/
!?*.[Cc]ache/
!.axoCover/settings.json
!.vscode/extensions.json
!.vscode/launch.json
!.vscode/settings.json
!.vscode/tasks.json
!**/[Pp]ackages/build/
!Directory.Build.rsp
.*crunch*.local.xml
.axoCover/*
.builds
.cr/personal
.fake/
.history/
.ionide/
.localhistory/
.mfractor/
.ntvs_analysis.dat
.paket/paket.exe
.sass-cache/
.vs/
.vscode
.vscode/*
.vshistory/
[Aa][Rr][Mm]/
[Aa][Rr][Mm]64/
[Bb]in/
[Bb]uild[Ll]og.*
[Dd]ebug/
[Dd]ebugPS/
[Dd]ebugPublic/
[Ee]xpress/
[Ll]og/
[Ll]ogs/
[Oo]bj/
[Rr]elease/
[Rr]eleasePS/
[Rr]eleases/
[Tt]est[Rr]esult*/
[Ww][Ii][Nn]32/
*_h.h
*_i.c
*_p.c
*_wpftmp.csproj
*- [Bb]ackup ([0-9]).rdl
*- [Bb]ackup ([0-9][0-9]).rdl
*- [Bb]ackup.rdl
*.[Cc]ache
*.[Pp]ublish.xml
*.[Rr]e[Ss]harper
*.a
*.app
*.appx
*.appxbundle
*.appxupload
*.aps
*.azurePubxml
*.bim_*.settings
*.bim.layout
*.binlog
*.btm.cs
*.btp.cs
*.build.csdef
*.cab
*.cachefile
*.code-workspace
*.coverage
*.coveragexml
*.d
*.dbmdl
*.dbproj.schemaview
*.dll
*.dotCover
*.DotSettings.user
*.dsp
*.dsw
*.dylib
*.e2e
*.exe
*.gch
*.GhostDoc.xml
*.gpState
*.ilk
*.iobj
*.ipdb
*.jfm
*.jmconfig
*.la
*.lai
*.ldf
*.lib
*.lo
*.log
*.mdf
*.meta
*.mm.*
*.mod
*.msi
*.msix
*.msm
*.msp
*.ncb
*.ndf
*.nuget.props
*.nuget.targets
*.nupkg
*.nvuser
*.o
*.obj
*.odx.cs
*.opendb
*.opensdf
*.opt
*.out
*.pch
*.pdb
*.pfx
*.pgc
*.pgd
*.pidb
*.plg
*.psess
*.publishproj
*.publishsettings
*.pubxml
*.pyc
*.rdl.data
*.rptproj.bak
*.rptproj.rsuser
*.rsp
*.rsuser
*.sap
*.sbr
*.scc
*.sdf
*.sln.docstates
*.sln.iml
*.slo
*.smod
*.snupkg
*.so
*.suo
*.svclog
*.tlb
*.tlh
*.tli
*.tlog
*.tmp
*.tmp_proj
*.tss
*.user
*.userosscache
*.userprefs
*.vbp
*.vbw
*.VC.db
*.VC.VC.opendb
*.VisualState.xml
*.vsp
*.vspscc
*.vspx
*.vssscc
*.xsd.cs
**/[Pp]ackages/*
**/*.DesktopClient/GeneratedArtifacts
**/*.DesktopClient/ModelManifest.xml
**/*.HTMLClient/GeneratedArtifacts
**/*.Server/GeneratedArtifacts
**/*.Server/ModelManifest.xml
*~
~$*
$tf/
AppPackages/
artifacts/
ASALocalRun/
AutoTest.Net/
Backup*/
BenchmarkDotNet.Artifacts/
bld/
BundleArtifacts/
ClientBin/
cmake_install.cmake
CMakeCache.txt
CMakeFiles
CMakeLists.txt.user
CMakeScripts
CMakeUserPresets.json
compile_commands.json
coverage*.info
coverage*.json
coverage*.xml
csx/
CTestTestfile.cmake
dlldata.c
DocProject/buildhelp/
DocProject/Help/*.hhc
DocProject/Help/*.hhk
DocProject/Help/*.hhp
DocProject/Help/*.HxC
DocProject/Help/*.HxT
DocProject/Help/html
DocProject/Help/Html2
ecf/
FakesAssemblies/
FodyWeavers.xsd
Generated_Code/
Generated\ Files/
healthchecksdb
install_manifest.txt
ipch/
Makefile
MigrationBackup/
mono_crash.*
nCrunchTemp_*
node_modules/
nunit-*.xml
OpenCover/
orleans.codegen.cs
Package.StoreAssociation.xml
paket-files/
project.fragment.lock.json
project.lock.json
publish/
PublishScripts/
rcf/
ScaffoldingReadMe.txt
ServiceFabricBackup/
StyleCopReport.xml
Testing
TestResult.xml
UpgradeLog*.htm
UpgradeLog*.XML
x64/
x86/
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# Distribution / packaging
!dist/.gitkeep

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
.ruff_cache/

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# IDE
.idea/
.vscode/
*.swp
*.swo
*~

# OS
.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
ehthumbs.db
Thumbs.db

# Project specific
__version__.py
_private

# PyInstaller
#  Usually these files are written by a python script from a template
#  before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
.python-version

# celery beat schedule file
celerybeat-schedule

# SageMath parsed files
*.sage.py

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# VS Code
.vscode/
.idea/

# macOS
.DS_Store

================
File: .pre-commit-config.yaml
================
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.3.4
    hooks:
      - id: ruff
        args: [--fix]
      - id: ruff-format
        args: [--respect-gitignore]
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: check-yaml
      - id: check-toml
      - id: check-added-large-files
      - id: debug-statements
      - id: check-case-conflict
      - id: mixed-line-ending
        args: [--fix=lf]

================
File: CHANGELOG.md
================
---
this_file: CHANGELOG.md
---

# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added
- Initial project structure and core functionality
  - `@opero` decorator for resilient function execution
  - `@opmap` decorator for parallel processing with resilience
  - Parameter-based fallback mechanism
  - Basic retry mechanism with exponential backoff
  - Rate limiting support
  - Async/sync function support
- Core utilities and helpers
  - Logging configuration and utilities
  - Async/sync conversion utilities
  - Error handling utilities
- Documentation and project files
  - Basic usage examples
  - API documentation
  - Development guidelines

### Changed
- Restructured project into modular directories
  - Organized core functionality into separate modules
  - Separated decorators into their own module
  - Created utilities module
- Improved error handling and logging
  - Added comprehensive logging throughout
  - Enhanced error messages and context
  - Added proper exception hierarchy
- Enhanced type safety and code quality
  - Added comprehensive type hints
  - Fixed linter errors
  - Improved code organization

### Fixed
- Parameter-based fallback mechanism
  - Corrected fallback value handling
  - Fixed error propagation
  - Improved logging of fallback attempts
- Retry mechanism
  - Fixed parameter order in retry function calls
  - Added proper error handling for retries
  - Improved retry logging
- Async/sync conversion
  - Fixed coroutine handling
  - Improved async function detection
  - Added proper async context management

## [0.1.0] - TBD

### Added
- Initial release of the `opero` package
- Core functionality:
  - `@opero` decorator
  - `@opmap` decorator
  - Parameter-based fallbacks
  - Retry mechanism
  - Rate limiting
  - Async support
- Basic documentation and examples

================
File: cleanup.py
================
logging.basicConfig(
logger = logging.getLogger(__name__)
LOG_FILE = Path("CLEANUP.txt")
os.chdir(Path(__file__).parent)
def new() -> None:
    if LOG_FILE.exists():
        LOG_FILE.unlink()
def prefix() -> None:
    readme = Path(".cursor/rules/0project.mdc")
    if readme.exists():
        log_message("\n=== PROJECT STATEMENT ===")
        content = readme.read_text(encoding="utf-8")
        log_message(content)
def suffix() -> None:
    todo = Path("TODO.md")
    if todo.exists():
        log_message("\n=== TODO.md ===")
        content = todo.read_text(encoding="utf-8")
def log_message(message: str) -> None:
    timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
    with LOG_FILE.open("a", encoding="utf-8") as f:
        f.write(log_line)
    logger.info(message)
def run_command(cmd: list[str], *, check: bool = True) -> subprocess.CompletedProcess:
        result = subprocess.run(  # noqa: S603
            log_message(result.stdout)
        log_message(f"Command failed: {' '.join(cmd)}")
        log_message(f"Error: {e.stderr}")
        return subprocess.CompletedProcess(cmd, 1, "", str(e))
def check_command_exists(cmd: str) -> bool:
        return which(cmd) is not None
class Cleanup:
    def __init__(self) -> None:
        self.workspace = Path.cwd()
    def _print_header(self, message: str) -> None:
        log_message(f"\n=== {message} ===")
    def _check_required_files(self) -> bool:
            if not (self.workspace / file).exists():
                log_message(f"Error: {file} is missing")
    def _generate_tree(self) -> None:
        if not check_command_exists("tree"):
            log_message("Warning: 'tree' command not found. Skipping tree generation.")
            rules_dir = Path(".cursor/rules")
            rules_dir.mkdir(parents=True, exist_ok=True)
            tree_result = run_command(
            with open(rules_dir / "filetree.mdc", "w", encoding="utf-8") as f:
                f.write("---\ndescription: File tree of the project\nglobs: \n---\n")
                f.write(tree_text)
            log_message("\nProject structure:")
            log_message(tree_text)
            log_message(f"Failed to generate tree: {e}")
    def _git_status(self) -> bool:
        result = run_command(["git", "status", "--porcelain"], check=False)
        return bool(result.stdout.strip())
    def _venv(self) -> None:
        log_message("Setting up virtual environment")
            run_command(["uv", "venv"])
            if venv_path.exists():
                os.environ["VIRTUAL_ENV"] = str(self.workspace / ".venv")
                log_message("Virtual environment created and activated")
                log_message("Virtual environment created but activation failed")
            log_message(f"Failed to create virtual environment: {e}")
    def _install(self) -> None:
        log_message("Installing package with all extras")
            self._venv()
            run_command(["uv", "pip", "install", "-e", ".[test,dev]"])
            log_message("Package installed successfully")
            log_message(f"Failed to install package: {e}")
    def _run_checks(self) -> None:
        log_message("Running code quality checks")
            log_message(">>> Running code fixes...")
            run_command(
            log_message(">>>Running type checks...")
            run_command(["python", "-m", "mypy", "src", "tests"], check=False)
            log_message(">>> Running tests...")
            run_command(["python", "-m", "pytest", "tests"], check=False)
            log_message("All checks completed")
            log_message(f"Failed during checks: {e}")
    def status(self) -> None:
        prefix()  # Add README.md content at start
        self._print_header("Current Status")
        self._check_required_files()
        self._generate_tree()
        result = run_command(["git", "status"], check=False)
        self._print_header("Environment Status")
        self._install()
        self._run_checks()
        suffix()  # Add TODO.md content at end
    def venv(self) -> None:
        self._print_header("Virtual Environment Setup")
    def install(self) -> None:
        self._print_header("Package Installation")
    def update(self) -> None:
        self.status()
        if self._git_status():
            log_message("Changes detected in repository")
                run_command(["git", "add", "."])
                run_command(["git", "commit", "-m", commit_msg])
                log_message("Changes committed successfully")
                log_message(f"Failed to commit changes: {e}")
            log_message("No changes to commit")
    def push(self) -> None:
        self._print_header("Pushing Changes")
            run_command(["git", "push"])
            log_message("Changes pushed successfully")
            log_message(f"Failed to push changes: {e}")
def repomix(
            cmd.append("--compress")
            cmd.append("--remove-empty-lines")
            cmd.append("-i")
            cmd.append(ignore_patterns)
        cmd.extend(["-o", output_file])
        run_command(cmd)
        log_message(f"Repository content mixed into {output_file}")
        log_message(f"Failed to mix repository: {e}")
def print_usage() -> None:
    log_message("Usage:")
    log_message("  cleanup.py status   # Show current status and run all checks")
    log_message("  cleanup.py venv     # Create virtual environment")
    log_message("  cleanup.py install  # Install package with all extras")
    log_message("  cleanup.py update   # Update and commit changes")
    log_message("  cleanup.py push     # Push changes to remote")
def main() -> NoReturn:
    new()  # Clear log file
    if len(sys.argv) < 2:
        print_usage()
        sys.exit(1)
    cleanup = Cleanup()
            cleanup.status()
            cleanup.venv()
            cleanup.install()
            cleanup.update()
            cleanup.push()
        log_message(f"Error: {e}")
    repomix()
    sys.stdout.write(Path("CLEANUP.txt").read_text(encoding="utf-8"))
    sys.exit(0)  # Ensure we exit with a status code
    main()

================
File: get-pip.py
================
        "This script does not work on Python {}.{}.".format(*this_python),
        "The minimum supported Python version is {}.{}.".format(*min_version),
        "Please use https://bootstrap.pypa.io/pip/{}.{}/get-pip.py instead.".format(*this_python),
    sys.exit(1)
def include_setuptools(args):
    env = not os.environ.get("PIP_NO_SETUPTOOLS")
    absent = not importlib.util.find_spec("setuptools")
def include_wheel(args):
    env = not os.environ.get("PIP_NO_WHEEL")
    absent = not importlib.util.find_spec("wheel")
def determine_pip_install_arguments():
    pre_parser = argparse.ArgumentParser()
    pre_parser.add_argument("--no-setuptools", action="store_true")
    pre_parser.add_argument("--no-wheel", action="store_true")
    pre, args = pre_parser.parse_known_args()
    args.append("pip")
    if include_setuptools(pre):
        args.append("setuptools")
    if include_wheel(pre):
        args.append("wheel")
def monkeypatch_for_cert(tmpdir):
    cert_path = os.path.join(tmpdir, "cacert.pem")
    with open(cert_path, "wb") as cert:
        cert.write(pkgutil.get_data("pip._vendor.certifi", "cacert.pem"))
    def cert_parse_args(self, args):
        if not self.parser.get_default_values().cert:
        return install_parse_args(self, args)
def bootstrap(tmpdir):
    monkeypatch_for_cert(tmpdir)
    args = determine_pip_install_arguments()
    sys.exit(pip_entry_point(args))
def main():
        tmpdir = tempfile.mkdtemp()
        pip_zip = os.path.join(tmpdir, "pip.zip")
        with open(pip_zip, "wb") as fp:
            fp.write(b85decode(DATA.replace(b"\n", b"")))
        sys.path.insert(0, pip_zip)
        bootstrap(tmpdir=tmpdir)
            shutil.rmtree(tmpdir, ignore_errors=True)
    main()

================
File: LICENSE
================
MIT License

Copyright (c) 2023 Adam Twardoch

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

================
File: package.toml
================
# Package configuration
[package]
include_cli = true        # Include CLI boilerplate
include_logging = true    # Include logging setup
use_pydantic = true      # Use Pydantic for data validation
use_rich = true          # Use Rich for terminal output

[features]
mkdocs = false           # Enable MkDocs documentation
vcs = true              # Initialize Git repository
github_actions = true   # Add GitHub Actions workflows

================
File: PROGRESS.md
================
# Progress

This file tracks the detailed progress of tasks for the `opero` package.

## Completed Tasks

- [x] Core Package Structure
  - [x] Set up project structure
  - [x] Create basic package files
  - [x] Set up dependency management
  - [x] Configure development tools
  - [x] Set up testing framework

- [x] Core Functionality
  - [x] Implement `@opero` decorator
  - [x] Implement `@opmap` decorator
  - [x] Add basic parameter-based fallbacks
  - [x] Add basic retry mechanism
  - [x] Add basic rate limiting
  - [x] Add basic async support

- [x] Documentation and Project Files
  - [x] Create README.md with basic documentation
  - [x] Create CHANGELOG.md for version tracking
  - [x] Create TODO.md for future goals
  - [x] Create LICENSE file
  - [x] Create .gitignore file
  - [x] Create pyproject.toml

- [x] Code Organization
  - [x] Restructure into modular directories
  - [x] Organize core functionality
  - [x] Organize decorators
  - [x] Organize utilities
  - [x] Clean up obsolete files

## In Progress Tasks

- [ ] Core Functionality Enhancements
  - [ ] Improve parameter-based fallbacks
  - [ ] Enhance retry mechanism
  - [ ] Optimize rate limiting
  - [ ] Improve async support
  - [ ] Add progress tracking

- [ ] Testing and Quality
  - [ ] Add more unit tests
  - [ ] Add integration tests
  - [ ] Fix remaining linter errors
  - [ ] Improve test coverage
  - [ ] Add performance tests

- [ ] Documentation
  - [ ] Expand API documentation
  - [ ] Add more usage examples
  - [ ] Create troubleshooting guide
  - [ ] Document best practices
  - [ ] Add architecture overview

## Upcoming Tasks

- [ ] Advanced Features
  - [ ] Implement caching system
  - [ ] Add distributed rate limiting
  - [ ] Add monitoring and metrics
  - [ ] Implement circuit breaker
  - [ ] Add distributed tracing

- [ ] Developer Experience
  - [ ] Create development tools
  - [ ] Add debugging utilities
  - [ ] Create benchmarking suite
  - [ ] Add code generation tools
  - [ ] Improve error messages

- [ ] Release Management
  - [ ] Set up CI/CD pipeline
  - [ ] Configure automated testing
  - [ ] Set up automated releases
  - [ ] Create release checklist
  - [ ] Plan version roadmap

================
File: pyproject.toml
================
# this_file: pyproject.toml
[project]
name = "opero"
dynamic = ["version"]
description = "Resilient, parallel task orchestration for Python"
readme = "README.md"
requires-python = ">=3.10"
license = { file = "LICENSE" }
keywords = [
    "orchestration",
    "resilience",
    "retry",
    "fallback",
    "parallel",
    "concurrency",
    "rate-limiting",
]
classifiers = [
    "Development Status :: 4 - Beta",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
    "Programming Language :: Python :: Implementation :: CPython",
    "Programming Language :: Python :: Implementation :: PyPy",
]
dependencies = [
    "tenacity>=8.0.0",
    "asynciolimiter>=1.0.0",
    "twat-cache>=2.3.0",
    "twat-mp>=2.6.0",
]

[project.optional-dependencies]
dev = [
    "black>=23.1.0",
    "mypy>=1.0.0",
    "pytest>=7.0.0",
    "pytest-cov>=4.0.0",
    "pytest-asyncio>=0.21.0",
    "ruff>=0.0.243",
]
test = [
    "pytest>=7.0.0",
    "pytest-cov>=4.0.0",
    "pytest-asyncio>=0.21.0",
    "tenacity>=8.0.0",
    "asynciolimiter>=1.0.0",
    "pathos>=0.3.0",
    "aiomultiprocess>=0.9.0",
]
pathos = ["pathos>=0.3.0"]
aiomultiprocess = ["aiomultiprocess>=0.9.0"]
all = ["pathos>=0.3.0", "aiomultiprocess>=0.9.0"]

[project.scripts]
# CLINAME = "opero.__main__:main"

[[project.authors]]
name = "Adam Twardoch"
email = "adam@twardoch.com"

[project.urls]
Documentation = "https://github.com/twardoch/opero#readme"
Issues = "https://github.com/twardoch/opero/issues"
Source = "https://github.com/twardoch/opero"

[build-system]
requires = [
    "hatchling>=1.27.0", # Core build backend for Hatch
    "hatch-vcs>=0.4.0",  # Version Control System plugin for Hatch
]
build-backend = "hatchling.build" # Use Hatchling as the build backend

[tool.coverage.paths]
opero = ["src/opero", "*/opero/src/opero"]
tests = ["tests", "*/opero/tests"]

[tool.coverage.report]
exclude_lines = ["no cov", "if __name__ == .__main__.:", "if TYPE_CHECKING:"]

[tool.coverage.run]
source_pkgs = ["opero", "tests"]
branch = true
parallel = true
omit = ["src/opero/__about__.py"]

[tool.hatch.build.hooks.vcs]
version-file = "src/opero/__version__.py"

[tool.hatch.build.targets.wheel]
packages = ["src/opero"]

[tool.hatch.envs.default]
dependencies = [
    "pytest",
    "pytest-cov",
    "pytest-asyncio",
    "tenacity",
    "asynciolimiter",
    "pathos",
    "aiomultiprocess",
]

[[tool.hatch.envs.all.matrix]]
python = ["3.10", "3.11", "3.12"]

[tool.hatch.envs.default.scripts]
test = "pytest {args:tests}"
test-cov = "pytest --cov-report=term-missing --cov-config=pyproject.toml --cov=src/opero --cov=tests {args:tests}"
type-check = "mypy src/opero tests"
lint = [
    "ruff check src/opero tests",
    "ruff format --respect-gitignore src/opero tests",
]
fix = [
    "ruff check  --fix --unsafe-fixes src/opero tests",
    "ruff format --respect-gitignore src/opero tests",
]

[tool.hatch.envs.lint]
detached = true
dependencies = ["tenacity>=8.0.0", "asynciolimiter>=1.0.0"]

[tool.hatch.envs.lint.scripts]
typing = "mypy --install-types --non-interactive {args:src/opero tests}"
style = ["ruff check {args:.}", "ruff format --respect-gitignore {args:.}"]
fmt = ["ruff format --respect-gitignore {args:.}", "ruff check --fix {args:.}"]
all = ["style", "typing"]

[tool.hatch.envs.test]
dependencies = [
    "tenacity>=8.0.0",
    "asynciolimiter>=1.0.0",
    "pytest-asyncio>=0.21.0",
]

[tool.hatch.envs.test.scripts]
test = "python -m pytest -n auto -p no:briefcase {args:tests}"
test-cov = "python -m pytest -n auto -p no:briefcase --cov-report=term-missing --cov-config=pyproject.toml --cov=src/opero --cov=tests {args:tests}"
bench = "python -m pytest -v -p no:briefcase tests/test_benchmark.py --benchmark-only"
bench-save = "python -m pytest -v -p no:briefcase tests/test_benchmark.py --benchmark-only --benchmark-json=benchmark/results.json"

[tool.hatch.version]
source = "vcs"
path = 'src/opero/__version__.py'
pattern = "__version__\\s*=\\s*version\\s*=\\s*['\"](?P<version>[^'\"]+)['\"]"

[tool.hatch.version.raw-options]
version_scheme = "post-release"

[tool.mypy]
python_version = "3.10"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
check_untyped_defs = true
disallow_untyped_decorators = true
no_implicit_optional = true
warn_redundant_casts = true
warn_unused_ignores = true
warn_no_return = true
warn_unreachable = true

[tool.ruff]
target-version = "py310"
line-length = 88

[tool.ruff.lint]
extend-select = [
    "A",
    "ARG",
    "B",
    "C",
    "DTZ",
    "E",
    "EM",
    "F",
    "FBT",
    "I",
    "ICN",
    "ISC",
    "N",
    "PLC",
    "PLE",
    "PLR",
    "PLW",
    "Q",
    "RUF",
    "S",
    "T",
    "TID",
    "UP",
    "W",
    "YTT",
]
ignore = ["ARG001", "E501", "I001", "RUF001", "PLR2004", "EXE003", "ISC001"]

[tool.ruff.lint.per-file-ignores]
"tests/*" = ["S101"]

[tool.pytest.ini_options]
addopts = "-v --durations=10 -p no:briefcase"
asyncio_mode = "auto"
console_output_style = "progress"
filterwarnings = ["ignore::DeprecationWarning", "ignore::UserWarning"]
log_cli = true
log_cli_level = "INFO"
markers = [
    "benchmark: marks tests as benchmarks (select with '-m benchmark')",
    "unit: mark a test as a unit test",
    "integration: mark a test as an integration test",
    "permutation: tests for permutation functionality",
    "parameter: tests for parameter parsing",
    "prompt: tests for prompt parsing",
]
norecursedirs = [
    ".*",
    "build",
    "dist",
    "venv",
    "__pycache__",
    "*.egg-info",
    "_private",
]
python_classes = ["Test*"]
python_files = ["test_*.py"]
python_functions = ["test_*"]
testpaths = ["tests"]

[tool.pytest-benchmark]
min_rounds = 100
min_time = 0.1
histogram = true
storage = "file"
save-data = true
compare = [
    "min",    # Minimum time
    "max",    # Maximum time
    "mean",   # Mean time
    "stddev", # Standard deviation
    "median", # Median time
    "iqr",    # Inter-quartile range
    "ops",    # Operations per second
    "rounds", # Number of rounds
]

================
File: README.md
================
---
this_file: README.md
---

# Opero: Resilient, Parallel Task Orchestration for Python

[![PyPI version](https://img.shields.io/pypi/v/opero.svg)](https://pypi.org/project/opero/)
[![Python versions](https://img.shields.io/pypi/pyversions/opero.svg)](https://pypi.org/project/opero/)
[![License](https://img.shields.io/github/license/twardoch/opero.svg)](https://github.com/twardoch/opero/blob/main/LICENSE)

Opero provides a clean, Pythonic interface for orchestrating resilient, parallelized operations. The name comes from the Latin word for "to work" or "to operate". It offers a simple yet powerful way to add resilience mechanisms to your functions through decorators.

## Key Features

- **Simple Decorator Interface**: Two focused decorators for all your needs
  - `@opero`: Add resilience mechanisms to any function
  - `@opmap`: Add resilience and parallel processing capabilities
- **Parameter-Based Fallbacks**: Try alternative parameter values when operations fail
- **Retry Mechanism**: Exponential backoff with jitter for robust retries
- **Rate Limiting**: Control operation frequency to avoid overwhelming resources
- **Parallel Processing**: Multiple execution modes (process, thread, async)
- **Async First**: Built for modern async workflows while supporting sync functions
- **Type Safety**: Comprehensive type hints for better IDE integration

## Installation

```bash
pip install opero
```

Optional dependencies:

```bash
# For enhanced multiprocessing support
pip install opero[pathos]

# For async multiprocessing
pip install opero[aiomultiprocess]

# Install all optional dependencies
pip install opero[all]
```

## Quick Start

### Basic Usage with `@opero`

```python
from opero import opero

@opero(
    # Enable caching with 1-hour TTL
    cache=True,
    cache_ttl=3600,
    
    # Configure retries
    retries=3,
    backoff_factor=1.5,
    
    # Add parameter-based fallbacks
    arg_fallback="model"
)
async def call_api(prompt: str, model: list[str] = ["gpt-4", "gpt-3.5"]):
    """
    Call an API with fallback models.
    Will try gpt-4 first, then fall back to gpt-3.5 if it fails.
    Results are cached for 1 hour.
    """
    response = await api_call(prompt=prompt, model=model[0])
    return response

# Usage
result = await call_api("Hello, world!")
```

### Parallel Processing with `@opmap`

```python
from opero import opmap

@opmap(
    # Use process-based parallelism
    mode="process",
    workers=4,
    
    # Enable caching
    cache=True,
    cache_ttl=1800,
    
    # Add fallbacks for API keys
    arg_fallback="api_key"
)
def process_item(item: dict, api_key: list[str] = ["primary", "backup"]):
    """
    Process items in parallel with resilience.
    Uses 4 worker processes and tries backup API key if primary fails.
    Results are cached for 30 minutes.
    """
    return make_api_call(item, api_key=api_key[0])

# Process multiple items in parallel
results = process_item([item1, item2, item3])
```

## Core Concepts

### Parameter-Based Fallbacks

The `arg_fallback` parameter allows you to specify which function parameter contains fallback values:

```python
@opero(arg_fallback="api_key")
async def fetch_data(url: str, api_key: list[str] = ["primary", "backup"]):
    """Try each API key in sequence until one succeeds."""
    return await make_request(url, api_key=api_key[0])
```

### Retry Mechanism

Configure retry behavior with exponential backoff:

```python
@opero(
    retries=3,              # Number of retries
    backoff_factor=1.5,     # Exponential backoff multiplier
    min_delay=0.1,          # Minimum delay between retries
    max_delay=30.0,         # Maximum delay between retries
    retry_on=ConnectionError # Retry only on specific exceptions
)
async def fetch_url(url: str):
    """Fetch a URL with retries on connection errors."""
    return await make_request(url)
```

### Rate Limiting

Control how frequently operations can be executed:

```python
@opero(rate_limit=10.0)  # Maximum 10 operations per second
async def rate_limited_api(query: str):
    """Make API calls without overwhelming the service."""
    return await api_call(query)
```

### Caching

Cache results to improve performance:

```python
@opero(
    cache=True,
    cache_ttl=3600,         # Cache for 1 hour
    cache_backend="redis",  # Use Redis for caching
    cache_namespace="api"   # Namespace for cache keys
)
async def expensive_operation(data: dict):
    """Expensive operation with results cached in Redis."""
    return await process_data(data)
```

## Advanced Usage

### Combining Multiple Features

You can combine multiple resilience features:

```python
@opero(
    # Caching
    cache=True,
    cache_ttl=3600,
    
    # Retries
    retries=3,
    backoff_factor=1.5,
    
    # Rate limiting
    rate_limit=10.0,
    
    # Fallbacks
    arg_fallback="endpoint"
)
async def resilient_api(
    data: dict,
    endpoint: list[str] = ["primary", "backup"]
):
    """
    Fully resilient API call with:
    - Caching for performance
    - Retries for transient failures
    - Rate limiting to avoid overwhelming the API
    - Fallback endpoints if primary fails
    """
    return await call_endpoint(endpoint[0], data)
```

### Parallel Processing Modes

The `@opmap` decorator supports different execution modes:

```python
# Process-based parallelism for CPU-bound tasks
@opmap(mode="process", workers=4)
def cpu_intensive(data: bytes):
    return process_data(data)

# Thread-based parallelism for I/O-bound tasks
@opmap(mode="thread", workers=10)
def io_intensive(url: str):
    return download_file(url)

# Async-based parallelism for async functions
@opmap(mode="async", workers=20)
async def async_operation(item: dict):
    return await process_item(item)
```

### Error Handling

Opero provides detailed error information:

```python
from opero import FallbackError

@opero(arg_fallback="api_key")
async def api_call(data: dict, api_key: list[str]):
    try:
        return await make_request(data, api_key=api_key[0])
    except FallbackError as e:
        # Access the original errors that caused fallbacks
        for error in e.errors:
            print(f"Attempt failed: {error}")
        raise
```

### Logging

Opero includes comprehensive logging:

```python
import logging
from opero import configure_logging

# Configure logging with your desired level
logger = configure_logging(level=logging.INFO)

@opero(retries=3)
async def logged_operation():
    # Opero will log retry attempts, fallbacks, etc.
    return await some_operation()
```

## Development

This project uses [Hatch](https://hatch.pypa.io/) for development workflow management.

### Setup Development Environment

```bash
# Install hatch
pip install hatch

# Create and activate environment
hatch shell

# Run tests
hatch run test

# Run tests with coverage
hatch run test-cov

# Run linting
hatch run lint

# Format code
hatch run format
```

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

================
File: TODO.md
================
---
this_file: TODO.md
---

# TODO

This file tracks planned features and improvements for the `opero` package.

## 1. High Priority

- [ ] Enhance core functionality
  - [ ] Improve parameter-based fallback mechanism
  - [ ] Add support for custom fallback strategies
  - [ ] Implement retry with exponential backoff and jitter
  - [ ] Add support for custom retry strategies
  - [ ] Implement rate limiting with token bucket algorithm
  - [ ] Add support for distributed rate limiting

- [ ] Improve caching system
  - [ ] Implement memory cache backend
  - [ ] Add disk cache backend
  - [ ] Add Redis cache backend
  - [ ] Support custom cache key generation
  - [ ] Add cache invalidation strategies

- [ ] Enhance parallel execution
  - [ ] Improve process-based parallelism
  - [ ] Add thread-based parallelism
  - [ ] Optimize async-based parallelism
  - [ ] Implement hybrid async-process parallelism
  - [ ] Add progress tracking and cancellation

## 2. Medium Priority

- [ ] Improve error handling and logging
  - [ ] Add structured error reporting
  - [ ] Implement context-based logging
  - [ ] Add performance metrics collection
  - [ ] Support distributed tracing
  - [ ] Add telemetry features

- [ ] Enhance configuration system
  - [ ] Add YAML/JSON configuration support
  - [ ] Implement environment variable configuration
  - [ ] Add dynamic configuration reloading
  - [ ] Support configuration validation

- [ ] Add monitoring and observability
  - [ ] Implement health checks
  - [ ] Add metrics collection
  - [ ] Support OpenTelemetry integration
  - [ ] Add performance profiling tools

## 3. Low Priority

- [ ] Improve documentation
  - [ ] Create comprehensive API reference
  - [ ] Add more usage examples
  - [ ] Write troubleshooting guide
  - [ ] Create architecture documentation
  - [ ] Add performance optimization guide

- [ ] Add developer tools
  - [ ] Create development environment setup script
  - [ ] Add code generation tools
  - [ ] Implement debugging utilities
  - [ ] Add benchmarking tools

- [ ] Enhance testing
  - [ ] Add property-based tests
  - [ ] Implement integration test suite
  - [ ] Add performance tests
  - [ ] Create stress tests
  - [ ] Add chaos testing support

Tip: Periodically run `python ./cleanup.py status` to see results of lints and tests. Use `uv pip ...` not `pip ...`

================
File: twat-cache.txt
================
This file is a merged representation of a subset of the codebase, containing files not matching ignore patterns, combined into a single document by Repomix. The content has been processed where empty lines have been removed.

================================================================
File Summary
================================================================

Purpose:
--------
This file contains a packed representation of the entire repository's contents.
It is designed to be easily consumable by AI systems for analysis, code review,
or other automated processes.

File Format:
------------
The content is organized as follows:
1. This summary section
2. Repository information
3. Directory structure
4. Multiple file entries, each consisting of:
  a. A separator line (================)
  b. The file path (File: path/to/file)
  c. Another separator line
  d. The full contents of the file
  e. A blank line

Usage Guidelines:
-----------------
- This file should be treated as read-only. Any changes should be made to the
  original repository files, not this packed version.
- When processing this file, use the file path to distinguish
  between different files in the repository.
- Be aware that this file may contain sensitive information. Handle it with
  the same level of security as you would the original repository.

Notes:
------
- Some files may have been excluded based on .gitignore rules and Repomix's configuration
- Binary files are not included in this packed representation. Please refer to the Repository Structure section for a complete list of file paths, including binary files
- Files matching these patterns are excluded: .specstory/**/*.md, .venv/**, _private/**, CLEANUP.txt, **/*.json, *.lock
- Files matching patterns in .gitignore are excluded
- Files matching default ignore patterns are excluded
- Empty lines have been removed from all files

Additional Info:
----------------

================================================================
Directory Structure
================================================================
.cursor/
  rules/
    0project.mdc
    cleanup.mdc
    filetree.mdc
    quality.mdc
.github/
  workflows/
    push.yml
    release.yml
docs/
  context_management.md
examples/
  backend_selection.py
src/
  twat_cache/
    engines/
      __init__.py
      aiocache.py
      base.py
      cachebox.py
      cachetools.py
      common.py
      diskcache.py
      functools_engine.py
      functools.py
      joblib.py
      klepto.py
      manager.py
      py.typed
      redis.py
    types/
      cachebox.pyi
    __init__.py
    __main__.py
    backend_selector.py
    cache.py
    config.py
    context.py
    decorators.py
    exceptions.py
    hybrid_cache.py
    logging.py
    paths.py
    py.typed
    type_defs.py
    utils.py
tests/
  __init__.py
  test_backend_selector.py
  test_cache.py
  test_config.py
  test_constants.py
  test_context_simple.py
  test_context.py
  test_decorators.py
  test_engines.py
  test_exceptions_simple.py
  test_exceptions.py
  test_fallback.py
  test_redis_cache.py
  test_twat_cache.py
.gitignore
.pre-commit-config.yaml
CHANGELOG.md
cleanup.py
LICENSE
NEXT.md
PROMPT.txt
pyproject.toml
README.md
TODO.md
VERSION.txt

================================================================
Files
================================================================

================
File: .cursor/rules/0project.mdc
================
---
description: About this project
globs:
---
# About this project

`twat-fs` is a file system utility library focused on robust and extensible file upload capabilities with multiple provider support. It provides:

- Multi-provider upload system with smart fallback (catbox.moe default, plus Dropbox, S3, etc.)
- Automatic retry for temporary failures, fallback for permanent ones
- URL validation and clean developer experience with type hints
- Simple CLI: `python -m twat_fs upload_file path/to/file.txt`
- Easy installation: `uv pip install twat-fs` (basic) or `uv pip install 'twat-fs[all,dev]'` (all features)

## Development Notes
- Uses `uv` for Python package management
- Quality tools: ruff, mypy, pytest
- Clear provider protocol for adding new storage backends
- Strong typing and runtime checks throughout

================
File: .cursor/rules/cleanup.mdc
================
---
description: Run `cleanup.py` script before and after changes
globs: 
---
Before you do any changes or if I say "cleanup", run the `cleanup.py update` script in the main folder. Analyze the results, describe recent changes in @LOG.md and edit @TODO.md to update priorities and plan next changes. PERFORM THE CHANGES, then run the `cleanup.py status` script and react to the results.

When you edit @TODO.md, lead in lines with empty GFM checkboxes if things aren't done (`- [ ] `) vs. filled (`- [x] `) if done.

================
File: .cursor/rules/filetree.mdc
================
---
description: File tree of the project
globs: 
---
[ 928]  .
├── [  96]  .cursor
│   └── [ 224]  rules
│       ├── [ 821]  0project.mdc
│       ├── [ 516]  cleanup.mdc
│       ├── [3.5K]  filetree.mdc
│       └── [2.0K]  quality.mdc
├── [  96]  .github
│   └── [ 128]  workflows
│       ├── [2.7K]  push.yml
│       └── [1.4K]  release.yml
├── [3.6K]  .gitignore
├── [ 470]  .pre-commit-config.yaml
├── [  96]  .specstory
│   └── [ 864]  history
│       ├── [2.7K]  .what-is-this.md
│       ├── [302K]  2025-03-04_03-25-codebase-analysis-and-todo-md-update.md
│       ├── [ 80K]  2025-03-04_05-35-implementation-of-todo-phases-1,-2,-3.md
│       ├── [115K]  2025-03-04_08-00-project-maintenance-and-documentation-update.md
│       ├── [146K]  2025-03-04_08-41-managing-todo-md-tasks.md
│       ├── [5.7K]  2025-03-04_11-30-untitled.md
│       ├── [ 35K]  caching-strategies-for-twat-cache-library.md
│       ├── [ 54K]  codebase-improvement-and-dependency-management-plan.md
│       ├── [124K]  documentation-update-and-task-management.md
│       ├── [217K]  implementing-todo-item.md
│       ├── [ 44K]  merging-files-from-wrong-location.md
│       ├── [238K]  project-documentation-and-command-execution.md
│       ├── [135K]  project-overview-and-todo-execution.md
│       ├── [ 13K]  project-review-and-implementation.md
│       ├── [142K]  project-setup-and-implementation-steps.md
│       ├── [162K]  project-setup-and-task-implementation.md
│       ├── [ 56K]  project-setup-and-task-management.md
│       ├── [ 53K]  refactoring-plan-for-twat-cache-system.md
│       ├── [173K]  refining-decorators-and-integration-goals-1.md
│       ├── [282K]  refining-decorators-and-integration-goals.md
│       ├── [ 95K]  running-lint-fix-command.md
│       ├── [ 11K]  todo-list-update-and-progress-logging.md
│       ├── [170K]  update-todo-and-log-for-project.md
│       ├── [182K]  updating-todo-and-running-development-commands.md
│       └── [260K]  updating-todos-and-project-documentation.md
├── [ 25K]  CHANGELOG.md
├── [ 986]  CLEANUP.txt
├── [1.0K]  LICENSE
├── [7.7K]  NEXT.md
├── [1.2K]  PROMPT.txt
├── [ 38K]  README.md
├── [377K]  REPO_CONTENT.txt
├── [3.7K]  TODO.md
├── [   7]  VERSION.txt
├── [ 13K]  cleanup.py
├── [ 192]  dist
├── [  96]  docs
│   └── [7.7K]  context_management.md
├── [  96]  examples
│   └── [6.9K]  backend_selection.py
├── [5.1K]  pyproject.toml
├── [ 160]  src
├── [ 544]  tests
│   ├── [  67]  __init__.py
│   ├── [ 13K]  test_backend_selector.py
│   ├── [ 13K]  test_cache.py
│   ├── [3.3K]  test_config.py
│   ├── [2.0K]  test_constants.py
│   ├── [6.0K]  test_context.py
│   ├── [2.5K]  test_context_simple.py
│   ├── [ 12K]  test_decorators.py
│   ├── [4.3K]  test_engines.py
│   ├── [6.9K]  test_exceptions.py
│   ├── [2.8K]  test_exceptions_simple.py
│   ├── [6.3K]  test_fallback.py
│   ├── [9.5K]  test_redis_cache.py
│   └── [7.6K]  test_twat_cache.py
└── [ 90K]  uv.lock

12 directories, 61 files

================
File: .cursor/rules/quality.mdc
================
---
description: Quality
globs: 
---
- **Verify Information**: Always verify information before presenting it. Do not make assumptions or speculate without clear evidence.
- **No Apologies**: Never use apologies.
- **No Whitespace Suggestions**: Don't suggest whitespace changes.
- **No Inventions**: Don't invent major changes other than what's explicitly requested.
- **No Unnecessary Confirmations**: Don't ask for confirmation of information already provided in the context.
- **Preserve Existing Code**: Don't remove unrelated code or functionalities. Pay attention to preserving existing structures.
- **No Implementation Checks**: Don't ask the user to verify implementations that are visible in the provided context.
- **No Unnecessary Updates**: Don't suggest updates or changes to files when there are no actual modifications needed.
- **No Current Implementation**: Don't show or discuss the current implementation unless specifically requested.
- **Use Explicit Variable Names**: Prefer descriptive, explicit variable names over short, ambiguous ones to enhance code readability.
- **Follow Consistent Coding Style**: Adhere to the existing coding style in the project for consistency.
- **Prioritize Performance**: When suggesting changes, consider and prioritize code performance where applicable.
- **Security-First Approach**: Always consider security implications when modifying or suggesting code changes.
- **Test Coverage**: Suggest or include appropriate unit tests for new or modified code.
- **Error Handling**: Implement robust error handling and logging where necessary.
- **Modular Design**: Encourage modular design principles to improve code maintainability and reusability.
- **Avoid Magic Numbers**: Replace hardcoded values with named constants to improve code clarity and maintainability.
- **Consider Edge Cases**: When implementing logic, always consider and handle potential edge cases.
- **Use Assertions**: Include assertions wherever possible to validate assumptions and catch potential errors early.

================
File: .github/workflows/push.yml
================
name: Build & Test
on:
  push:
    branches: [main]
    tags-ignore: ["v*"]
  pull_request:
    branches: [main]
  workflow_dispatch:
permissions:
  contents: write
  id-token: write
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
jobs:
  quality:
    name: Code Quality
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Run Ruff lint
        uses: astral-sh/ruff-action@v3
        with:
          version: "latest"
          args: "check --output-format=github"
      - name: Run Ruff Format
        uses: astral-sh/ruff-action@v3
        with:
          version: "latest"
          args: "format --check --respect-gitignore"
  test:
    name: Run Tests
    needs: quality
    strategy:
      matrix:
        python-version: ["3.10"]
        os: [ubuntu-latest]
      fail-fast: true
    runs-on: ${{ matrix.os }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install UV
        uses: astral-sh/setup-uv@v5
        with:
          version: "latest"
          python-version: ${{ matrix.python-version }}
          enable-cache: true
          cache-suffix: ${{ matrix.os }}-${{ matrix.python-version }}
      - name: Install test dependencies
        run: |
          uv pip install --system --upgrade pip
          uv pip install --system ".[test]"
      - name: Run tests with Pytest
        run: uv run pytest -n auto --maxfail=1 --disable-warnings --cov-report=xml --cov-config=pyproject.toml --cov=src/twat_cache --cov=tests tests/
      - name: Upload coverage report
        uses: actions/upload-artifact@v4
        with:
          name: coverage-${{ matrix.python-version }}-${{ matrix.os }}
          path: coverage.xml
  build:
    name: Build Distribution
    needs: test
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install UV
        uses: astral-sh/setup-uv@v5
        with:
          version: "latest"
          python-version: "3.12"
          enable-cache: true
      - name: Install build tools
        run: uv pip install build hatchling hatch-vcs
      - name: Build distributions
        run: uv run python -m build --outdir dist
      - name: Upload distribution artifacts
        uses: actions/upload-artifact@v4
        with:
          name: dist-files
          path: dist/
          retention-days: 5

================
File: .github/workflows/release.yml
================
name: Release
on:
  push:
    tags: ["v*"]
permissions:
  contents: write
  id-token: write
jobs:
  release:
    name: Release to PyPI
    runs-on: ubuntu-latest
    environment:
      name: pypi
      url: https://pypi.org/p/twat-cache
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install UV
        uses: astral-sh/setup-uv@v5
        with:
          version: "latest"
          python-version: "3.12"
          enable-cache: true
      - name: Install build tools
        run: uv pip install build hatchling hatch-vcs
      - name: Build distributions
        run: uv run python -m build --outdir dist
      - name: Verify distribution files
        run: |
          ls -la dist/
          test -n "$(find dist -name '*.whl')" || (echo "Wheel file missing" && exit 1)
          test -n "$(find dist -name '*.tar.gz')" || (echo "Source distribution missing" && exit 1)
      - name: Publish to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          password: ${{ secrets.PYPI_TOKEN }}
      - name: Create GitHub Release
        uses: softprops/action-gh-release@v1
        with:
          files: dist/*
          generate_release_notes: true
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

================
File: docs/context_management.md
================
# Cache Context Management

The `twat_cache` library provides powerful context management utilities that allow you to control the lifecycle of cache engines and ensure proper resource cleanup. This document explains how to use these context management features effectively.

## Overview

Cache context management in `twat_cache` serves several important purposes:

1. **Resource Management**: Ensures that cache resources (files, connections, memory) are properly cleaned up when no longer needed
2. **Temporary Configuration**: Allows you to temporarily override cache settings for specific code blocks
3. **Engine Selection**: Provides a way to explicitly select a specific cache engine for a block of code
4. **Error Handling**: Guarantees cleanup even when exceptions occur

## Available Context Managers

### `engine_context`

The `engine_context` function is a context manager that creates and manages a cache engine instance:

```python
from twat_cache.context import engine_context

# Basic usage
with engine_context() as cache:
    # Use the cache engine
    cache.set("key", "value")
    value = cache.get("key")

# With specific configuration
with engine_context(maxsize=100, folder_name="my_cache") as cache:
    # Use the cache engine with custom configuration
    cache.set("key", "value")

# With specific engine
with engine_context(engine_name="redis") as cache:
    # Use the Redis cache engine
    cache.set("key", "value")
```

### `CacheContext` Class

The `CacheContext` class provides a reusable context manager for working with cache engines:

```python
from twat_cache.context import CacheContext
from twat_cache.config import create_cache_config

# Create a context with default settings
context = CacheContext()
with context as cache:
    # Use the cache engine
    cache.set("key", "value")

# Create a context with specific configuration
config = create_cache_config(maxsize=100, folder_name="my_cache")
context = CacheContext(config=config)
with context as cache:
    # Use the cache engine with custom configuration
    cache.set("key", "value")

# Create a context with a specific engine
redis_context = CacheContext(engine_name="redis")
with redis_context as cache:
    # Use the Redis cache engine
    cache.set("key", "value")
```

## Manual Engine Creation

If you need more control over the engine lifecycle, you can use the `get_or_create_engine` function:

```python
from twat_cache.context import get_or_create_engine

# Create an engine
engine = get_or_create_engine(maxsize=100, folder_name="my_cache")

try:
    # Use the engine
    engine.set("key", "value")
    value = engine.get("key")
finally:
    # Important: You must manually clean up the engine
    engine.cleanup()
```

> **Warning**: When using `get_or_create_engine`, you are responsible for calling `cleanup()` on the engine when it's no longer needed. For automatic cleanup, use `engine_context` or `CacheContext` instead.

## Practical Examples

### Temporary Cache Configuration

```python
from twat_cache.context import engine_context
from twat_cache.decorators import ucache

# Default cache configuration
@ucache()
def slow_function(x):
    # ... expensive computation ...
    return result

# Override cache configuration for a specific section
with engine_context(maxsize=1000, ttl=3600) as cache:
    # Use the cache directly
    key = f"special_key_{x}"
    result = cache.get(key)
    if result is None:
        result = compute_expensive_result(x)
        cache.set(key, result)
```

### Using Different Cache Backends

```python
from twat_cache.context import engine_context

# Process small data with in-memory cache
with engine_context(engine_name="cachetools") as memory_cache:
    for item in small_data:
        process_with_cache(item, memory_cache)

# Process large data with disk cache
with engine_context(engine_name="diskcache") as disk_cache:
    for item in large_data:
        process_with_cache(item, disk_cache)
```

### Redis Cache for Distributed Applications

```python
from twat_cache.context import engine_context

# Use Redis for distributed caching
with engine_context(
    engine_name="redis",
    redis_host="redis.example.com",
    redis_port=6379,
    redis_password="secret",
    ttl=3600,
) as redis_cache:
    # Cache is now shared across all application instances
    result = redis_cache.get("shared_key")
    if result is None:
        result = compute_expensive_result()
        redis_cache.set("shared_key", result)
```

## Best Practices

1. **Use Context Managers**: Always use context managers (`with` statements) when possible to ensure proper resource cleanup.

2. **Specify Engine Requirements**: When you need specific features, explicitly specify the engine name or configuration parameters that provide those features.

3. **Handle Exceptions**: Context managers will clean up resources even when exceptions occur, but you should still handle exceptions appropriately for your application logic.

4. **Reuse Contexts**: For repeated operations with the same configuration, create a `CacheContext` instance once and reuse it.

5. **Avoid Nesting**: While it's technically possible to nest context managers, it can lead to confusion. Try to keep your cache context structure simple.

## Advanced Usage

### Custom Engine Selection Logic

```python
from twat_cache.context import engine_context
from twat_cache.engines.manager import get_engine_manager

# Get available engines
manager = get_engine_manager()
available_engines = manager.get_available_engines()

# Choose an engine based on custom logic
if "redis" in available_engines and is_distributed_environment():
    engine_name = "redis"
elif "diskcache" in available_engines and data_size > 1_000_000:
    engine_name = "diskcache"
else:
    engine_name = "cachetools"

# Use the selected engine
with engine_context(engine_name=engine_name) as cache:
    # Use the cache
    cache.set("key", "value")
```

### Combining with Decorators

```python
from twat_cache.context import engine_context
from twat_cache.decorators import ucache
from functools import wraps

def with_custom_cache(func):
    """Decorator that uses a custom cache configuration."""
    @wraps(func)
    def wrapper(*args, **kwargs):
        with engine_context(maxsize=1000, ttl=3600) as cache:
            # Create a cache key
            key = f"{func.__name__}:{args}:{kwargs}"
            
            # Check cache
            result = cache.get(key)
            if result is not None:
                return result
                
            # Call function and cache result
            result = func(*args, **kwargs)
            cache.set(key, result)
            return result
    return wrapper

# Use the custom cache decorator
@with_custom_cache
def expensive_function(x, y):
    # ... expensive computation ...
    return result
```

## Error Handling

The context managers in `twat_cache` are designed to handle errors gracefully:

```python
from twat_cache.context import engine_context
from twat_cache.exceptions import CacheOperationError

try:
    with engine_context() as cache:
        # This might raise an exception
        cache.set("key", complex_object)
        result = cache.get("key")
except CacheOperationError as e:
    # Handle cache-specific errors
    print(f"Cache operation failed: {e}")
except Exception as e:
    # Handle other errors
    print(f"An error occurred: {e}")
finally:
    # The cache engine will be cleaned up automatically
    # No need to call cache.cleanup() here
    print("Cache resources have been cleaned up")
```

## Conclusion

Cache context management in `twat_cache` provides a flexible and robust way to work with cache engines while ensuring proper resource management. By using these context management utilities, you can write cleaner, more maintainable code that efficiently handles cache resources.

================
File: examples/backend_selection.py
================
def example_type_specific_config():
    print("\n=== Example 1: Type-specific Configuration Helpers ===")
        @ucache(config=configure_for_numpy())
        def process_array(data: np.ndarray) -> np.ndarray:
            print("  Processing NumPy array...")
            time.sleep(1)  # Simulate processing time
        arr = np.array([1, 2, 3, 4, 5])
        print("  First call (cache miss):")
        start = time.time()
        result1 = process_array(arr)
        print(f"  Result: {result1}")
        print(f"  Time: {time.time() - start:.4f} seconds")
        print("\n  Second call (cache hit):")
        result2 = process_array(arr)
        print(f"  Result: {result2}")
        print("  NumPy not installed, skipping NumPy example")
    @ucache(config=configure_for_json())
    def fetch_json_data(url: str) -> dict[str, Any]:
        print(f"  Fetching JSON data from {url}...")
        time.sleep(1)  # Simulate network request
    print("\n  JSON example:")
    json_result1 = fetch_json_data("https://example.com/api/data")
    print(f"  Result: {json_result1}")
    json_result2 = fetch_json_data("https://example.com/api/data")
    print(f"  Result: {json_result2}")
def example_hybrid_caching():
    print("\n=== Example 2: Hybrid Caching Based on Result Size ===")
    @hybrid_cache()
    def get_data(size: str) -> dict[str, Any] | list[int]:
        print(f"  Generating {size} data...")
            return list(range(100000))  # Large list
    print("\n  Small data example:")
    small_result1 = get_data("small")
    print(f"  Result size: {len(str(small_result1))} bytes")
    small_result2 = get_data("small")
    print(f"  Result size: {len(str(small_result2))} bytes")
    print("\n  Large data example:")
    large_result1 = get_data("large")
    print(f"  Result size: {len(str(large_result1))} bytes")
    large_result2 = get_data("large")
    print(f"  Result size: {len(str(large_result2))} bytes")
def example_smart_caching():
    print("\n=== Example 3: Smart Caching with Automatic Backend Selection ===")
    @smart_cache()
    def process_data(data_type: str, size: int) -> Any:
        print(f"  Processing {data_type} data of size {size}...")
            return {f"key_{i}": f"value_{i}" for i in range(size)}
            return list(range(size))
            print(f"\n  {data_type.capitalize()} data of size {size}:")
            result1 = process_data(data_type, size)
            result_size = len(str(result1)) if data_type != "int" else 8
            print(f"  Result type: {type(result1).__name__}, size: {result_size} bytes")
            result2 = process_data(data_type, size)
            print(f"  Result type: {type(result2).__name__}, size: {result_size} bytes")
    print("Backend Selection Strategy Examples")
    print("==================================")
    example_type_specific_config()
    example_hybrid_caching()
    example_smart_caching()
    print("\nAll examples completed successfully!")

================
File: src/twat_cache/engines/__init__.py
================
    engines["functools"] = cast(type[BaseCacheEngine[Any, Any]], FunctoolsCacheEngine)
    engines["cachetools"] = cast(type[BaseCacheEngine[Any, Any]], CacheToolsEngine)
    engines["diskcache"] = cast(type[BaseCacheEngine[Any, Any]], DiskCacheEngine)
    engines["aiocache"] = cast(type[BaseCacheEngine[Any, Any]], AioCacheEngine)
    engines["klepto"] = cast(type[BaseCacheEngine[Any, Any]], KleptoEngine)
    engines["joblib"] = cast(type[BaseCacheEngine[Any, Any]], JoblibEngine)
    engines["cachebox"] = cast(type[BaseCacheEngine[Any, Any]], CacheBoxEngine)
    engines["redis"] = cast(type[BaseCacheEngine[Any, Any]], RedisCacheEngine)
def get_available_engines() -> list[str]:
        for name, engine_cls in engines.items()
        if hasattr(engine_cls, "is_available") and engine_cls.is_available()

================
File: src/twat_cache/engines/aiocache.py
================
class AioCacheEngine(BaseCacheEngine[P, R]):
    def is_available(cls) -> bool:
        return is_package_available("aiocache")
    def __init__(self, config: CacheConfig) -> None:
        super().__init__(config)
            raise ImportError(msg)
        if is_package_available("redis"):
            self._cache: Cache = RedisCache(
        elif is_package_available("pymemcache"):
            self._cache = MemcachedCache(
            self._cache = SimpleMemoryCache(
    def cache(self, func: Callable[P, R]) -> Callable[P, R]:
            raise RuntimeError(msg)
        return cast(
            cached(
    def get(self, key: CacheKey) -> R | None:
        return cast(R | None, self._cache.get(str(key)))
    def set(self, key: CacheKey, value: R) -> None:
        self._cache.set(str(key), value, ttl=self._config.ttl)
    def clear(self) -> None:
        self._cache.clear()
    def stats(self) -> dict[str, Any]:
            "size": len(self._cache),
    def is_available(self) -> bool:

================
File: src/twat_cache/engines/base.py
================
def is_package_available(package_name: str) -> bool:
    return util.find_spec(package_name) is not None
class BaseCacheEngine(ABC, Generic[P, R]):
    def __init__(self, config: CacheConfig) -> None:
        self._created_at = time.time()
        self.validate_config()
                self._cache_path = get_cache_path(self._config.folder_name)
                    ensure_dir_exists(self._cache_path, mode=0o700 if self._config.secure else 0o755)
                logger.error(f"Failed to initialize cache path: {e}")
                raise PathError(msg) from e
        logger.debug(f"Initialized {self.__class__.__name__} with config: {config}")
    def stats(self) -> dict[str, Any]:
            "path": str(self._cache_path) if self._cache_path else None,
            "uptime": time.time() - self._created_at,
    def validate_config(self) -> None:
                validate_cache_path(self._config.folder_name)
                logger.error(f"Invalid cache folder configuration: {e}")
                raise ConfigurationError(msg) from e
            logger.error(msg)
            raise ConfigurationError(msg)
    def is_available(cls) -> bool:
    def cleanup(self) -> None:
        logger.debug(f"Cleaning up {self.__class__.__name__} resources")
        self._cache.clear()
    def _check_ttl(self, timestamp: float | None) -> bool:
        return (time.time() - timestamp) < self._config.ttl
    def _make_key(self, func: Callable[P, R], args: Any, kwargs: Any) -> CacheKey:
        return create_cache_key(func, args, kwargs)
    def cache(self, func: Callable[P, R]) -> Callable[P, R]:
    def _get_cached_value(self, key: CacheKey) -> R | None:
    def _set_cached_value(self, key: CacheKey, value: R) -> None:
    def clear(self) -> None:
    def get(self, key: str) -> R | None:
        return self._get_cached_value(key)
    def set(self, key: str, value: R) -> None:
        self._set_cached_value(key, value)
    def name(self) -> str:
class CacheEngine(BaseCacheEngine[P, R]):
                oldest_key = next(iter(self._cache))

================
File: src/twat_cache/engines/cachebox.py
================
class CacheBoxEngine(BaseCacheEngine[P, R]):
    def is_available(cls) -> bool:
        return is_package_available("cachebox")
    def __init__(self, config: CacheConfig) -> None:
        super().__init__(config)
            raise ImportError(msg)
        cache_type = cache_types.get(config.policy, LRUCache)
        self._cache: Cache = cache_type(
    def cache(self, func: Callable[P, R]) -> Callable[P, R]:
            raise RuntimeError(msg)
        return cast(
            cached(
    def get(self, key: CacheKey) -> R | None:
        return cast(R | None, self._cache.get(str(key)))
    def set(self, key: CacheKey, value: R) -> None:
        self._cache[str(key)] = value
    def clear(self) -> None:
        self._cache.clear()
    def stats(self) -> dict[str, Any]:
            "size": len(self._cache),
    def is_available(self) -> bool:

================
File: src/twat_cache/engines/cachetools.py
================
class CacheToolsEngine(BaseCacheEngine[P, R]):
    def is_available(cls) -> bool:
        return is_package_available("cachetools")
    def __init__(self, config: CacheConfig) -> None:
        super().__init__(config)
            raise ImportError(msg)
            cache_type = cache_types.get(config.policy, LRUCache)
        self._cache: Cache = cache_type(
    def cache(self, func: Callable[P, R]) -> Callable[P, R]:
            raise RuntimeError(msg)
        return cast(
            cached(
    def get(self, key: CacheKey) -> R | None:
        return cast(R | None, self._cache.get(str(key)))
    def set(self, key: CacheKey, value: R) -> None:
        self._cache[str(key)] = value
    def clear(self) -> None:
        self._cache.clear()
    def stats(self) -> dict[str, Any]:
            "hits": getattr(self._cache, "hits", 0),
            "misses": getattr(self._cache, "misses", 0),
            "size": len(self._cache),
            "ttl": getattr(self._cache, "ttl", None),
    def is_available(self) -> bool:

================
File: src/twat_cache/engines/common.py
================
K = TypeVar("K")
V = TypeVar("V")
def ensure_dir_exists(dir_path: Path, mode: int = 0o700) -> None:
        dir_path.mkdir(mode=mode, parents=True, exist_ok=True)
        logger.error(f"Failed to create cache directory {dir_path}: {e}")
        raise PathError(msg) from e
def safe_key_serializer(key: Any) -> str:
        if isinstance(key, str):
        if isinstance(key, int | float | bool | type(None)):
            return str(key)
        if isinstance(key, list | tuple | set):
            return json.dumps([safe_key_serializer(k) for k in key])
        if isinstance(key, dict):
            return json.dumps({str(k): safe_key_serializer(v) for k, v in key.items()})
        return json.dumps(repr(key))
        logger.error(f"Failed to serialize cache key: {e}")
        raise CacheKeyError(msg) from e
def safe_value_serializer(value: Any) -> str:
        return json.dumps(value)
            return repr(value)
            logger.error(f"Failed to serialize cache value: {inner_e}")
            raise CacheValueError(
def safe_temp_file(
        fd, path = tempfile.mkstemp(prefix=prefix, suffix=suffix)
        os.chmod(path, 0o600)  # Ensure secure permissions
        return Path(path), os.fdopen(fd, "w+b")
        logger.error(f"Failed to create temporary file: {e}")
        raise ResourceError(msg) from e
def get_func_qualified_name(func: Callable[..., Any]) -> str:
    module = inspect.getmodule(func)
    if hasattr(func, "__qualname__"):
def create_cache_key(func: Callable[P, R], args: Any, kwargs: Any) -> CacheKey:
    func_name = get_func_qualified_name(func)
    hashable_args = tuple(args)
    hashable_kwargs = tuple(sorted((k, v) for k, v in kwargs.items()))

================
File: src/twat_cache/engines/diskcache.py
================
class DiskCacheEngine(BaseCacheEngine[P, R]):
    def is_available(cls) -> bool:
    def __init__(self, config: CacheConfig) -> None:
        super().__init__(config)
            logger.error("Cache path is required for disk cache")
            raise EngineNotAvailableError(
        if not self.is_available():
            logger.error("diskcache is not available")
            ensure_dir_exists(
            self._disk_cache = Cache(
                directory=str(self._cache_path),
            logger.debug(f"Initialized DiskCache at {self._cache_path}")
            logger.error(f"Failed to initialize DiskCache: {e}")
            raise ResourceError(msg) from e
    def cleanup(self) -> None:
            if hasattr(self, "_disk_cache") and self._disk_cache is not None:
                self._disk_cache.close()
                logger.debug(f"Closed DiskCache at {self._cache_path}")
            logger.error(f"Error closing DiskCache: {e}")
            super().cleanup()
    def cache(self, func: Callable[P, R]) -> Callable[P, R]:
        @wraps(func)
        def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
            key = self._make_key(func, args, kwargs)
                cache_result = self._disk_cache.get(key, default=None, expire_time=True)
                    if isinstance(cache_result, tuple) and len(cache_result) >= 2:
                            isinstance(expire_time, int | float)
                            and time.time() < expire_time
                            logger.trace(f"Cache hit for {func.__name__}")
                            return cast(R, cached_value)
                            self._disk_cache.delete(key)
                logger.trace(f"Cache miss for {func.__name__}")
                result = func(*args, **kwargs)
                    expire_time = time.time() + self._config.ttl
                self._disk_cache.set(
                    self._size = self._disk_cache.volume()
                logger.error(f"Error in DiskCache: {e}")
                return func(*args, **kwargs)
    def clear(self) -> None:
                self._disk_cache.clear()
                logger.debug(f"Cleared DiskCache at {self._cache_path}")
            logger.error(f"Error clearing DiskCache: {e}")
            raise CacheOperationError(msg) from e
    def stats(self) -> dict[str, Any]:
        base_stats = super().stats
                disk_stats_dict = self._disk_cache.stats(enable=True)
                if isinstance(disk_stats_dict, dict):
                    base_stats["disk_hits"] = disk_stats_dict.get("hits", 0)
                    base_stats["disk_misses"] = disk_stats_dict.get("misses", 0)
                    base_stats["disk_size"] = self._disk_cache.volume()
            logger.error(f"Error getting DiskCache stats: {e}")

================
File: src/twat_cache/engines/functools_engine.py
================
class FunctoolsCacheEngine(BaseCacheEngine[P, R]):
    def __init__(self, config: CacheConfig) -> None:
        super().__init__(config)
        self._maxsize = config.maxsize or float("inf")
        logger.debug(
    def get(self, key: str) -> R | None:
            return self._cache.cache_getitem(lambda k: k, key)  # type: ignore
    def set(self, key: str, value: R) -> None:
        self._cache(lambda k: k)(key)  # type: ignore
        self._cache.cache_setitem(lambda k: k, key, value)  # type: ignore
    def clear(self) -> None:
        self._cache.clear()
    def name(self) -> str:
    def stats(self) -> dict[str, Any]:
            "current_size": len(self._cache),
            "size": len(self._cache),
    def validate_config(self) -> None:
        super().validate_config()
        if maxsize is not None and not isinstance(maxsize, int):
            raise ValueError(msg)
    def _get_backend_key_components(self) -> list[str]:
    def cache(
        def decorator(f: Callable[P, R]) -> Callable[P, R]:
            @wraps(f)
            def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
                key = json.dumps(tuple(args) + tuple(kwargs.items()), sort_keys=True)
                result = self._get_cached_value(key)
                        result = f(*args, **kwargs)
                        self._set_cached_value(key, result)
                        self._set_cached_value(key, cast(R, e))
                elif isinstance(result, Exception):
        return decorator(func)
    def _get_cached_value(self, key: CacheKey) -> R | None:
        if expiry is not None and time.time() >= expiry:
    def _set_cached_value(self, key: CacheKey, value: R) -> None:
        if len(self._cache) >= self._maxsize:
                oldest_key = next(iter(self._cache))
            time.time() + self._config.ttl if self._config.ttl is not None else None
    def is_available(cls) -> bool:

================
File: src/twat_cache/engines/functools.py
================
class FunctoolsCacheEngine(BaseCacheEngine[P, R]):
    def is_available(cls) -> bool:
    def __init__(self, config: CacheConfig) -> None:
        super().__init__(config)
    def cache(self, func: Callable[P, R]) -> Callable[P, R]:
            wrapper = functools.lru_cache(maxsize=self._maxsize)(func)
            wrapper = functools.cache(func)
        def cached_func(*args: P.args, **kwargs: P.kwargs) -> R:
            key = self._make_key(*args, **kwargs)
                now = time.time()
                        self.clear()
                        result = func(*args, **kwargs)
                result = wrapper(*args, **kwargs)
                self._last_access[key] = time.time()
                logger.warning(f"Cache miss for {key}: {e}")
    def clear(self) -> None:
        if hasattr(self, "_cache"):
            self._cache.clear()
            self._last_access.clear()
    def validate_config(self) -> None:
            raise ValueError(msg)
    def stats(self) -> dict[str, Any]:
            "size": len(self._cache),

================
File: src/twat_cache/engines/joblib.py
================
class DummyMemory:
    def cache(self, func: F) -> F:
    def clear(self) -> None:
class JoblibEngine(BaseCacheEngine[P, R]):
    def __init__(self, config: CacheConfig) -> None:
        super().__init__(config)
        folder_name = getattr(self._config, "folder_name", None)
            JoblibMemory(str(get_cache_path(self._folder_name)), verbose=0)
            else DummyMemory()
    def _get_cached_value(self, key: CacheKey) -> R | None:
    def _set_cached_value(self, key: CacheKey, value: R) -> None:
        self._memory.clear()
    def is_available(cls) -> bool:
    def name(self) -> str:
        return cast(F, self._memory.cache(func))

================
File: src/twat_cache/engines/klepto.py
================
class KleptoEngine(BaseCacheEngine[P, R]):
    def is_available(cls) -> bool:
        return is_package_available("klepto")
    def __init__(self, config: CacheConfig) -> None:
        super().__init__(config)
            raise ImportError(msg)
        maxsize = config.maxsize or float("inf")
            self._cache = lru_cache(maxsize=maxsize)
            self._cache = lfu_cache(maxsize=maxsize)
            self._cache = rr_cache(maxsize=maxsize)
            self._cache = sql_archive(
                str(config.cache_dir / "cache.db"),
            self._cache = file_archive(
                str(config.cache_dir),
        self._cache.load()
        logger.debug(
    def _get_cached_value(self, key: CacheKey) -> R | None:
                if time.time() >= self._expiry[key]:
                        self._cache.sync()
            return cast(R, self._cache[key])
            logger.warning(f"Error retrieving from klepto cache: {e}")
    def _set_cached_value(self, key: CacheKey, value: R) -> None:
                self._expiry[key] = time.time() + self._config.ttl
            self._size = len(self._cache)
            logger.warning(f"Error storing to klepto cache: {e}")
    def clear(self) -> None:
            self._cache.clear()
            self._expiry.clear()
            logger.warning(f"Error clearing klepto cache: {e}")
    def stats(self) -> dict[str, Any]:
        base_stats = super().stats
            base_stats.update(
                    "currsize": len(self._cache),
                        type(self._cache).__name__ if self._cache else None
            logger.warning(f"Error getting klepto cache stats: {e}")
    def __del__(self) -> None:
                self._cache.close()
            logger.warning(f"Error closing klepto cache: {e}")
    def cache(self, func: Callable[P, R]) -> Callable[P, R]:
            raise RuntimeError(msg)
        return cast(
            self._cache(func),
    def get(self, key: CacheKey) -> R | None:
        return cast(R | None, self._cache.get(str(key)))
    def set(self, key: CacheKey, value: R) -> None:
        self._cache[str(key)] = value
        self._cache.dump()
            "hits": getattr(self._cache, "hits", 0),
            "misses": getattr(self._cache, "misses", 0),
            "size": len(self._cache),

================
File: src/twat_cache/engines/manager.py
================
E = TypeVar("E", bound=BaseCacheEngine[Any, Any])
    logger.debug("aiocache not available")
    logger.debug("cachetools not available")
    logger.debug("klepto not available")
    logger.debug("joblib not available")
    logger.debug("redis not available")
class CacheEngineManager:
    def __init__(self) -> None:
        self._register_builtin_engines()
    def _register_builtin_engines(self) -> None:
        self.register_engine("functools", FunctoolsCacheEngine)
        self.register_engine("cachetools", CacheToolsEngine)
        self.register_engine("diskcache", DiskCacheEngine)
        self.register_engine("aiocache", AioCacheEngine)
        self.register_engine("klepto", KleptoEngine)
        self.register_engine("joblib", JoblibEngine)
        self.register_engine("cachebox", CacheBoxEngine)
            self.register_engine("redis", RedisCacheEngine)
    def register_engine(self, name: str, engine_cls: type[E]) -> None:
            logger.warning(f"Overwriting existing engine registration for {name}")
        self._engines[name] = cast(type[BaseCacheEngine[Any, Any]], engine_cls)
    def get_engine(self, name: str) -> type[BaseCacheEngine[Any, Any]] | None:
        return self._engines.get(name)
    def list_engines(self) -> list[str]:
        return list(self._engines.keys())
    def get_available_engines(self) -> list[str]:
        return [name for name, engine in self._engines.items() if engine.is_available()]
    def select_engine(
        available = self.get_available_engines()
            logger.warning("No cache engines are available")
                    engine_cls: type[BaseCacheEngine[Any, Any]] | None = self.get_engine(engine_name)
                    if engine_cls and engine_cls.is_available():
        fallback: type[BaseCacheEngine[Any, Any]] | None = self.get_engine(available[0])
        if fallback and fallback.is_available():
def get_engine_manager() -> CacheEngineManager:
    return CacheEngineManager()

================
File: src/twat_cache/engines/py.typed
================
1

================
File: src/twat_cache/engines/redis.py
================
class RedisCacheEngine(BaseCacheEngine[P, R]):
    def is_available(cls) -> bool:
        return is_package_available("redis")
    def __init__(self, config: CacheConfig) -> None:
        if not self.is_available():
            raise EngineNotAvailableError(msg)
        super().__init__(config)
        host = config.get_redis_host() if hasattr(config, "get_redis_host") else "localhost"
        port = config.get_redis_port() if hasattr(config, "get_redis_port") else 6379
        db = config.get_redis_db() if hasattr(config, "get_redis_db") else 0
        password = config.get_redis_password() if hasattr(config, "get_redis_password") else None
            self._redis = redis.Redis(
            self._redis.ping()
            logger.debug(f"Connected to Redis at {host}:{port} (db={db})")
            logger.error(msg)
            raise ConfigurationError(msg) from e
    def validate_config(self) -> None:
        super().validate_config()
        if hasattr(self._config, "get_redis_port") and self._config.get_redis_port() is not None:
            port = self._config.get_redis_port()
            if not isinstance(port, int) or port <= 0 or port > 65535:
                raise ConfigurationError(msg)
    def cleanup(self) -> None:
            if hasattr(self, "_redis"):
                self._redis.close()
                logger.debug("Closed Redis connection")
            logger.error(f"Error closing Redis connection: {e}")
    def _get_full_key(self, key: CacheKey) -> str:
        if isinstance(key, tuple):
            key_str = str(hash(key))
            key_str = str(key)
    def _get_cached_value(self, key: CacheKey) -> R | None:
            full_key = self._get_full_key(key)
            data = self._redis.get(full_key)
                    data = zlib.decompress(data)
                value = pickle.loads(data)
                self._last_access = time.time()
                return cast(R, value)
                logger.error(f"Error deserializing cached value: {e}")
            logger.error(f"Error getting cached value from Redis: {e}")
    def _set_cached_value(self, key: CacheKey, value: R) -> None:
                data = pickle.dumps(value)
                    data = zlib.compress(data)
                logger.error(f"Error serializing value: {e}")
                raise CacheOperationError(msg) from e
                self._redis.setex(full_key, int(self._config.ttl), data)
                self._redis.set(full_key, data)
            self._last_update = time.time()
            logger.error(f"Error setting cached value in Redis: {e}")
    def clear(self) -> None:
            keys = self._redis.keys(pattern)
                self._redis.delete(*keys)
                logger.debug(f"Cleared {len(keys)} keys from Redis namespace {self._namespace}")
            logger.error(f"Error clearing Redis cache: {e}")
    def cache(self, func: Callable[P, R]) -> Callable[P, R]:
        @wraps(func)
        def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
            key = self._make_key(func, args, kwargs)
            cached = self._get_cached_value(key)
            result = func(*args, **kwargs)
            self._set_cached_value(key, result)
    def stats(self) -> dict[str, Any]:
        base_stats = super().stats
        current_time = time.time()
                size = len(self._redis.keys(pattern))
                logger.error(f"Error getting Redis cache size: {e}")
            info = self._redis.info()
            redis_stats.update(
                    "redis_version": info.get("redis_version"),
                    "used_memory": info.get("used_memory"),
                    "used_memory_human": info.get("used_memory_human"),
                    "connected_clients": info.get("connected_clients"),
            logger.debug(f"Error getting Redis info: {e}")

================
File: src/twat_cache/types/cachebox.pyi
================
from typing import Any, TypeVar, Generic

T = TypeVar("T")

class Cache(Generic[T]):
    def __init__(self, maxsize: int = 0) -> None: ...
    def get(self, key: Any) -> T | None: ...
    def __setitem__(self, key: Any, value: T) -> None: ...
    def clear(self) -> None: ...
    def memoize(self) -> Any: ...

================
File: src/twat_cache/__init__.py
================


================
File: src/twat_cache/__main__.py
================
T = TypeVar("T")
def get_cache_path(folder_name: str | None = None) -> Path:
    def generate_uuid() -> str:
        caller_frame = inspect.stack()[2]
        caller_path = Path(caller_file).resolve()
        return str(uuid.uuid5(uuid.NAMESPACE_URL, str(caller_path)))
        folder_name = generate_uuid()
        Path.home() / ".cache" / "twat_cache" / str(folder_name)
    cache_dir.mkdir(parents=True, exist_ok=True)
    DISK_CACHE: Any | None = Cache(get_cache_path("twat_cache"))
    JOBLIB_MEMORY: Any | None = Memory(get_cache_path("twat_cache"), verbose=0)
def ucache(
    def decorator(func: Callable[..., T]) -> Callable[..., T]:
        config = CacheConfig(
        return twat_cache(config)(func)

================
File: src/twat_cache/backend_selector.py
================
F = TypeVar("F", bound=Callable[..., Any])
R = TypeVar("R")
class DataSize(Enum):
class DataPersistence(Enum):
class AccessPattern(Enum):
class ConcurrencyLevel(Enum):
def estimate_data_size(data: Any) -> DataSize:
        size = sys.getsizeof(data)
        if isinstance(data, list | tuple | set | dict):
            if isinstance(data, list | tuple | set):
                    size += sys.getsizeof(item)
            elif isinstance(data, dict):
                for key, value in data.items():
                    size += sys.getsizeof(key) + sys.getsizeof(value)
def get_type_based_backend(data_type: type) -> str:
    for special_type, backend in TYPE_TO_BACKEND.items():
        if isinstance(special_type, str) and special_type == type_name:
    for base_type, backend in TYPE_TO_BACKEND.items():
        if not isinstance(base_type, str) and issubclass(data_type, base_type):
def get_available_backend(preferred_backend: str) -> str:
    if is_backend_available(preferred_backend):
    for alt_backend in ALTERNATIVE_BACKENDS.get(preferred_backend, []):
        if is_backend_available(alt_backend):
            logger.debug(f"Preferred backend '{preferred_backend}' not available, using '{alt_backend}' instead")
    logger.debug("No suitable backend found, defaulting to 'functools'")
def is_backend_available(backend_name: str) -> bool:
    package_name = package_map.get(backend_name)
        return is_package_available(package_name)
def select_backend_for_data(
        backends.append(config.preferred_engine)
        data_type = type(data)
        backends.append(get_type_based_backend(data_type))
        size = estimate_data_size(data)
        backends.append(SIZE_TO_BACKEND[size])
        backends.append(PERSISTENCE_TO_BACKEND[persistence])
        backends.append(ACCESS_TO_BACKEND[access_pattern])
        backends.append(CONCURRENCY_TO_BACKEND[concurrency])
    backends.append("cachetools")
        available_backend = get_available_backend(backend)
def configure_for_type(data_type: type, **kwargs) -> CacheConfig:
    backend = get_type_based_backend(data_type)
        config_params["folder_name"] = f"{data_type.__name__.lower()}_cache"
        config_params["folder_name"] = f"{data_type.__name__.lower()}_joblib"
    config_params.update(kwargs)
    return CacheConfig(**config_params)
def configure_for_numpy() -> CacheConfig:
        return configure_for_type(np.ndarray, folder_name="numpy_cache")
        logger.warning("NumPy not available, returning generic configuration")
        return CacheConfig(preferred_engine=get_available_backend("joblib"))
def configure_for_pandas() -> CacheConfig:
        return configure_for_type(pd.DataFrame, folder_name="pandas_cache")
        logger.warning("Pandas not available, returning generic configuration")
def configure_for_images() -> CacheConfig:
    return CacheConfig(
        preferred_engine=get_available_backend("diskcache"),
def configure_for_json() -> CacheConfig:
def hybrid_cache_config(
        "small_result_engine": get_available_backend(small_result_engine),
        "large_result_engine": get_available_backend(large_result_engine),
def analyze_function_return_type(func: Callable) -> type | None:
        hints = get_type_hints(func)
def smart_cache_config(func: F = None, **kwargs) -> CacheConfig:
        return_type = analyze_function_return_type(func)
            backend = get_type_based_backend(return_type)
            config_params["preferred_engine"] = get_available_backend(backend)
def detect_result_type(result: Any) -> str:
    return select_backend_for_data(data=result)

================
File: src/twat_cache/cache.py
================
class CacheStats(NamedTuple):
class CacheEntry(NamedTuple):
def register_cache(
    _active_caches[name] = CacheEntry(cache, wrapper, stats)
    logger.debug(f"Registered cache: {name}")
def clear_cache(name: str | None = None) -> None:
            _active_caches[name].cache.clear()
            _active_caches[name] = entry._replace(
                stats=CacheStats(0, 0, 0, entry.stats.maxsize)
    for cache_name, entry in list(_active_caches.items()):  # Iterate over a copy
        entry.cache.clear()
        _active_caches[cache_name] = entry._replace(
    cache_dir = Path.home() / ".cache" / "twat_cache"
    if cache_dir.exists():
        for path in cache_dir.glob("*"):
            if path.is_dir():
                    for file in path.glob("*"):
                        file.unlink()
                    path.rmdir()
                    logger.warning(f"Failed to clear cache directory {path}: {e}")
        logger.debug("Cleared cache directories")
def get_stats(name: str | None = None) -> dict[str, Any]:
        "total_caches": len(_active_caches),
    for entry in _active_caches.values():
def update_stats(
    _active_caches[name] = CacheEntry(
        CacheStats(hits, misses, new_size, entry.stats.maxsize),

================
File: src/twat_cache/config.py
================
class CacheConfig(BaseModel):
    maxsize: int | None = Field(default=None, ge=1)
    ttl: float | None = Field(default=None, ge=0)
    class Config:
    @field_validator("maxsize")
    def validate_maxsize(cls, v: int | None) -> int | None:
            raise ValueError(msg)
    @field_validator("ttl")
    def validate_ttl(cls, v: float | None) -> float | None:
    @field_validator("policy")
    def validate_policy(cls, v: EvictionPolicy) -> EvictionPolicy:
    @field_validator("redis_port")
    def validate_redis_port(cls, v: int) -> int:
    def to_dict(self) -> dict[str, Any]:
    def from_dict(cls, data: dict[str, Any]) -> "CacheConfig":
        return cls(**data)
    def get_maxsize(self) -> int | None:
    def get_folder_name(self) -> str | None:
    def get_use_sql(self) -> bool:
    def get_preferred_engine(self) -> str | None:
    def get_cache_type(self) -> str | None:
    def get_redis_host(self) -> str:
        return os.environ.get("TWAT_CACHE_REDIS_HOST", self.redis_host)
    def get_redis_port(self) -> int:
        port_str = os.environ.get("TWAT_CACHE_REDIS_PORT")
                return int(port_str)
    def get_redis_db(self) -> int:
        db_str = os.environ.get("TWAT_CACHE_REDIS_DB")
                return int(db_str)
    def get_redis_password(self) -> str | None:
        return os.environ.get("TWAT_CACHE_REDIS_PASSWORD", self.redis_password)
    def validate_config(self) -> None:
def create_cache_config(
    return CacheConfig(

================
File: src/twat_cache/context.py
================
E = TypeVar("E", bound=BaseCacheEngine[Any, Any])
def engine_context(
    engine_config = config or create_cache_config(**kwargs)
    manager = get_engine_manager()
        engine_cls = manager.get_engine(engine_name)
            logger.error(f"Engine '{engine_name}' not found")
            raise EngineError(msg)
        engine_cls = manager.select_engine(engine_config)
            logger.error("No suitable engine found")
    engine = engine_cls(engine_config)
    logger.debug(f"Created engine {engine.__class__.__name__}")
            logger.debug(f"Cleaning up engine {engine.__class__.__name__}")
            engine.cleanup()
            logger.error(f"Error during engine cleanup: {e}")
class CacheContext:
    def __init__(
        self.config = config or create_cache_config(**kwargs)
    def __enter__(self) -> BaseCacheEngine[Any, Any]:
            engine_cls = manager.get_engine(self.engine_name)
                logger.error(f"Engine '{self.engine_name}' not found")
            engine_cls = manager.select_engine(self.config)
        self.engine = engine_cls(self.config)
        logger.debug(f"Created engine {self.engine.__class__.__name__}")
    def __exit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None:
                logger.debug(f"Cleaning up engine {self.engine.__class__.__name__}")
                self.engine.cleanup()
def get_or_create_engine(

================
File: src/twat_cache/decorators.py
================
class CacheDecorator(Protocol[P, R]):
    def __call__(self, func: Callable[P, R]) -> Callable[P, R]: ...
class AsyncCacheDecorator(Protocol[P, AsyncR]):
    def __call__(
HAS_AIOCACHE = bool(importlib.util.find_spec("aiocache"))
HAS_CACHEBOX = bool(importlib.util.find_spec("cachebox"))
HAS_CACHETOOLS = bool(importlib.util.find_spec("cachetools"))
HAS_DISKCACHE = bool(importlib.util.find_spec("diskcache"))
HAS_JOBLIB = bool(importlib.util.find_spec("joblib"))
HAS_KLEPTO = bool(importlib.util.find_spec("klepto"))
def get_cache_dir(folder_name: str | None = None) -> Path:
    if folder_name and not all(c.isalnum() or c in "-_." for c in folder_name):
        raise ValueError(msg)
    base_dir = Path.home() / ".cache" / "twat_cache"
        cache_dir.mkdir(parents=True, exist_ok=True)
        raise OSError(msg) from e
def make_key(serializer: Callable[[Any], str] | None = None, *args: Any, **kwargs: Any) -> str:
    def default_serializer(obj: Any) -> Any:
        if isinstance(obj, str | int | float | bool | type(None)):
        return str(obj)
    converted_args = tuple(ser(arg) for arg in args)
    converted_kwargs = {k: ser(v) for k, v in sorted(kwargs.items())}
    return json.dumps((converted_args, converted_kwargs), sort_keys=True)
def mcache(
    logger.debug(f"Creating memory cache with maxsize={maxsize}, ttl={ttl}, policy={policy}")
    config = create_cache_config(
        logger.debug("Using CacheBox engine for memory caching")
        engine: BaseCacheEngine[P, R] = cachebox.CacheBoxEngine(config)  # type: ignore[abstract]
        return cast(CacheDecorator[P, R], engine.cache)
        logger.debug("Using CacheTools engine for memory caching")
        engine = cachetools.CacheToolsEngine(config)  # type: ignore[abstract]
        logger.debug("Using functools engine for memory caching (fallback)")
        engine = functools_engine.FunctoolsEngine(config)  # type: ignore[abstract]
def bcache(
    logger.debug(f"Creating disk cache with folder={folder_name}, maxsize={maxsize}, ttl={ttl}, use_sql={use_sql}")
        logger.debug("Using DiskCache engine for disk caching")
        engine: BaseCacheEngine[P, R] = diskcache.DiskCacheEngine(config)  # type: ignore[abstract]
        logger.debug("Using Klepto engine for disk caching (SQL)")
        engine = klepto.KleptoEngine(config)
        logger.warning("No disk cache backends available, falling back to memory cache")
        return mcache(maxsize=maxsize, ttl=ttl, policy=policy, secure=secure)
def fcache(
    logger.debug(f"Creating file cache with folder={folder_name}, maxsize={maxsize}, ttl={ttl}, compress={compress}")
        logger.debug("Using Joblib engine for file caching")
        engine: BaseCacheEngine[P, R] = JoblibEngine(config)
        logger.debug("Using Klepto engine for file caching (fallback)")
        engine = KleptoEngine(config)
        logger.warning("No file cache backends available, falling back to memory cache")
        return mcache(maxsize=maxsize, ttl=ttl)
def acache(
    logger.debug(f"Creating async cache with maxsize={maxsize}, ttl={ttl}, policy={policy}")
        logger.debug("Using AioCache engine for async caching")
        def decorator(
            engine: Any = aiocache.AioCacheEngine(config)  # type: ignore[abstract]
            cache_func = engine.cache(func)
            @functools.wraps(func)
            async def wrapper(*args: P.args, **kwargs: P.kwargs) -> AsyncR:
                return await cache_func(*args, **kwargs)
        logger.warning("AioCache not available, using memory cache with async wrapper")
        mem_cache = mcache(maxsize=maxsize, ttl=ttl, policy=str(policy))
            cached_func = mem_cache(func)  # type: ignore
                if asyncio.iscoroutinefunction(func):
                    return await cached_func(*args, **kwargs)  # type: ignore
                    return await cached_func(*args, **kwargs)
def _get_available_backends() -> dict[str, bool]:
def _select_best_backend(
    backends = _get_available_backends()
    if preferred and backends.get(preferred):
def _create_engine(config: CacheConfig, func: Callable[P, R]) -> BaseCacheEngine[P, R]:
    is_async = asyncio.iscoroutinefunction(func)
    backend = _select_best_backend(config.preferred_engine, is_async=is_async, needs_disk=needs_disk)
        return aiocache.AioCacheEngine(config)  # type: ignore[abstract]
        return cachebox.CacheBoxEngine(config)  # type: ignore[abstract]
        return cachetools.CacheToolsEngine(config)  # type: ignore[abstract]
        return diskcache.DiskCacheEngine(config)  # type: ignore[abstract]
        return joblib.JoblibEngine(config)  # type: ignore[abstract]
        return klepto.KleptoEngine(config)  # type: ignore[abstract]
        return functools_engine.FunctoolsCacheEngine(config)  # type: ignore[abstract]
def ucache(
    logger.debug(f"Creating universal cache with preferred_engine={preferred_engine}")
    def decorator(func: Callable[P, R]) -> Callable[P, R]:
        backend = preferred_engine or _select_best_backend(is_async=is_async, needs_disk=needs_disk)
        logger.debug(f"Selected {backend} backend for {func.__name__}")
        engine = _create_engine(config, func)
        cached_func = engine.cache(func)
        def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
                return cached_func(*args, **kwargs)
                logger.warning(f"Cache error in {func.__name__}: {e}. Executing without cache.")
                return func(*args, **kwargs)

================
File: src/twat_cache/exceptions.py
================
class TwatCacheError(Exception):
    def __init__(self, message: str, *args: Any) -> None:
        super().__init__(message, *args)
class ConfigurationError(TwatCacheError):
class EngineError(TwatCacheError):
class EngineNotAvailableError(EngineError):
    def __init__(self, message: str, reason: str | None = None) -> None:
        super().__init__(message)
class CacheOperationError(TwatCacheError):
class CacheKeyError(CacheOperationError):
class CacheValueError(CacheOperationError):
class SerializationError(CacheOperationError):
class ResourceError(TwatCacheError):
class ConcurrencyError(ResourceError):
class PathError(ResourceError):

================
File: src/twat_cache/hybrid_cache.py
================
def hybrid_cache(
        small_result_config = CacheConfig(preferred_engine="cachetools", maxsize=128)
        large_result_config = CacheConfig(preferred_engine="diskcache", compress=True)
    def decorator(func: Callable[P, R]) -> Callable[P, R]:
        @functools.wraps(func)
        def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
            cached_func = ucache(
            result = cached_func(*args, **kwargs)
                    size = sys.getsizeof(result)
                    if isinstance(result, list | tuple | set):
                            size += sys.getsizeof(item)
                    elif isinstance(result, dict):
                        for key, value in result.items():
                            size += sys.getsizeof(key) + sys.getsizeof(value)
                        logger.debug(f"Using large result config for {func_key} (size: {size} bytes)")
                        logger.debug(f"Using small result config for {func_key} (size: {size} bytes)")
                    logger.debug(f"Could not determine size for {func_key}, using small result config")
def smart_cache(
        config = CacheConfig()
            result = func(*args, **kwargs)
            backend = detect_result_type(result)
                return cached_func(*args, **kwargs)
            backend_config = CacheConfig(
                preferred_engine=backend, **{k: v for k, v in config.model_dump().items() if v is not None}

================
File: src/twat_cache/logging.py
================
logger.remove()
LOG_LEVEL = os.environ.get("TWAT_CACHE_LOG_LEVEL", "INFO").upper()
logger.add(
if os.environ.get("TWAT_CACHE_LOG_FILE", "0") == "1":
        xdg_cache_home = os.environ.get("XDG_CACHE_HOME")
            log_dir = Path(xdg_cache_home) / "logs"
            log_dir = Path.home() / ".cache" / "logs"
        log_dir.mkdir(parents=True, exist_ok=True)
            str(log_file),
        logger.info(f"File logging enabled at {log_file}")
        logger.warning(f"Failed to setup file logging: {e}")
        logger.info("Falling back to console-only logging")

================
File: src/twat_cache/paths.py
================
    logger.warning("platformdirs not available, falling back to ~/.cache")
def get_cache_path(
        folder_name = f"twat_cache_{uuid.uuid4().hex[:8]}"
        base_path = Path(tempfile.gettempdir()) / "twat_cache"
        base_path = Path(platformdirs.user_cache_dir("twat_cache"))
        base_path = Path.home() / ".cache" / "twat_cache"
    if create and not cache_path.exists():
            cache_path.mkdir(parents=True, exist_ok=True)
            logger.debug(f"Created cache directory: {cache_path}")
            logger.error(f"Failed to create cache directory {cache_path}: {e}")
def validate_cache_path(path: str | Path) -> None:
        path = Path(path).resolve()
        if not path.exists():
            path.mkdir(parents=True, exist_ok=True)
        elif not path.is_dir():
            raise ValueError(msg)
        raise ValueError(msg) from e
def clear_cache(folder_name: str | None = None) -> None:
        cache_path = get_cache_path(folder_name, create=False)
        if cache_path.exists():
                for item in cache_path.iterdir():
                    if item.is_file():
                        item.unlink()
                    elif item.is_dir():
                        shutil.rmtree(item)
                logger.info(f"Cleared cache directory: {cache_path}")
                logger.error(f"Failed to clear cache directory {cache_path}: {e}")
            base_paths.append(Path(platformdirs.user_cache_dir("twat_cache")))
        base_paths.append(Path.home() / ".cache" / "twat_cache")
            if base_path.exists():
                    shutil.rmtree(base_path)
                    logger.info(f"Cleared cache base directory: {base_path}")
                    logger.error(

================
File: src/twat_cache/py.typed
================
1

================
File: src/twat_cache/type_defs.py
================
T = TypeVar("T")
P = ParamSpec("P")
R = TypeVar("R")
F = TypeVar("F", bound=Callable[..., Any])
AsyncR = TypeVar("AsyncR")
class CacheDecorator(Protocol[P, R]):
    def __call__(self, func: Callable[P, R]) -> Callable[P, R]: ...
class AsyncCacheDecorator(Protocol[P, AsyncR]):
    def __call__(
class CacheConfig(Protocol):
    def get_maxsize(self) -> int | None:
    def get_folder_name(self) -> str | None:
    def get_use_sql(self) -> bool:
    def get_preferred_engine(self) -> str | None:
    def get_cache_type(self) -> str | None:
    def validate_config(self) -> None:
    def model_dump(self) -> dict[str, Any]:
class CacheStats(ABC):
    def hits(self) -> int:
    def misses(self) -> int:
    def size(self) -> int:
    def maxsize(self) -> int | None:
    def clear(self) -> None:
class CacheEngine(ABC, Generic[P, R]):
    def __init__(self, config: CacheConfig) -> None:
        self.validate_config()
    def cache(self, func: Callable[P, R]) -> Callable[P, R]:
    def stats(self) -> dict[str, Any]:
class KeyMaker(Protocol):
    def __call__(self, *args: Any, **kwargs: Any) -> CacheKey:
class Serializer(Protocol):
    def dumps(self, obj: Any) -> bytes:
    def loads(self, data: bytes) -> Any:

================
File: src/twat_cache/utils.py
================
def get_cache_dir(folder_name: str | None = None) -> Path:
    if folder_name and not all(c.isalnum() or c in "-_." for c in folder_name):
        raise ValueError(msg)
    base_dir = Path.home() / ".cache" / "twat_cache"
        cache_dir.mkdir(parents=True, exist_ok=True)
        raise OSError(msg) from e
def make_key(
    def default_serializer(obj: Any) -> Any:
        if isinstance(obj, str | int | float | bool | type(None)):
        return str(obj)
    converted_args = tuple(ser(arg) for arg in args)
    converted_kwargs = {k: ser(v) for k, v in sorted(kwargs.items())}
    return json.dumps((converted_args, converted_kwargs), sort_keys=True)

================
File: tests/__init__.py
================


================
File: tests/test_backend_selector.py
================
class TestDataSizeEstimation:
    def test_small_data(self) -> None:
        assert estimate_data_size(small_data) == DataSize.SMALL
        small_list = list(range(100))
        assert estimate_data_size(small_list) == DataSize.SMALL
        small_dict = {i: i * 2 for i in range(50)}
        assert estimate_data_size(small_dict) == DataSize.SMALL
    def test_medium_data(self) -> None:
        assert estimate_data_size(medium_string) == DataSize.MEDIUM
        medium_list = list(range(5000))
        if estimate_data_size(medium_list) != DataSize.MEDIUM:
            medium_list = list(range(50000))
            assert estimate_data_size(medium_list) == DataSize.MEDIUM
    def test_large_data(self) -> None:
        assert estimate_data_size(large_string) == DataSize.LARGE
    def test_numpy_array_size(self) -> None:
        small_array = np.ones((10, 10))
        assert estimate_data_size(small_array) == DataSize.SMALL
        medium_array = np.ones((500, 500))
        size = estimate_data_size(medium_array)
    def test_pandas_dataframe_size(self) -> None:
        small_df = pd.DataFrame({"a": range(100), "b": range(100)})
        assert estimate_data_size(small_df) == DataSize.SMALL
        medium_df = pd.DataFrame(np.random.rand(1000, 10))
        size = estimate_data_size(medium_df)
class TestTypeBasedBackendSelection:
    def test_primitive_types(self) -> None:
        assert get_type_based_backend(int) in ("cachetools", "functools")
        assert get_type_based_backend(str) in ("cachetools", "functools")
        assert get_type_based_backend(bool) in ("cachetools", "functools")
        assert get_type_based_backend(float) in ("cachetools", "functools")
    def test_container_types(self) -> None:
        assert get_type_based_backend(list) in ("cachetools", "diskcache")
        assert get_type_based_backend(dict) in ("cachetools", "diskcache")
        assert get_type_based_backend(tuple) in ("cachetools", "diskcache")
        assert get_type_based_backend(set) in ("cachetools", "diskcache")
    def test_numpy_types(self) -> None:
        assert get_type_based_backend(np.ndarray) in ("joblib", "diskcache")
    def test_pandas_types(self) -> None:
        assert get_type_based_backend(pd.DataFrame) in ("joblib", "diskcache")
        assert get_type_based_backend(pd.Series) in ("joblib", "diskcache")
    def test_custom_types(self) -> None:
        class CustomClass:
        assert get_type_based_backend(CustomClass) == "diskcache"
class TestBackendAvailability:
    def test_is_backend_available(self) -> None:
        assert is_backend_available("diskcache") is True
        assert is_backend_available("cachetools") is True
        assert is_backend_available("klepto") is True
        assert is_backend_available("joblib") is True
        assert is_backend_available("nonexistent_backend") is False
    def test_get_available_backend(self) -> None:
        assert get_available_backend("diskcache") == "diskcache"
        assert get_available_backend("nonexistent_backend") in get_engine_manager().get_available_engines()
class TestSelectBackendForData:
    def test_select_with_data_instance(self) -> None:
        assert select_backend_for_data(data=42) in ("cachetools", "functools")
        assert select_backend_for_data(data=medium_list) in ("diskcache", "klepto")
        np_array = np.ones((100, 100))
        assert select_backend_for_data(data=np_array) in ("joblib", "diskcache")
        df = pd.DataFrame({"a": range(100), "b": range(100)})
        assert select_backend_for_data(data=df) in ("joblib", "diskcache")
    def test_select_with_explicit_parameters(self) -> None:
        backend = select_backend_for_data(
    def test_select_with_config(self) -> None:
        config = create_cache_config(preferred_engine="diskcache")
        assert select_backend_for_data(config=config) == "diskcache"
        config = create_cache_config(cache_type="memory")
        assert select_backend_for_data(config=config) in ("cachetools", "functools")
        config = create_cache_config(use_sql=True)
        assert select_backend_for_data(config=config) in ("diskcache", "klepto")
class TestTypeSpecificConfigurations:
    def test_configure_for_type(self) -> None:
        int_config = configure_for_type(int)
        list_config = configure_for_type(list)
        np_config = configure_for_type(np.ndarray)
    def test_configure_for_numpy(self) -> None:
        config = configure_for_numpy()
    def test_configure_for_pandas(self) -> None:
        config = configure_for_pandas()
    def test_configure_for_images(self) -> None:
        config = configure_for_images()
    def test_configure_for_json(self) -> None:
        config = configure_for_json()
class TestHybridCacheConfig:
    def test_hybrid_cache_config(self) -> None:
        config = hybrid_cache_config()
        assert isinstance(config, dict)
        custom_config = hybrid_cache_config(
class TestFunctionAnalysis:
    def test_analyze_function_return_type(self) -> None:
        def returns_int() -> int:
        def returns_list() -> list[int]:
        def returns_dict() -> dict[str, Any]:
        def returns_numpy() -> np.ndarray:
            return np.ones((10, 10))
        assert analyze_function_return_type(returns_int) == int
        assert analyze_function_return_type(returns_list) == list[int]
        assert analyze_function_return_type(returns_dict) == dict[str, Any]
        assert analyze_function_return_type(returns_numpy) == np.ndarray
        def no_annotation():
        assert analyze_function_return_type(no_annotation) is None
    def test_smart_cache_config(self) -> None:
        int_config = smart_cache_config(returns_int)
        numpy_config = smart_cache_config(returns_numpy)
        custom_config = smart_cache_config(returns_int, maxsize=100, ttl=60)
class TestResultTypeDetection:
    def test_detect_result_type(self) -> None:
        assert detect_result_type(42) == "int"
        assert detect_result_type(3.14) == "float"
        assert detect_result_type("hello") == "str"
        assert detect_result_type(True) == "bool"
        assert detect_result_type([1, 2, 3]) == "list"
        assert detect_result_type({"a": 1, "b": 2}) == "dict"
        assert detect_result_type((1, 2, 3)) == "tuple"
        assert detect_result_type({1, 2, 3}) == "set"
        assert detect_result_type(np.ones(10)) == "numpy.ndarray"
        assert detect_result_type(pd.DataFrame({"a": [1, 2, 3]})) == "pandas.DataFrame"
        assert detect_result_type(pd.Series([1, 2, 3])) == "pandas.Series"
        assert detect_result_type(CustomClass()) == "CustomClass"

================
File: tests/test_cache.py
================
def test_config_validation() -> None:
    config = create_cache_config(maxsize=CACHE_SIZE)
    with pytest.raises(ValueError):
        create_cache_config(maxsize=-1)
        create_cache_config(ttl=-1)
def test_cache_path() -> None:
    path = get_cache_path()
    assert path.exists()
    assert path.is_dir()
    with tempfile.TemporaryDirectory() as temp_dir:
        temp_path = Path(temp_dir)
        path = get_cache_path(folder_name="test", cache_dir=str(temp_path))
def test_cache_clear() -> None:
        test_file.touch()
        clear_cache(folder_name="test", cache_dir=str(temp_path))
        assert not test_file.exists()
def test_cache_stats() -> None:
    clear_cache()
    @mcache()
    def process_list(lst: list[int]) -> int:
        return sum(lst)
    result1 = process_list(TEST_LIST)
    result2 = process_list(TEST_LIST)
    stats = get_stats()
    assert isinstance(stats, dict)
def test_cache_security() -> None:
        cache_path = get_cache_path()
        mode = cache_path.stat().st_mode & 0o777
def test_cache_decorators() -> None:
    def mem_func(x: int) -> int:
    assert mem_func(SQUARE_INPUT) == SQUARE_RESULT
    assert mem_func(SQUARE_INPUT) == SQUARE_RESULT  # Should use cache
    @bcache()
    def disk_func(x: int) -> int:
    assert disk_func(SQUARE_INPUT) == SQUARE_RESULT
    assert disk_func(SQUARE_INPUT) == SQUARE_RESULT  # Should use cache
    @fcache()
    def file_func(x: int) -> int:
    assert file_func(SQUARE_INPUT) == SQUARE_RESULT
    assert file_func(SQUARE_INPUT) == SQUARE_RESULT  # Should use cache
def test_cache_basic():
    def square(x: int) -> int:
    result1 = square(TEST_VALUE)
    result2 = square(TEST_VALUE)
def test_cache_size():
    @mcache(maxsize=SMALL_CACHE_SIZE)
    for i in range(SMALL_CACHE_SIZE * 2):
        square(i)
def test_cache_clear():
    square.cache_clear()
def test_cache_stats():
    square(TEST_VALUE)
    assert square.cache_info().misses == 1
    assert square.cache_info().hits == 0
    assert square.cache_info().hits == 1
def test_cache_permissions():
        @bcache(folder_name=TEST_CACHE_DIR)
        def func(x: int) -> int:
        func(TEST_VALUE)
        mode = os.stat(cache_path).st_mode & 0o777
def test_cache_types():
    assert mem_func(TEST_VALUE) == TEST_RESULT
    assert mem_func(TEST_VALUE) == TEST_RESULT  # Should use cache
    assert disk_func(TEST_VALUE) == TEST_RESULT
    assert disk_func(TEST_VALUE) == TEST_RESULT  # Should use cache
    @fcache(folder_name=TEST_CACHE_DIR)
    assert file_func(TEST_VALUE) == TEST_RESULT
    assert file_func(TEST_VALUE) == TEST_RESULT  # Should use cache
def test_list_processing() -> None:
    @ucache()
    def process_list(data: list[int]) -> int:
        return sum(data)
    result1 = process_list(test_list)
    result2 = process_list(test_list)
def test_different_backends() -> None:
        create_cache_config(maxsize=100),  # Memory cache
        create_cache_config(folder_name="disk_test", use_sql=True),  # SQL cache
        create_cache_config(folder_name="disk_test", use_sql=False),  # Disk cache
        @ucache(
        def cached_function(x: int) -> int:
        result = cached_function(5)
        results.append(result)
        assert all(r == results[0] for r in results)
def test_cache_with_complex_types() -> None:
    @ucache(maxsize=100)
    def process_list(items: list[int]) -> int:
        return sum(items)
def test_cache_exceptions() -> None:
    def failing_function(x: int) -> None:
        raise ValueError(msg)
        failing_function(5)
def test_kwargs_handling() -> None:
    def kwarg_function(x: int, multiplier: int = 1) -> int:
    assert kwarg_function(TEST_INPUT) == TEST_INPUT
    assert kwarg_function(TEST_INPUT, multiplier=1) == TEST_INPUT
    assert kwarg_function(TEST_INPUT, multiplier=2) == TEST_INPUT * 2
def test_ttl_caching(tmp_path: Path) -> None:
    @ucache(folder_name=str(tmp_path), ttl=TTL_DURATION)
    assert cached_function(TEST_INPUT) == TEST_RESULT
    time.sleep(TTL_DURATION + 0.1)
def test_security_features(tmp_path: Path) -> None:
    @ucache(folder_name=str(tmp_path), secure=True)
    cached_function(TEST_INPUT)
    assert cache_path.exists()
async def test_async_caching() -> None:
    @ucache(use_async=True)
    async def async_function(x: int) -> int:
        await asyncio.sleep(0.1)  # Simulate async work
    result1 = await async_function(TEST_INPUT)
    result2 = await async_function(TEST_INPUT)
def test_specific_decorators() -> None:
    @mcache(maxsize=100)
    assert mem_func(5) == 25
    assert mem_func(5) == 25  # Should use cache
    @bcache(folder_name="test_disk")
    assert disk_func(5) == 25
    assert disk_func(5) == 25  # Should use cache
    @fcache(folder_name="test_file")
    assert file_func(5) == 25
    assert file_func(5) == 25  # Should use cache
def test_basic_memory_cache():
    square.clear()
    result3 = square(TEST_VALUE)

================
File: tests/test_config.py
================
def test_cache_config_defaults():
    config = CacheConfig()
def test_cache_config_validation():
    config = create_cache_config(
    with pytest.raises(ValidationError):
        create_cache_config(maxsize=-1)
        create_cache_config(ttl=-1)
        create_cache_config(compression_level=MAX_COMPRESSION + 1)
def test_cache_config_permissions():
    config = create_cache_config(secure=True)
    config = create_cache_config(secure=False)
def test_cache_config_size_limits():
    config = create_cache_config(maxsize=SMALL_CACHE_SIZE)
    config = create_cache_config(maxsize=DEFAULT_CACHE_SIZE)
    config = create_cache_config(maxsize=None)
def test_cache_config_ttl():
    config = create_cache_config(ttl=SHORT_TTL)
    config = create_cache_config(ttl=DEFAULT_TTL)
    config = create_cache_config(ttl=None)
def test_cache_config_compression():
    config = create_cache_config(compression_level=MIN_COMPRESSION)
    config = create_cache_config(compression_level=MAX_COMPRESSION)
    config = create_cache_config(compression_level=DEFAULT_COMPRESSION)

================
File: tests/test_constants.py
================


================
File: tests/test_context_simple.py
================
def test_cache_context_init():
    config = CacheConfig(maxsize=100)
    context = CacheContext(config=config, engine_name="memory")
@patch("twat_cache.context.get_engine_manager")
def test_cache_context_enter_exit(mock_get_engine_manager):
    mock_manager = MagicMock()
    mock_engine_cls = MagicMock()
    mock_engine = MagicMock()
    with CacheContext(config=config) as engine:
    mock_engine.cleanup.assert_called_once()
def test_cache_context_error_handling(mock_get_engine_manager):
    mock_engine.cleanup.side_effect = Exception("Test exception")
    pytest.main(["-v", __file__])

================
File: tests/test_context.py
================
def test_engine_context():
    with tempfile.TemporaryDirectory() as temp_dir:
        with engine_context(folder_name=temp_dir) as engine:
            assert isinstance(engine, BaseCacheEngine)
            def add(x, y):
            result = add(2, 3)
def test_engine_context_with_error():
                raise ValueError(msg)
def test_engine_context_with_invalid_engine():
    with pytest.raises(EngineError):
        with engine_context(engine_name="non_existent_engine"):
def test_cache_context_class():
        context = CacheContext(folder_name=temp_dir)
            def multiply(x, y):
            result = multiply(2, 3)
def test_get_or_create_engine():
        engine = get_or_create_engine(folder_name=temp_dir)
            def divide(x, y):
            result = divide(6, 3)
            engine.cleanup()
def test_nested_contexts():
    with tempfile.TemporaryDirectory() as temp_dir1:
        with tempfile.TemporaryDirectory() as temp_dir2:
            with engine_context(folder_name=temp_dir1) as engine1:
                with engine_context(folder_name=temp_dir2) as engine2:
                    assert add(2, 3) == 5
                    assert multiply(2, 3) == 6
def test_context_with_config():
    config = create_cache_config(
    with engine_context(config=config) as engine:

================
File: tests/test_decorators.py
================
HAS_AIOCACHE = bool(importlib.util.find_spec("aiocache"))
HAS_CACHEBOX = bool(importlib.util.find_spec("cachebox"))
HAS_CACHETOOLS = bool(importlib.util.find_spec("cachetools"))
HAS_DISKCACHE = bool(importlib.util.find_spec("diskcache"))
HAS_JOBLIB = bool(importlib.util.find_spec("joblib"))
HAS_KLEPTO = bool(importlib.util.find_spec("klepto"))
def test_basic_memory_cache():
    @bcache()
    def square(x: int) -> int:
    result1 = square(TEST_VALUE)
    result2 = square(TEST_VALUE)
@pytest.mark.skipif(not HAS_CACHEBOX, reason="cachebox not available")
def test_cachebox_memory():
    @bcache(engine="cachebox")
@pytest.mark.skipif(not HAS_CACHETOOLS, reason="cachetools not available")
def test_cachetools_memory():
    @bcache(engine="cachetools")
@pytest.mark.skipif(not HAS_DISKCACHE, reason="diskcache not available")
def test_diskcache_basic():
    @bcache(engine="diskcache", folder_name=TEST_FOLDER)
@pytest.mark.skipif(not HAS_KLEPTO, reason="klepto not available")
def test_klepto_sql():
    @bcache(engine="klepto", folder_name=TEST_FOLDER)
@pytest.mark.skipif(not HAS_JOBLIB, reason="joblib not available")
def test_joblib_file():
    @bcache(engine="joblib", folder_name=TEST_FOLDER)
def test_klepto_file():
@pytest.mark.skipif(not HAS_AIOCACHE, reason="aiocache not available")
def test_aiocache_memory():
    @ucache(engine="aiocache")
    async def square(x: int) -> int:
def test_memory_cache_cachebox() -> None:
        pytest.skip("cachebox not available")
    @mcache(maxsize=CACHE_SIZE)
    def expensive_func(x: int) -> int:
    expensive_func(1)
    expensive_func(2)
    expensive_func(3)
    expensive_func(1)  # Should recompute
def test_memory_cache_cachetools() -> None:
        pytest.skip("cachetools not available")
    assert expensive_func(SQUARE_INPUT) == SQUARE_RESULT
def test_basic_disk_cache() -> None:
        pytest.skip("diskcache not available")
    @bcache(maxsize=CACHE_SIZE)
def test_basic_disk_cache_sql() -> None:
        pytest.skip("klepto not available")
    @bcache(maxsize=CACHE_SIZE, use_sql=True)
def test_file_cache_joblib() -> None:
        pytest.skip("joblib not available")
    @fcache(maxsize=CACHE_SIZE)
def test_file_cache_klepto() -> None:
def test_universal_cache() -> None:
    @ucache(maxsize=CACHE_SIZE)
async def test_async_cache() -> None:
        pytest.skip("aiocache not available")
    @acache(maxsize=CACHE_SIZE)
    async def expensive_func(x: int) -> int:
        await asyncio.sleep(0.1)
    result1 = await expensive_func(SQUARE_INPUT)
    result2 = await expensive_func(SQUARE_INPUT)
def test_key_generation() -> None:
    @mcache()
    def test_func(x: Any) -> str:
        return str(x)
    test_func(TEST_KEY)
    test_func(TEST_BOOL)
    test_func(TEST_INT)
def test_ttl_cache() -> None:
    @mcache(ttl=CACHE_TTL)
    time.sleep(CACHE_TTL + 0.1)

================
File: tests/test_engines.py
================
def engine_manager() -> EngineManager:
    return EngineManager()
def base_config() -> CacheConfig:
    return create_cache_config(maxsize=CACHE_SIZE)
def base_engine(base_config: CacheConfig) -> CacheEngine[Any, Any]:
    return FunctoolsCacheEngine(base_config)
def create_test_engine() -> CacheEngine:
    config = CacheConfig(
    return FunctoolsCacheEngine(config)
def test_engine_initialization() -> None:
    engine = create_test_engine()
    assert isinstance(engine, CacheEngine)
def test_key_generation() -> None:
    assert isinstance(engine._make_key(TEST_KEY), str)
    assert isinstance(engine._make_key(TEST_INT), str)
    assert isinstance(engine._make_key(TEST_BOOL), str)
    assert isinstance(engine._make_key(TEST_LIST), str)
    assert isinstance(engine._make_key((1, 2, 3)), str)
    assert isinstance(engine._make_key({"a": 1}), str)
def test_cache_operations() -> None:
    create_test_engine()
    def test_function(x: int) -> int:
    result = test_function(SQUARE_INPUT)
def test_cache_eviction() -> None:
    for i in range(CACHE_SIZE + 1):
        engine.set(str(i), i)
    assert len(engine._cache) <= CACHE_SIZE
def test_cache_ttl() -> None:
    time.sleep(CACHE_TTL + 0.1)
def test_cache_clear() -> None:
    for i in range(CACHE_SIZE):
    engine.clear()
    assert len(engine._cache) == 0
def test_cache_stats() -> None:

================
File: tests/test_exceptions_simple.py
================
def test_exception_hierarchy():
    assert issubclass(TwatCacheError, Exception)
    assert issubclass(ConfigurationError, TwatCacheError)
    assert issubclass(EngineError, TwatCacheError)
    assert issubclass(CacheOperationError, TwatCacheError)
    assert issubclass(ResourceError, TwatCacheError)
    assert issubclass(EngineNotAvailableError, EngineError)
    assert issubclass(CacheKeyError, CacheOperationError)
    assert issubclass(CacheValueError, CacheOperationError)
    assert issubclass(SerializationError, CacheOperationError)
    assert issubclass(ConcurrencyError, ResourceError)
    assert issubclass(PathError, ResourceError)
def test_exception_messages():
    base_exc = TwatCacheError("Base error message")
    assert str(base_exc) == "Base error message"
    config_exc = ConfigurationError("Configuration error message")
    assert str(config_exc) == "Configuration error message"
    engine_exc = EngineError("Engine error message")
    assert str(engine_exc) == "Engine error message"
    operation_exc = CacheOperationError("Operation error message")
    assert str(operation_exc) == "Operation error message"
    resource_exc = ResourceError("Resource error message")
    assert str(resource_exc) == "Resource error message"
    engine_not_available_exc = EngineNotAvailableError("Engine not available message")
    assert str(engine_not_available_exc) == "Engine not available message"
    key_exc = CacheKeyError("Key error message")
    assert str(key_exc) == "Key error message"
    value_exc = CacheValueError("Value error message")
    assert str(value_exc) == "Value error message"
    serialization_exc = SerializationError("Serialization error message")
    assert str(serialization_exc) == "Serialization error message"
    concurrency_exc = ConcurrencyError("Concurrency error message")
    assert str(concurrency_exc) == "Concurrency error message"
    path_exc = PathError("Path error message")
    assert str(path_exc) == "Path error message"
    pytest.main(["-v", __file__])

================
File: tests/test_exceptions.py
================
def test_base_exception():
    exc = TwatCacheError("Test error message")
    assert str(exc) == "Test error message"
    exc = TwatCacheError("Test error message", "arg1", "arg2")
    assert str(exc) == "('Test error message', 'arg1', 'arg2')"
def test_exception_hierarchy():
    assert issubclass(ConfigurationError, TwatCacheError)
    assert issubclass(EngineError, TwatCacheError)
    assert issubclass(EngineNotAvailableError, EngineError)
    assert issubclass(CacheOperationError, TwatCacheError)
    assert issubclass(CacheKeyError, CacheOperationError)
    assert issubclass(CacheValueError, CacheOperationError)
    assert issubclass(SerializationError, TwatCacheError)
    assert issubclass(ResourceError, TwatCacheError)
    assert issubclass(ConcurrencyError, TwatCacheError)
    assert issubclass(PathError, TwatCacheError)
def test_engine_not_available_error():
    exc = EngineNotAvailableError("test_engine")
    assert str(exc) == "Cache engine 'test_engine' is not available"
    exc = EngineNotAvailableError("test_engine", "Not installed")
    assert str(exc) == "Cache engine 'test_engine' is not available: Not installed"
def test_ensure_dir_exists():
    with tempfile.TemporaryDirectory() as temp_dir:
        test_dir = Path(temp_dir) / "test_dir"
        ensure_dir_exists(test_dir)
        assert test_dir.exists()
        assert test_dir.is_dir()
            assert (test_dir.stat().st_mode & 0o777) == 0o700
        test_dir2 = Path(temp_dir) / "test_dir2"
        ensure_dir_exists(test_dir2, mode=0o755)
        assert test_dir2.exists()
            assert (test_dir2.stat().st_mode & 0o777) == 0o755
def test_ensure_dir_exists_error():
    with tempfile.NamedTemporaryFile() as temp_file:
        with pytest.raises(PathError) as excinfo:
            ensure_dir_exists(Path(temp_file.name))
        assert "Failed to create cache directory" in str(excinfo.value)
def test_safe_key_serializer():
    assert safe_key_serializer("test") == "test"
    assert safe_key_serializer(123) == "123"
    assert safe_key_serializer(True) == "True"
    assert safe_key_serializer(None) == "None"
    assert safe_key_serializer([1, 2, 3]) == '["1", "2", "3"]'
    assert safe_key_serializer({"a": 1, "b": 2}) == '{"a": "1", "b": "2"}'
    assert safe_key_serializer([1, {"a": 2}]) == '["1", {"a": "2"}]'
def test_safe_key_serializer_error():
    class UnserializableObject:
        def __repr__(self):
            raise RuntimeError(msg)
    with pytest.raises(CacheKeyError) as excinfo:
        safe_key_serializer(UnserializableObject())
    assert "Failed to serialize cache key" in str(excinfo.value)
def test_safe_value_serializer():
    assert safe_value_serializer("test") == '"test"'
    assert safe_value_serializer(123) == "123"
    assert safe_value_serializer([1, 2, 3]) == "[1, 2, 3]"
    assert safe_value_serializer({"a": 1, "b": 2}) == '{"a": 1, "b": 2}'
def test_safe_value_serializer_error():
    class CustomObject:
    result = safe_value_serializer(CustomObject())
    assert result.startswith("<")
    class BadObject:
    with pytest.raises(CacheValueError) as excinfo:
        safe_value_serializer(BadObject())
    assert "Failed to serialize cache value" in str(excinfo.value)
def test_safe_temp_file():
    path, file_obj = safe_temp_file()
        assert path.exists()
        assert path.is_file()
        file_obj.write(b"test data")
        file_obj.flush()
            assert (path.stat().st_mode & 0o777) == 0o600
        file_obj.close()
        if path.exists():
            path.unlink()
def test_catching_exceptions():
        raise ConfigurationError(msg)
        assert isinstance(e, ConfigurationError)
        assert str(e) == "Test config error"
        raise EngineNotAvailableError(msg, "Not installed")
        assert isinstance(e, EngineNotAvailableError)
        raise CacheKeyError(msg)
        assert isinstance(e, CacheKeyError)
        assert str(e) == "Invalid key"

================
File: tests/test_fallback.py
================
def test_backend_availability() -> None:
    available = _get_available_backends()
    assert isinstance(available, dict)
def test_backend_selection_preferred() -> None:
    backend = _select_best_backend(preferred="functools")
    backend = _select_best_backend(preferred="nonexistent")
def test_backend_selection_async() -> None:
    with patch.dict("sys.modules", {"aiocache": object()}):
        backend = _select_best_backend(is_async=True)
    with patch.dict("sys.modules", {"aiocache": None}):
def test_backend_selection_disk() -> None:
    with patch.dict("sys.modules", {"diskcache": object()}):
        backend = _select_best_backend(needs_disk=True)
    with patch.dict("sys.modules", {"diskcache": None, "joblib": object()}):
    with patch.dict(
        "sys.modules", {"diskcache": None, "joblib": None, "klepto": object()}
    with patch.dict("sys.modules", {"diskcache": None, "joblib": None, "klepto": None}):
def test_memory_cache_fallback() -> None:
    @mcache(maxsize=CACHE_SIZE)
    def test_func(x: int) -> int:
        result = test_func(SQUARE_INPUT)
def test_disk_cache_fallback() -> None:
    @bcache(maxsize=CACHE_SIZE)
async def test_async_cache_fallback() -> None:
    @acache(maxsize=CACHE_SIZE)
    async def test_func(x: int) -> int:
        await asyncio.sleep(0.1)
        result = await test_func(SQUARE_INPUT)
def test_universal_cache_fallback() -> None:
    @ucache(maxsize=CACHE_SIZE)
def test_backend_selection_priority() -> None:
            "cachebox": object(),
            "cachetools": object(),
        backend = _select_best_backend()
            "diskcache": object(),
            "joblib": object(),
            "klepto": object(),
def test_backend_selection_requirements() -> None:
            "aiocache": object(),
        backend = _select_best_backend(is_async=True, needs_disk=True)

================
File: tests/test_redis_cache.py
================
class TestRedisCacheEngine:
    def mock_redis(self):
        with patch("redis.Redis") as mock_redis:
            mock_instance = MagicMock()
    def fake_redis(self):
        return fakeredis.FakeStrictRedis()
    def redis_engine(self, mock_redis):
        config = create_cache_config(
        engine = RedisCacheEngine(config)
    def real_redis_engine(self, fake_redis):
    def test_is_available(self):
        with patch("twat_cache.engines.redis.is_package_available") as mock_check:
            assert RedisCacheEngine.is_available() is True
            assert RedisCacheEngine.is_available() is False
    def test_init_with_unavailable_redis(self):
        with patch("twat_cache.engines.redis.RedisCacheEngine.is_available") as mock_check:
            config = create_cache_config()
            with pytest.raises(EngineNotAvailableError):
                RedisCacheEngine(config)
    def test_init_with_connection_error(self):
            mock_redis.return_value.ping.side_effect = redis.exceptions.ConnectionError("Connection refused")
            with pytest.raises(ConfigurationError):
    def test_validate_config(self, redis_engine):
        redis_engine.validate_config()  # Should not raise
        with patch.object(redis_engine._config, "get_redis_port") as mock_get_port:
                redis_engine.validate_config()
    def test_get_full_key(self, redis_engine):
        full_key = redis_engine._get_full_key(key)
        assert full_key.startswith("test_redis:")
        assert isinstance(full_key, str)
    def test_get_cached_value(self, redis_engine, mock_redis):
        mock_redis.get.return_value = pickle.dumps("test_value")
        value = redis_engine._get_cached_value("test_key")
        mock_redis.get.assert_called_once_with("test_redis:test_key")
        mock_redis.get.reset_mock()
        value = redis_engine._get_cached_value("missing_key")
        mock_redis.get.assert_called_once()
        value = redis_engine._get_cached_value("error_key")
    def test_set_cached_value(self, redis_engine, mock_redis):
        redis_engine._set_cached_value("test_key", "test_value")
        mock_redis.set.assert_called_once()
        mock_redis.set.reset_mock()
        mock_redis.setex.assert_called_once()
        class UnpickleableObject:
            def __reduce__(self):
                raise TypeError(msg)
        with pytest.raises(Exception):
            redis_engine._set_cached_value("error_key", UnpickleableObject())
    def test_clear(self, redis_engine, mock_redis):
        redis_engine.clear()
        mock_redis.keys.assert_called_once_with("test_redis:*")
        mock_redis.delete.assert_called_once()
        mock_redis.keys.reset_mock()
        mock_redis.delete.reset_mock()
        mock_redis.keys.assert_called_once()
        mock_redis.delete.assert_not_called()
    def test_cache_decorator(self, redis_engine, mock_redis):
        def test_func(x):
        result = test_func(5)
        mock_redis.get.return_value = pickle.dumps(10)
    def test_stats(self, redis_engine, mock_redis):
    def test_real_redis_operations(self, real_redis_engine):
        real_redis_engine._set_cached_value("test_key", "test_value")
        value = real_redis_engine._get_cached_value("test_key")
        value = real_redis_engine._get_cached_value("missing_key")
        real_redis_engine.clear()
        real_redis_engine._set_cached_value("complex_key", complex_data)
        value = real_redis_engine._get_cached_value("complex_key")
    def test_compression(self, real_redis_engine):
        real_redis_engine._set_cached_value("compressed_key", "test_value" * 100)
        value = real_redis_engine._get_cached_value("compressed_key")

================
File: tests/test_twat_cache.py
================
def base_config() -> CacheConfig:
    return create_cache_config(maxsize=100)
def temp_path(tmp_path: Path) -> Path:
def lru_engine(base_config: CacheConfig) -> FunctoolsCacheEngine:
    return FunctoolsCacheEngine(base_config)
def test_lru_cache_get_set(lru_engine: FunctoolsCacheEngine) -> None:
    assert lru_engine._get_cached_value(key) is None
    lru_engine._set_cached_value(key, value)
    assert lru_engine._get_cached_value(key) == value
def test_lru_cache_clear(lru_engine: FunctoolsCacheEngine) -> None:
    lru_engine.clear()
def test_lru_cache_maxsize(lru_engine: FunctoolsCacheEngine) -> None:
    for i in range(150):  # More than maxsize
        lru_engine._set_cached_value(f"key_{i}", f"value_{i}")
    assert lru_engine._get_cached_value("key_0") is None
    assert lru_engine._get_cached_value("key_149") is not None
def disk_engine(base_config: CacheConfig, temp_path: Path) -> DiskCacheEngine:
    config = create_cache_config(maxsize=100, cache_dir=str(temp_path))
    return DiskCacheEngine(config)
def test_disk_cache_persistence(disk_engine: DiskCacheEngine) -> None:
    disk_engine._set_cached_value(key, value)
    assert disk_engine._get_cached_value(key) == value
    new_engine = DiskCacheEngine(config)
    assert new_engine._get_cached_value(key) == value
def joblib_engine(base_config: CacheConfig, temp_path: Path) -> JoblibEngine:
    return JoblibEngine(config)
@pytest.mark.skipif(not HAS_NUMPY, reason="numpy not available")
def test_joblib_numpy_array(joblib_engine: JoblibEngine) -> None:
    value = np.array([[1, 2], [3, 4]])
    joblib_engine._set_cached_value(key, value)
    cached = joblib_engine._get_cached_value(key)
    assert isinstance(cached, np.ndarray)
    assert np.array_equal(cached, value)
def klepto_engine(base_config: CacheConfig, temp_path: Path) -> KleptoEngine:
    return KleptoEngine(config)
def test_klepto_persistence(klepto_engine: KleptoEngine) -> None:
    klepto_engine._set_cached_value(key, value)
    assert klepto_engine._get_cached_value(key) == value
async def async_engine(base_config: CacheConfig) -> AioCacheEngine:
    return AioCacheEngine(base_config)
async def test_async_cache_operations(async_engine: AioCacheEngine) -> None:
    assert await async_engine._get_cached_value(key) is None
    await async_engine._set_cached_value(key, value)
    assert await async_engine._get_cached_value(key) == value
    await async_engine.clear()
def cachebox_engine(base_config: CacheConfig) -> CacheBoxEngine:
    return CacheBoxEngine(base_config)
def test_cachebox_operations(cachebox_engine: CacheBoxEngine) -> None:
    assert cachebox_engine._get_cached_value(key) is None
    cachebox_engine._set_cached_value(key, value)
    assert cachebox_engine._get_cached_value(key) == value
    cachebox_engine.clear()
def cachetools_engine(base_config: CacheConfig) -> CacheToolsEngine:
    return CacheToolsEngine(base_config)
def test_cachetools_operations(cachetools_engine: CacheToolsEngine) -> None:
    assert cachetools_engine._get_cached_value(key) is None
    cachetools_engine._set_cached_value(key, value)
    assert cachetools_engine._get_cached_value(key) == value
    cachetools_engine.clear()
def functools_engine(base_config: CacheConfig) -> FunctoolsCacheEngine:
def test_functools_cache_get_set(functools_engine: FunctoolsCacheEngine) -> None:
    assert functools_engine._get_cached_value(key) is None
    functools_engine._set_cached_value(key, value)
    assert functools_engine._get_cached_value(key) == value
def test_functools_cache_clear(functools_engine: FunctoolsCacheEngine) -> None:
    functools_engine.clear()
def test_functools_cache_maxsize(functools_engine: FunctoolsCacheEngine) -> None:
        functools_engine._set_cached_value(f"key_{i}", f"value_{i}")
    assert functools_engine._get_cached_value("key_0") is None
    assert functools_engine._get_cached_value("key_149") is not None

================
File: .gitignore
================
.mypy_cache
*_autogen/
.DS_Store
__version__.py
__pycache__/
_Chutzpah*
_deps
_NCrunch_*
_pkginfo.txt
_Pvt_Extensions
_ReSharper*/
_TeamCity*
_UpgradeReport_Files/
!?*.[Cc]ache/
!.axoCover/settings.json
!.vscode/extensions.json
!.vscode/launch.json
!.vscode/settings.json
!.vscode/tasks.json
!**/[Pp]ackages/build/
!Directory.Build.rsp
.*crunch*.local.xml
.axoCover/*
.builds
.cr/personal
.fake/
.history/
.ionide/
.localhistory/
.mfractor/
.ntvs_analysis.dat
.paket/paket.exe
.sass-cache/
.vs/
.vscode
.vscode/*
.vshistory/
[Aa][Rr][Mm]/
[Aa][Rr][Mm]64/
[Bb]in/
[Bb]uild[Ll]og.*
[Dd]ebug/
[Dd]ebugPS/
[Dd]ebugPublic/
[Ee]xpress/
[Ll]og/
[Ll]ogs/
[Oo]bj/
[Rr]elease/
[Rr]eleasePS/
[Rr]eleases/
[Tt]est[Rr]esult*/
[Ww][Ii][Nn]32/
*_h.h
*_i.c
*_p.c
*_wpftmp.csproj
*- [Bb]ackup ([0-9]).rdl
*- [Bb]ackup ([0-9][0-9]).rdl
*- [Bb]ackup.rdl
*.[Cc]ache
*.[Pp]ublish.xml
*.[Rr]e[Ss]harper
*.a
*.app
*.appx
*.appxbundle
*.appxupload
*.aps
*.azurePubxml
*.bim_*.settings
*.bim.layout
*.binlog
*.btm.cs
*.btp.cs
*.build.csdef
*.cab
*.cachefile
*.code-workspace
*.coverage
*.coveragexml
*.d
*.dbmdl
*.dbproj.schemaview
*.dll
*.dotCover
*.DotSettings.user
*.dsp
*.dsw
*.dylib
*.e2e
*.exe
*.gch
*.GhostDoc.xml
*.gpState
*.ilk
*.iobj
*.ipdb
*.jfm
*.jmconfig
*.la
*.lai
*.ldf
*.lib
*.lo
*.log
*.mdf
*.meta
*.mm.*
*.mod
*.msi
*.msix
*.msm
*.msp
*.ncb
*.ndf
*.nuget.props
*.nuget.targets
*.nupkg
*.nvuser
*.o
*.obj
*.odx.cs
*.opendb
*.opensdf
*.opt
*.out
*.pch
*.pdb
*.pfx
*.pgc
*.pgd
*.pidb
*.plg
*.psess
*.publishproj
*.publishsettings
*.pubxml
*.pyc
*.rdl.data
*.rptproj.bak
*.rptproj.rsuser
*.rsp
*.rsuser
*.sap
*.sbr
*.scc
*.sdf
*.sln.docstates
*.sln.iml
*.slo
*.smod
*.snupkg
*.so
*.suo
*.svclog
*.tlb
*.tlh
*.tli
*.tlog
*.tmp
*.tmp_proj
*.tss
*.user
*.userosscache
*.userprefs
*.vbp
*.vbw
*.VC.db
*.VC.VC.opendb
*.VisualState.xml
*.vsp
*.vspscc
*.vspx
*.vssscc
*.xsd.cs
**/[Pp]ackages/*
**/*.DesktopClient/GeneratedArtifacts
**/*.DesktopClient/ModelManifest.xml
**/*.HTMLClient/GeneratedArtifacts
**/*.Server/GeneratedArtifacts
**/*.Server/ModelManifest.xml
*~
~$*
$tf/
AppPackages/
artifacts/
ASALocalRun/
AutoTest.Net/
Backup*/
BenchmarkDotNet.Artifacts/
bld/
BundleArtifacts/
ClientBin/
cmake_install.cmake
CMakeCache.txt
CMakeFiles
CMakeLists.txt.user
CMakeScripts
CMakeUserPresets.json
compile_commands.json
coverage*.info
coverage*.json
coverage*.xml
csx/
CTestTestfile.cmake
dlldata.c
DocProject/buildhelp/
DocProject/Help/*.hhc
DocProject/Help/*.hhk
DocProject/Help/*.hhp
DocProject/Help/*.HxC
DocProject/Help/*.HxT
DocProject/Help/html
DocProject/Help/Html2
ecf/
FakesAssemblies/
FodyWeavers.xsd
Generated_Code/
Generated\ Files/
healthchecksdb
install_manifest.txt
ipch/
Makefile
MigrationBackup/
mono_crash.*
nCrunchTemp_*
node_modules/
nunit-*.xml
OpenCover/
orleans.codegen.cs
Package.StoreAssociation.xml
paket-files/
project.fragment.lock.json
project.lock.json
publish/
PublishScripts/
rcf/
ScaffoldingReadMe.txt
ServiceFabricBackup/
StyleCopReport.xml
Testing
TestResult.xml
UpgradeLog*.htm
UpgradeLog*.XML
x64/
x86/
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# Distribution / packaging
!dist/.gitkeep

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
.ruff_cache/

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# IDE
.idea/
.vscode/
*.swp
*.swo
*~

# OS
.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
ehthumbs.db
Thumbs.db

# Project specific
__version__.py
_private

================
File: .pre-commit-config.yaml
================
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.3.4
    hooks:
      - id: ruff
        args: [--fix]
      - id: ruff-format
        args: [--respect-gitignore]
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: check-yaml
      - id: check-toml
      - id: check-added-large-files
      - id: debug-statements
      - id: check-case-conflict
      - id: mixed-line-ending
        args: [--fix=lf]

================
File: CHANGELOG.md
================
---
this_file: CHANGELOG.md
---

# Changelog

All notable changes to the `twat_cache` package will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [v2.5.3] - 2025-03-04

### Added
- Enhanced hybrid caching capabilities
  - Improved automatic backend switching based on data characteristics
  - Added more intelligent data type detection
  - Optimized threshold determination for backend selection
- Expanded test coverage
  - Added more comprehensive tests for Redis engine
  - Improved test coverage for hybrid caching
  - Added tests for edge cases in backend selection
- Improved documentation
  - Updated README with more detailed examples
  - Added more comprehensive docstrings
  - Reorganized TODO list for better clarity and actionability

### Fixed
- Resolved several edge cases in Redis connection handling
- Fixed type annotation issues in context management
- Addressed potential race conditions in multi-threaded environments
- Improved error handling in serialization/deserialization processes

### Changed
- Refactored backend selection logic for better performance
- Improved context management API for more intuitive usage
- Updated dependencies to latest compatible versions
- Reorganized project structure for better maintainability

## [v2.3.0] - 2025-03-04

### Added
- Improved context management utilities
  - Enhanced error handling in context managers
  - Added support for explicit engine selection in context managers
  - Improved resource cleanup mechanisms
- Enhanced backend selection strategy
  - Refined data type detection for optimal backend selection
  - Added support for hybrid caching with automatic backend switching
  - Improved fallback mechanisms for unavailable backends
- Comprehensive documentation updates
  - Added detailed examples for all cache backends
  - Created documentation for context management
  - Updated installation and usage instructions

### Fixed
- Resolved remaining type compatibility issues across the codebase
- Fixed edge cases in Redis connection handling
- Addressed potential memory leaks in long-running cache instances
- Improved error handling in serialization/deserialization processes

### Changed
- Refactored internal cache key generation for better performance
- Standardized logging format across all cache engines
- Improved test coverage for all components
- Updated dependencies to latest compatible versions

## [v2.2.0] - 2024-09-01

### Added

- Comprehensive test suite for backend selection strategy
  - Tests for data size estimation
  - Tests for type-based backend selection
  - Tests for backend availability
  - Tests for various data types (primitive, containers, NumPy arrays, pandas DataFrames)
- Redis cache engine implementation
  - Full Redis cache engine with serialization, compression, and TTL support
  - Redis-specific configuration options
  - Proper connection handling and error management
  - Registration in the engine manager
- Documentation for cache context management
  - Detailed guide on using context managers
  - Examples for basic and advanced usage patterns
  - Best practices for resource management
  - Error handling recommendations

### Fixed

- Type compatibility issues in context.py with CacheConfig protocol
  - Updated protocol in type_defs.py to use validate_config instead of validate
  - Fixed type casting to avoid type errors
  - Improved type annotations for better static analysis
- Linter errors in engine implementations
  - Fixed import issues with optional dependencies
  - Addressed type compatibility warnings
  - Improved error handling in Redis engine

### Changed

- Expanded test coverage for cache engines
  - Added comprehensive tests for Redis cache engine
  - Tests for initialization, configuration validation, key generation
  - Tests for caching operations and error handling
- Refactored common patterns across engine implementations
  - Standardized engine initialization and validation procedures
  - Created unified interfaces for all cache backends
  - Improved code organization and maintainability

## [v2.1.1] - 2024-08-26

### Added

- New custom exceptions module with proper inheritance hierarchy
  - Base `TwatCacheError` exception class
  - Specialized exceptions for different error categories (configuration, engine, resource, etc.)
  - Improved error messages with context-specific information
- Context management for cache engines
  - `CacheContext` class that provides resource cleanup on exit
  - Safe resource management with proper error handling
  - Support for engine selection based on configuration
- Tests for exceptions and context management modules
  - Simple test suite for exception hierarchy and messages
  - Context management tests with mocking for proper isolation
- Implemented data type-based cache backend selection strategy
  - New `backend_selector` module for automatic backend selection
  - Helper functions for common data types (numpy, pandas, images, json)
  - Smart backend selection based on data characteristics
- Added hybrid caching capabilities
  - `hybrid_cache` decorator for size-based backend selection
  - `smart_cache` decorator for automatic backend selection
- Added example demonstrating the backend selection strategy
- Created CHANGELOG.md for tracking version changes
- Updated TODO.md to reflect completed items and reprioritize tasks

### Fixed

- Fixed error handling in various cache engines
- Improved cleanup logic for better resource management
- Made error messages more consistent across the codebase
- Installation issues with pip install -e . by replacing hatchling with setuptools
- Path handling in CacheContext

### Changed

- Standardized initialization patterns across engine implementations
- Improved shutdown procedures to properly release resources
- Removed setup.py in favor of pyproject.toml for modern Python packaging
- Refactored context manager implementation for better error handling 


## \[1.8.1] - 2025-02-18

### Added

* Added comprehensive type stubs for all cache engines
* Added detailed rationale section in README.md comparing different caching libraries
* Added extensive code examples for each caching strategy

### Changed

* Updated documentation with detailed type hints and protocols
* Improved error messages in cache engines
* Enhanced configuration validation

### Fixed

* Fixed type hints in async cache implementations
* Fixed decorator type hints for better IDE support
* Fixed linter errors in multiple files

### Issues

* Still missing type stubs for external dependencies
* Some linter warnings in test files need addressing
* Need to improve async test coverage

### Next Steps

1. Fix remaining linter issues:
   * Add missing type stubs for external dependencies
   * Address warnings in test files
   * Improve type safety in async code

2. Enhance test coverage:
   * Add more async test cases
   * Add stress tests for race conditions
   * Add integration tests for all backends

3. Improve documentation:
   * Add migration guide
   * Add performance comparison guide
   * Add troubleshooting section

## \[1.8.0] - 2025-02-17

### Added

* Completed core features:
  * Memory-based caching (mcache)
  * Disk-based caching (bcache)
  * File-based caching (fcache)
  * Async-capable caching (via ucache)
  * Universal caching (ucache)
* Implemented all cache engines:
  * FunctoolsCacheEngine (base memory cache)
  * CacheToolsEngine (flexible memory cache)
  * DiskCacheEngine (SQL-based disk cache)
  * JoblibEngine (file-based cache)
  * KleptoEngine (flexible cache)
  * AioCacheEngine (async cache)
* Added comprehensive features:
  * TTL support for all caches
  * Multiple eviction policies (LRU, LFU, FIFO, RR)
  * Secure file permissions for disk caches
  * Compression support for file caches
  * Race condition handling
  * Proper error handling and logging
* Added proper type hints and protocols
* Added comprehensive test suite
* Updated documentation with examples

### Changed

* Reorganized backend priorities for each decorator:
  * mcache: cachebox > cachetools > functools
  * bcache: diskcache > klepto SQL > mcache
  * fcache: joblib > klepto file > mcache
  * ucache: auto-selects best backend
* Enhanced test coverage with backend-specific tests
* Improved error handling and logging
* Updated configuration system to use Pydantic

### Fixed

* Fixed CacheConfig initialization issues
* Fixed cache stats tracking
* Fixed maxsize enforcement
* Fixed race conditions in multi-threaded scenarios
* Fixed file permissions for disk caches
* Fixed type hints and protocols

### Issues

* Missing type stubs for some dependencies:
  * aiocache
  * diskcache
  * joblib
  * klepto
* Some linter warnings in test files
* Need more async test coverage
* Need more stress tests for race conditions

### Next Steps

1. Type System:
   * Add missing type stubs for dependencies
   * Fix remaining linter warnings
   * Add more type safety

2. Testing:
   * Add more async tests
   * Add stress tests for race conditions
   * Add performance benchmarks
   * Add integration tests

3. Documentation:
   * Add API reference
   * Add performance guide
   * Add migration guide
   * Add troubleshooting guide

4. Performance:
   * Add caching statistics
   * Add cache warming
   * Add cache prefetching
   * Add cache compression options

5. Security:
   * Add encryption support
   * Add access control
   * Add audit logging
   * Add cache poisoning protection

## [1.7.9] - 2025-02-16

### Changed

* Updated TODO.md with clearer structure and priorities
* Reorganized Phase 2 tasks to focus on critical issues first
* Added more detailed subtasks for each component

### Fixed

* Fixed Pydantic field validators in CacheConfig
* Updated model\_config settings
* Improved error messages

### Issues

* Critical: CacheEngine import failing in test\_engines.py
* CacheConfig initialization still broken (TypeError: CacheConfig() takes no arguments)
* Cache stats tracking not working
* Cache maxsize enforcement not working
* Multiple linting issues identified:
  * Magic numbers in comparisons
  * Boolean argument type issues
  * Module naming conflicts with stdlib
  * Unused arguments in lambda functions
  * Complex function (clear\_cache)

### Next Steps

1. Fix Critical Engine Issues:
   * Fix CacheEngine import in base.py
   * Fix test\_engines.py import error
   * Fix CacheConfig initialization
   * Address linting issues

2. Fix Cache Engine Core:
   * Implement proper stats tracking
   * Fix maxsize enforcement
   * Add proper cache key generation
   * Add proper error handling

3. Improve Test Coverage:
   * Fix failing tests
   * Add engine-specific tests
   * Add performance benchmarks

4. Documentation:
   * Add configuration guide
   * Add backend selection guide
   * Add performance optimization guide

## [1.7.8] - 2025-02-16

### Changed

* Reorganized TODO.md with clearer structure and priorities
* Added Phase 3 for advanced features
* Updated test results analysis

### Fixed

* Fixed Pydantic field validators in CacheConfig
* Updated field validator decorators to use @classmethod
* Fixed model\_config settings

### Issues

* CacheConfig initialization still broken (TypeError: CacheConfig() takes no arguments)
* Cache stats tracking not working (hits/misses not being counted)
* Cache maxsize enforcement not working (cache growing beyond limit)
* Cache clearing behavior inconsistent
* Field validation in CacheConfig needs fixing

### Next Steps

1. Fix CacheConfig initialization:
   * Update Pydantic model to properly accept constructor arguments
   * Fix field validation and property access
   * Add proper error messages
   * Update model\_config for proper initialization

2. Fix cache engine issues:
   * Fix stats tracking in FunctoolsCacheEngine
   * Fix maxsize enforcement
   * Fix cache key generation
   * Fix cache clearing behavior

3. Fix failing tests:
   * Fix CacheConfig initialization tests
   * Fix cache stats tracking tests
   * Fix maxsize enforcement tests
   * Fix cache clearing tests

## [1.7.7] - 2025-02-16

### Changed

* Reorganized TODO.md to better reflect current priorities
* Updated CacheConfig implementation to use proper Pydantic fields
* Fixed linting issues in configuration system

### Fixed

* Removed leading underscores from CacheConfig field names
* Fixed field validation in CacheConfig
* Fixed property access in CacheConfig

### Issues

* CacheConfig initialization is broken (TypeError: CacheConfig() takes no arguments)
* Cache stats tracking is not working (hits/misses not being counted)
* Cache maxsize enforcement is not working (cache growing beyond limit)
* Cache clearing behavior is inconsistent
* Field validation in CacheConfig needs fixing

### Next Steps

1. Fix CacheConfig initialization:
   * Implement proper Pydantic model initialization
   * Fix field validation
   * Add proper error messages

2. Fix cache engine issues:
   * Fix stats tracking in FunctoolsCacheEngine
   * Fix maxsize enforcement
   * Fix cache key generation
   * Fix cache clearing behavior

3. Improve test coverage:
   * Add missing engine-specific tests
   * Add benchmark tests
   * Add stress tests
   * Add integration tests

## [1.7.6] - 2025-02-16

### Changed

* Reorganized TODO.md to better track progress
* Updated test suite with comprehensive engine tests
* Improved error handling in cache engines

### Fixed

* Renamed `lru.py` to `functools.py` for clarity
* Removed redundant SQL and Redis implementations
* Added proper type hints and docstrings

## [1.7.5] - 2025-02-15

### Fixed

* Fixed README.md formatting

## [1.7.3] - 2025-02-15

### Changed

* Minor documentation updates
* Internal code improvements

## [1.7.0] - 2025-02-13

### Added

* Added pyupgrade to development dependencies
* Added new fix command in pyproject.toml for automated code fixes
* Enhanced test environment with specific pytest dependencies
  * Added pytest-xdist for parallel test execution
  * Added pytest-benchmark for performance testing

### Changed

* Reorganized imports in core module
* Updated gitignore to exclude \_private directory

### Fixed

* Fixed import statement in **init**.py
* Improved development tooling configuration

## [1.6.2] - 2025-02-06

### Fixed

* Bug fixes and stability improvements

## [1.6.1] - 2025-02-06

### Changed

* Minor updates and improvements

## [1.6.0] - 2025-02-06

### Added

* Initial GitHub repository setup
* Comprehensive project structure
* Basic caching functionality
* Multiple backend support (memory, SQL, joblib)
* Automatic cache directory management
* Type hints and modern Python features

## [1.1.0] - 2025-02-03

### Added

* Early development version
* Core caching functionality

## [1.0.0] - 2025-02-03

### Added

* Initial release
* Basic memory caching implementation

## \[1.8.2] - 2025-02-19

### Changed

* Refocused project goals on simplicity and ease of use
* Streamlined TODO list to prioritize core functionality
* Reorganized implementation plan to focus on essential features

### Added

* Clear implementation status tracking in TODO.md
* Detailed testing strategy
* Documentation approach guidelines

### Removed

* Unnecessary complexity from implementation plan
* Redundant feature requests
* Overly complex API specifications

### Next Steps

1. Fix Critical Issues:
   * Remove magic numbers from tests
   * Fix boolean parameter warnings
   * Address unused imports
   * Fix function complexity issues

2. Improve Core Functionality:
   * Complete fallback mechanism
   * Add proper logging
   * Implement cache stats
   * Add inspection utilities

3. Enhance Testing:
   * Add test constants
   * Implement async tests
   * Add stress tests
   * Add backend-specific tests

[1.7.9]: https://github.com/twardoch/twat-cache/compare/v1.7.8...v1.7.9

[1.7.8]: https://github.com/twardoch/twat-cache/compare/v1.7.7...v1.7.8

[1.7.7]: https://github.com/twardoch/twat-cache/compare/v1.7.6...v1.7.7

[1.7.6]: https://github.com/twardoch/twat-cache/compare/v1.7.5...v1.7.6

[1.7.5]: https://github.com/twardoch/twat-cache/compare/v1.7.3...v1.7.5

[1.7.3]: https://github.com/twardoch/twat-cache/compare/v1.7.0...v1.7.3

[1.7.0]: https://github.com/twardoch/twat-cache/compare/v1.6.2...v1.7.0

[1.6.2]: https://github.com/twardoch/twat-cache/compare/v1.6.1...v1.6.2

[1.6.1]: https://github.com/twardoch/twat-cache/compare/v1.6.0...v1.6.1

[1.6.0]: https://github.com/twardoch/twat-cache/compare/v1.1.0...v1.6.0

[1.1.0]: https://github.com/twardoch/twat-cache/compare/v1.0.0...v1.1.0

[1.0.0]: https://github.com/twardoch/twat-cache/releases/tag/v1.0.0

# Development Log

### Completed

* Updated TODO.md with clearer structure and priorities
* Fixed Pydantic field validators
* Updated model\_config settings
* Identified critical import issue in test\_engines.py

### In Progress

* Fixing CacheEngine import issue
* Fixing CacheConfig initialization
* Addressing linting issues
* Improving test coverage

### Next Steps

1. Fix Critical Issues:
   * Fix CacheEngine import
   * Fix CacheConfig initialization
   * Fix test failures
   * Address linting issues

2. Improve Core Functionality:
   * Fix stats tracking
   * Fix maxsize enforcement
   * Improve cache key generation
   * Add proper error handling

3. Enhance Testing:
   * Fix failing tests
   * Add comprehensive test suite
   * Add performance benchmarks

# Critical Considerations

1. Engine System:
   * CacheEngine must be properly exported from base.py
   * All engines must properly implement the base interface
   * Stats tracking must be consistent across all engines

2. Configuration System:
   * CacheConfig must properly initialize with constructor arguments
   * Field validation must be robust and informative
   * Cache directory management must be reliable

3. Testing System:
   * All engines must be properly tested
   * Performance benchmarks must be comprehensive
   * Error handling must be thoroughly tested

## [1.7.8]: https://github.com/twardoch/twat-cache/compare/v1.7.7...v1.7.8

## [1.7.7]: https://github.com/twardoch/twat-cache/compare/v1.7.6...v1.7.7

## [1.7.6]: https://github.com/twardoch/twat-cache/compare/v1.7.5...v1.7.6

## [1.7.5]: https://github.com/twardoch/twat-cache/compare/v1.7.3...v1.7.5

## [1.7.3]: https://github.com/twardoch/twat-cache/compare/v1.7.0...v1.7.3

## [1.7.0]: https://github.com/twardoch/twat-cache/compare/v1.6.2...v1.7.0

## [1.6.2]: https://github.com/twardoch/twat-cache/compare/v1.6.1...v1.6.2

## [1.6.1]: https://github.com/twardoch/twat-cache/compare/v1.6.0...v1.6.1

## [1.6.0]: https://github.com/twardoch/twat-cache/compare/v1.1.0...v1.6.0

## [1.1.0]: https://github.com/twardoch/twat-cache/compare/v1.0.0...v1.1.0

## [1.0.0]: https://github.com/twardoch/twat-cache/releases/tag/v1.0.0

# Development Log

### Completed

* Fixed Pydantic field validators in CacheConfig:
  * Updated validators to use @classmethod decorator
  * Fixed model\_config settings
  * Improved error messages
* Reorganized TODO.md:
  * Added clearer structure
  * Prioritized critical fixes
  * Added Phase 3 for advanced features
* Updated test suite:
  * Added more comprehensive test cases
  * Identified failing tests
  * Added test categories

### In Progress

* Fixing CacheConfig initialization issues:
  * Working on proper Pydantic model setup
  * Fixing field validation and property access
  * Adding proper error messages
* Fixing cache engine issues:
  * Implementing proper stats tracking
  * Fixing maxsize enforcement
  * Improving cache key generation
  * Fixing cache clearing behavior

### Next Steps

1. Fix Critical Issues:
   * Fix CacheConfig initialization
   * Fix cache stats tracking
   * Fix maxsize enforcement
   * Fix cache clearing behavior

2. Enhance Test Coverage:
   * Fix failing tests
   * Add missing engine-specific tests
   * Add performance tests

3. Improve Documentation:
   * Update configuration guide
   * Add troubleshooting guide
   * Add performance tips
   * Add examples

# Critical Considerations

1. Configuration System:
   * CacheConfig must properly initialize with constructor arguments
   * Field validation must work correctly
   * Property access must be reliable
   * Error messages must be helpful

2. Cache Engine Behavior:
   * Stats tracking must be accurate
   * Maxsize enforcement must be reliable
   * Cache clearing must be consistent
   * Key generation must handle all types

3. Test Coverage:
   * All critical functionality must be tested
   * Edge cases must be covered
   * Performance must be verified
   * Error handling must be tested

# Conslidation UI Project Analysis & Plan

## Progress Log

### 2024-03-21

* Updated TODO.md with completed tasks and reorganized remaining work
* Major progress in core functionality:
  * Completed all basic cache engine implementations
  * Fixed cache stats tracking and clearing behavior
  * Improved type hints and configuration system
  * Added comprehensive test coverage for core features
* Next steps prioritized:
  1. Add missing engine-specific tests (eviction, concurrency, etc.)
  2. Implement serialization protocol
  3. Add async test coverage
  4. Create comprehensive documentation

Critical Considerations:

1. Engine-specific Tests:
   * Need stress tests for memory limits and eviction policies
   * Add concurrent access testing with high load
   * Test edge cases like cache corruption and recovery
   * Verify proper cleanup of resources

2. Serialization Protocol:
   * Add support for complex objects and custom types
   * Consider compression for large cached values
   * Implement versioning for cached data format
   * Add validation for serialized data integrity

3. Async Support:
   * Improve error handling for timeouts and connection issues
   * Ensure proper cleanup of async resources
   * Add retry mechanisms for transient failures
   * Test interaction with different event loops

4. Documentation:
   * Add detailed performance characteristics
   * Document tradeoffs between different cache engines
   * Create troubleshooting guide
   * Add migration guide for users of removed implementations

## Recent Changes

### 2024-03-21

* Fixed CacheConfig implementation:
  * Added proper field aliases for configuration parameters
  * Implemented abstract base class methods
  * Added Pydantic model configuration for proper field handling
  * Fixed validation method
* Updated test suite:
  * Fixed failing tests related to CacheConfig instantiation
  * Added tests for field aliases and validation
  * Improved test coverage for configuration handling

### Next Steps

1. Fix remaining test failures:
   * Cache stats tracking
   * List/unhashable type handling
   * Cache clearing behavior
2. Implement missing engine-specific tests:
   * Test cache eviction policies
   * Test concurrent access
   * Test error handling
   * Test serialization/deserialization
   * Test memory limits

### Critical Considerations

1. Engine-specific Tests:
   * Need stress tests for memory limits and eviction policies
   * Add concurrent access testing with high load
   * Test edge cases like cache corruption and recovery
   * Verify proper cleanup of resources

2. Serialization Protocol:
   * Add support for complex objects and custom types
   * Consider compression for large cached values
   * Implement versioning for cached data format
   * Add validation for serialized data integrity

3. Async Support:
   * Improve error handling for timeouts and connection issues
   * Ensure proper cleanup of async resources
   * Add retry mechanisms for transient failures
   * Test interaction with different event loops

4. Documentation:
   * Add detailed performance characteristics
   * Document tradeoffs between different cache engines
   * Create troubleshooting guide
   * Add migration guide for users of removed implementations

### 2024-03-19

#### Current Status

* Ran development setup and tests
* Found critical issues that need immediate attention:
  1. Pydantic field validator errors in CacheConfig
  2. 27 remaining linting issues
  3. Test failures across all test files

#### Critical Issues

1. **Pydantic Integration Issues**
   * Error: `@field_validator` cannot be applied to instance methods

   * Location: `src/twat_cache/config.py`

   * Impact: Breaking all tests due to config initialization failure

2. **Linting Issues**
   * 27 remaining issues including:

     * Magic numbers in comparisons
     * Print statements in production code
     * Boolean positional arguments
     * Module naming conflicts
     * Complex function warnings

#### Next Steps

1. **Immediate Fixes**
   * Fix Pydantic field validator implementation in CacheConfig
   * Address critical linting issues affecting functionality
   * Fix test failures

2. **Short-term Tasks**
   * Complete cache stats tracking implementation
   * Add proper type constraints
   * Implement missing test cases

3. **Documentation Updates**
   * Document configuration system
   * Add validation rules
   * Update type hints documentation

================
File: cleanup.py
================
LOG_FILE = Path("CLEANUP.txt")
os.chdir(Path(__file__).parent)
def new() -> None:
    if LOG_FILE.exists():
        LOG_FILE.unlink()
def prefix() -> None:
    readme = Path(".cursor/rules/0project.mdc")
    if readme.exists():
        log_message("\n=== PROJECT STATEMENT ===")
        content = readme.read_text()
        log_message(content)
def suffix() -> None:
    todo = Path("TODO.md")
    if todo.exists():
        log_message("\n=== TODO.md ===")
        content = todo.read_text()
def log_message(message: str) -> None:
    timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
    with LOG_FILE.open("a") as f:
        f.write(log_line)
def run_command(cmd: list[str], check: bool = True) -> subprocess.CompletedProcess:
        result = subprocess.run(
            log_message(result.stdout)
        log_message(f"Command failed: {' '.join(cmd)}")
        log_message(f"Error: {e.stderr}")
        return subprocess.CompletedProcess(cmd, 1, "", str(e))
def check_command_exists(cmd: str) -> bool:
        return which(cmd) is not None
class Cleanup:
    def __init__(self) -> None:
        self.workspace = Path.cwd()
    def _print_header(self, message: str) -> None:
        log_message(f"\n=== {message} ===")
    def _check_required_files(self) -> bool:
            if not (self.workspace / file).exists():
                log_message(f"Error: {file} is missing")
    def _generate_tree(self) -> None:
        if not check_command_exists("tree"):
            log_message("Warning: 'tree' command not found. Skipping tree generation.")
            rules_dir = Path(".cursor/rules")
            rules_dir.mkdir(parents=True, exist_ok=True)
            tree_result = run_command(
            with open(rules_dir / "filetree.mdc", "w") as f:
                f.write("---\ndescription: File tree of the project\nglobs: \n---\n")
                f.write(tree_text)
            log_message("\nProject structure:")
            log_message(tree_text)
            log_message(f"Failed to generate tree: {e}")
    def _git_status(self) -> bool:
        result = run_command(["git", "status", "--porcelain"], check=False)
        return bool(result.stdout.strip())
    def _venv(self) -> None:
        log_message("Setting up virtual environment")
            run_command(["uv", "venv"])
            if venv_path.exists():
                os.environ["VIRTUAL_ENV"] = str(self.workspace / ".venv")
                log_message("Virtual environment created and activated")
                log_message("Virtual environment created but activation failed")
            log_message(f"Failed to create virtual environment: {e}")
    def _install(self) -> None:
        log_message("Installing package with all extras")
            self._venv()
            run_command(["uv", "pip", "install", "-e", ".[test,dev]"])
            log_message("Package installed successfully")
            log_message(f"Failed to install package: {e}")
    def _run_checks(self) -> None:
        log_message("Running code quality checks")
            log_message(">>> Running code fixes...")
            run_command(
            log_message(">>>Running type checks...")
            run_command(["python", "-m", "mypy", "src", "tests"], check=False)
            log_message(">>> Running tests...")
            run_command(["python", "-m", "pytest", "tests"], check=False)
            log_message("All checks completed")
            log_message(f"Failed during checks: {e}")
    def status(self) -> None:
        prefix()  # Add README.md content at start
        self._print_header("Current Status")
        self._check_required_files()
        self._generate_tree()
        result = run_command(["git", "status"], check=False)
        self._print_header("Environment Status")
        self._install()
        self._run_checks()
        suffix()  # Add TODO.md content at end
    def venv(self) -> None:
        self._print_header("Virtual Environment Setup")
    def install(self) -> None:
        self._print_header("Package Installation")
    def update(self) -> None:
        self.status()
        if self._git_status():
            log_message("Changes detected in repository")
                run_command(["git", "add", "."])
                run_command(["git", "commit", "-m", commit_msg])
                log_message("Changes committed successfully")
                log_message(f"Failed to commit changes: {e}")
            log_message("No changes to commit")
    def push(self) -> None:
        self._print_header("Pushing Changes")
            run_command(["git", "push"])
            log_message("Changes pushed successfully")
            log_message(f"Failed to push changes: {e}")
def repomix(
            cmd.append("--compress")
            cmd.append("--remove-empty-lines")
            cmd.append("-i")
            cmd.append(ignore_patterns)
        cmd.extend(["-o", output_file])
        run_command(cmd)
        log_message(f"Repository content mixed into {output_file}")
        log_message(f"Failed to mix repository: {e}")
def print_usage() -> None:
    log_message("Usage:")
    log_message("  cleanup.py status   # Show current status and run all checks")
    log_message("  cleanup.py venv     # Create virtual environment")
    log_message("  cleanup.py install  # Install package with all extras")
    log_message("  cleanup.py update   # Update and commit changes")
    log_message("  cleanup.py push     # Push changes to remote")
def main() -> NoReturn:
    new()  # Clear log file
    if len(sys.argv) < 2:
        print_usage()
        sys.exit(1)
    cleanup = Cleanup()
            cleanup.status()
            cleanup.venv()
            cleanup.install()
            cleanup.update()
            cleanup.push()
        log_message(f"Error: {e}")
    repomix()
    sys.stdout.write(Path("CLEANUP.txt").read_text())
    sys.exit(0)  # Ensure we exit with a status code
    main()

================
File: LICENSE
================
MIT License

Copyright (c) 2025 Adam Twardoch

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

================
File: NEXT.md
================
---
this_file: NEXT.md
---

# Data Type-Based Cache Backend Selection Strategy

## 1. Introduction

The `twat_cache` library supports multiple caching backends, each with its own strengths and weaknesses. This document outlines a strategy for selecting the most appropriate cache backend based on the type of data being cached. This strategy aims to optimize performance, memory usage, and reliability for different data types.

## 2. Available Cache Backends

Based on the codebase, we support the following cache backends:

1. **functools** - Simple in-memory caching using Python's built-in `functools.lru_cache`
2. **cachetools** - Flexible in-memory caching with multiple eviction policies
3. **diskcache** - Persistent disk-based caching with SQLite backend
4. **joblib** - Specialized caching for scientific computing (NumPy arrays, pandas DataFrames)
5. **klepto** - Advanced caching with multiple backends and policies
6. **aiocache** - Async-compatible caching
7. **cachebox** - High-performance caching

## 3. Data Type Characteristics

Different types of data have different characteristics that make them more suitable for certain cache backends:

### 3.1. Size
- **Small data** (<10KB): Simple strings, numbers, small dictionaries/lists
- **Medium data** (10KB-1MB): Larger collections, medium-sized images, medium JSON objects
- **Large data** (1MB-100MB): Large datasets, images, serialized models
- **Very large data** (>100MB): Video files, very large datasets, complex ML models

### 3.2. Serialization Complexity
- **Simple**: Basic Python types (int, float, str, bool)
- **Moderate**: Lists, dictionaries with simple values, custom classes with simple attributes
- **Complex**: Nested objects, custom classes with many attributes, objects with circular references
- **Specialized**: NumPy arrays, pandas DataFrames, scientific computing objects, ML models

### 3.3. Access Patterns
- **Read-heavy**: Data that is read frequently but written rarely
- **Write-heavy**: Data that is updated frequently
- **Balanced**: Similar read and write frequencies
- **Temporal locality**: Data accessed in clusters over time
- **Spatial locality**: Related data items accessed together

### 3.4. Persistence Requirements
- **Ephemeral**: Only needed during program execution
- **Session-persistent**: Needed across multiple runs in a session
- **Long-term persistent**: Needed across multiple sessions/days

### 3.5. Concurrency Requirements
- **Single-threaded**: No concurrent access
- **Multi-threaded**: Concurrent access from multiple threads
- **Multi-process**: Concurrent access from multiple processes
- **Distributed**: Concurrent access from multiple machines

## 4. Backend Selection Strategy

### 4.1. Default Backend Selection by Data Type

| Data Type | Recommended Backend | Alternative Backend | Justification |
|-----------|---------------------|---------------------|---------------|
| **Simple Python types** (int, str, bool) | cachetools | functools | Efficient in-memory storage, low overhead |
| **Collections** (list, dict, set) | cachetools | klepto | Flexible policies, good performance for medium collections |
| **NumPy arrays** | joblib | diskcache | Optimized for NumPy arrays, efficient serialization |
| **Pandas DataFrames** | joblib | diskcache | Optimized for pandas objects, handles large datasets well |
| **Images/Binary data** | diskcache | klepto | Efficient storage of binary data on disk |
| **Large JSON objects** | diskcache | klepto | Persistent storage, good for structured data |
| **Machine Learning models** | joblib | diskcache | Specialized serialization, handles large binary objects |
| **Custom classes** | klepto | diskcache | Flexible serialization options |
| **Async coroutine results** | aiocache | cachetools | Designed for async compatibility |

### 4.2. Selection by Data Characteristics

#### 4.2.1. By Size

| Data Size | Recommended Backend | Justification |
|-----------|---------------------|---------------|
| **Small** (<10KB) | cachetools | Fast in-memory access, low overhead |
| **Medium** (10KB-1MB) | klepto or diskcache | Good balance of performance and memory usage |
| **Large** (1MB-100MB) | diskcache or joblib | Efficient disk storage, specialized serialization |
| **Very large** (>100MB) | joblib | Optimized for large scientific data |

#### 4.2.2. By Persistence Requirements

| Persistence Need | Recommended Backend | Justification |
|------------------|---------------------|---------------|
| **Ephemeral** | cachetools or functools | Fast in-memory caching, no disk overhead |
| **Session-persistent** | klepto or diskcache | Can persist to disk but still relatively fast |
| **Long-term persistent** | diskcache | Reliable SQL-based storage, data survives restarts |

#### 4.2.3. By Access Pattern

| Access Pattern | Recommended Backend | Policy Setting | Justification |
|----------------|---------------------|----------------|---------------|
| **Read-heavy** | Any backend | LRU policy | Standard caching approach |
| **Write-heavy** | cachetools or diskcache | Small maxsize | Prevent cache thrashing |
| **Temporal locality** | cachetools | LRU policy | Optimized for recently used items |
| **Frequency-based** | cachetools or klepto | LFU policy | Keeps frequently accessed items |
| **Random access** | klepto | RR (Random Replacement) | Simple eviction suitable for random patterns |

#### 4.2.4. By Concurrency Requirements

| Concurrency Need | Recommended Backend | Justification |
|------------------|---------------------|---------------|
| **Single-threaded** | Any backend | All work well in this simple case |
| **Multi-threaded** | cachetools or diskcache | Thread-safe implementations |
| **Multi-process** | diskcache | Shared storage accessible across processes |
| **Distributed** | diskcache with network path | Can be accessed from multiple machines |

### 4.3. Implementation Approach

To implement this strategy in the `twat_cache` library, we recommend:

1. **Automatic Detection**: Analyze the return type of cached functions to suggest an appropriate backend.
   
2. **Configuration Helpers**: Provide helper functions for common data types:
   ```python
   # Example API
   from twat_cache import configure_for_numpy, configure_for_json, configure_for_images
   
   @cache(config=configure_for_numpy())
   def process_array(data: np.ndarray) -> np.ndarray:
       # Process the array
       return result
   ```

3. **Smart Defaults**: When no specific engine is requested, select based on:
   - Function return type annotations if available
   - Runtime analysis of actual returned values
   - Configuration hints like `maxsize` and `folder_name`

4. **Hybrid Caching**: For functions that return different types based on input:
   ```python
   @cache(config=hybrid_cache_config(
       small_result_engine="cachetools",
       large_result_engine="diskcache",
       size_threshold=1024*1024  # 1MB
   ))
   def get_data(id: str) -> dict | bytes:
       # Returns either metadata (dict) or raw data (bytes)
       # The appropriate cache will be selected based on return size
   ```

## 5. Implementation Plan

1. Add a data type analyzer utility to inspect function returns
2. Create configuration factory functions for common data types
3. Implement smart backend selection based on data characteristics
4. Add hybrid caching capability for mixed-return functions
5. Update documentation with examples of type-based configuration

## 6. Conclusion

By matching data types to appropriate cache backends, we can significantly improve the performance and efficiency of the caching system. This strategy provides a framework for making these decisions systematically, whether manually configured by users or automatically determined by the library.

The core principle is: **Let the data characteristics guide the cache backend selection.**

================
File: PROMPT.txt
================
1. Read @README.md — note that in the .venv you have the actual code of the 3rd party libs to consult!

2. Read @LOG.md and @TODO.md

3. Step back and reason about the rationale: we wanted a simple package that exposes some decorators and passes all the cache heavy lifting to the 3rd parties. Check the implementation and tell me how we match. 

4. Reduce the common API of our code to the minimum, don't try to expose a universal API that does everything. For that, implement an easy passthru that just exposes the objects or functions provided by the 3rd parties

5. What DOES matter, though, is ease of use. We wanted the 3 specialized decorators (mcache, bcache, fcache) and a universal ucache decorator. What I want is a seamless integration of the engines so that if a given engine isn't available, we issue a logger warning but a fallback mechanism kicks in. 

6. Rewrite our goals in @TODO.md to match the above

7. Adjusts the tests so they’re sensible and realistic. 

Run often:

```
uv venv; source .venv/bin/activate; uv pip install -e .[all,dev,test]; tree -h -I '*_cache' -I __pycache__ .; tree -h -I '*.dist-info' -I 'mypy' -I 'mypyc' -I 'data' .venv; hatch run lint:fix; hatch test;
```

REPEAT IT OFTEN.

Then adjust our implementation, and continue TODO

================
File: pyproject.toml
================
# this_file: twat_cache/pyproject.toml
# Build System Configuration
# -------------------------
# Specifies the build system and its requirements for packaging the project
[build-system]
requires = [
    "hatchling>=1.27.0", # Core build backend for Hatch
    "hatch-vcs>=0.4.0", # Version Control System plugin for Hatch
]
build-backend = "hatchling.build" # Use Hatchling as the build backend

# Project Metadata Configuration
# ------------------------------
# Comprehensive project description, requirements, and compatibility information
[project]
name = "twat-cache"
dynamic = ["version"]
description = "Advanced caching library for Python, part of the twat framework"
readme = "README.md"
requires-python = ">=3.10" # Minimum Python version required
license = "MIT"
keywords = [
    "caching", 
    "cache", 
    "memorization", 
    "performance", 
    "optimization"
]
classifiers = [
    "Development Status :: 4 - Beta",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
    "Programming Language :: Python :: Implementation :: CPython",
    "Programming Language :: Python :: Implementation :: PyPy",
]
dependencies = [
    "pydantic>=2.0",
    "loguru>=0.7.0",
    "diskcache>=5.6.1",
    "joblib>=1.3.2",
    "cachetools>=5.3.2",
]

[[project.authors]]
name = "Adam Twardoch"
email = "adam+github@twardoch.com"

[project.urls]
Documentation = "https://github.com/twardoch/twat/tree/main/docs"
Issues = "https://github.com/twardoch/twat/issues"
Source = "https://github.com/twardoch/twat"

[project.entry-points."twat.plugins"]
cache = "twat_cache"

[tool.hatch.build.targets.wheel]
packages = ["src/twat_cache"]

[tool.hatch.envs.default]
dependencies = ["mypy>=1.0.0", "ruff>=0.1.0"]

[project.optional-dependencies]
test = [
    "pytest>=7.0.0",
    "pytest-cov>=4.0.0",
    "pytest-benchmark[histogram]>=4.0.0",
    "pytest-xdist>=3.5.0", # For parallel test execution
]
dev = ["pre-commit>=3.6.0"]
all = [
    'platformdirs>=4.3.6',
    'cachebox>=4.5.1',
    'cachetools>=5.5.1',
    'aiocache>=0.12.3',
    'klepto>=0.2.6',
    'diskcache>=5.6.3',
    'joblib>=1.4.2',
]

[tool.hatch.envs.test]
dependencies = [".[test]"]

[tool.hatch.envs.test.scripts]
test = "python -m pytest -n auto {args:tests}"
test-cov = "python -m pytest -n auto --cov-report=term-missing --cov-config=pyproject.toml --cov=src/twat_cache --cov=tests {args:tests}"
bench = "python -m pytest -v -p no:briefcase tests/test_benchmark.py --benchmark-only"
bench-save = "python -m pytest -v -p no:briefcase tests/test_benchmark.py --benchmark-only --benchmark-json=benchmark/results.json"
bench-hist = "python -m pytest -v -p no:briefcase tests/test_benchmark.py --benchmark-only --benchmark-histogram=benchmark/hist"
bench-compare = "python -m pytest-benchmark compare benchmark/results.json --sort fullname --group-by func"

[tool.hatch.envs.lint]
detached = true
dependencies = ["black>=23.1.0", "mypy>=1.0.0", "ruff>=0.1.0"]

[tool.hatch.envs.lint.scripts]
typing = "mypy --install-types --non-interactive {args:src/twat_cache tests}"
style = ["ruff check {args:.}", "ruff format {args:.}"]
fmt = ["ruff format {args:.}", "ruff check --fix {args:.}"]
all = ["style", "typing"]

[tool.ruff]
target-version = "py310"
line-length = 88
lint.extend-select = [
    "I", # isort
    "N", # pep8-naming
    "B", # flake8-bugbear
    "RUF", # Ruff-specific rules
]
lint.ignore = [
    "ARG001", # Unused function argument
    "E501", # Line too long
    "I001",
]

[tool.ruff.format]
quote-style = "double"
indent-style = "space"
skip-magic-trailing-comma = false
line-ending = "lf"

[tool.ruff.lint.per-file-ignores]
"tests/*" = ["S101"]

[tool.coverage.run]
source_pkgs = ["twat_cache", "tests"]
branch = true
parallel = true
omit = ["src/twat_cache/__about__.py"]

[tool.coverage.paths]
twat_cache = ["src/twat_cache", "*/twat-cache/src/twat_cache"]
tests = ["tests", "*/twat-cache/tests"]

[tool.coverage.report]
exclude_lines = ["no cov", "if __name__ == .__main__.:", "if TYPE_CHECKING:"]

[tool.mypy]
python_version = "3.10"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
check_untyped_defs = true
disallow_untyped_decorators = true
no_implicit_optional = true
warn_redundant_casts = true
warn_unused_ignores = true
warn_no_return = true
warn_unreachable = true

[tool.pytest.ini_options]
markers = ["benchmark: marks tests as benchmarks (select with '-m benchmark')"]
addopts = "-v -p no:briefcase"
testpaths = ["tests"]
python_files = ["test_*.py"]
filterwarnings = ["ignore::DeprecationWarning", "ignore::UserWarning"]
asyncio_mode = "auto"
asyncio_default_fixture_loop_scope = "function"

[tool.pytest-benchmark]
min_rounds = 100
min_time = 0.1
histogram = true
storage = "file"
save-data = true
compare = [
    "min", # Minimum time
    "max", # Maximum time
    "mean", # Mean time
    "stddev", # Standard deviation
    "median", # Median time
    "iqr", # Inter-quartile range
    "ops", # Operations per second
    "rounds", # Number of rounds
]

[tool.hatch.version]
source = "vcs"

================
File: README.md
================
# twat-cache

A flexible caching utility package for Python functions that provides a unified interface for caching function results using various high-performance backends.

## Features

- Simple decorator interface for caching function results
- Multiple caching backends with automatic selection:
  1. `cachebox` - Very fast Rust-based cache (optional)
  2. `cachetools` - Flexible in-memory caching (optional)
  3. `aiocache` - Async-capable caching (optional)
  4. `klepto` - Scientific computing caching (optional)
  5. `diskcache` - SQL-based disk cache (optional)
  6. `joblib` - Efficient array caching (optional)
  7. `redis` - Distributed caching with Redis (optional)
  8. Memory-based LRU cache (always available)
- Automatic cache directory management
- Type hints and modern Python features
- Lazy backend loading - only imports what you use
- Automatic backend selection based on availability and use case
- Smart backend selection based on data characteristics
- TTL support for cache expiration
- Multiple eviction policies (LRU, LFU, FIFO, RR)
- Async function support
- Compression options for large data
- Secure file permissions for sensitive data
- Hybrid caching with automatic backend switching
- Context management for cache engines
- Comprehensive test suite for all components

## Recent Updates (v2.3.0)

### Enhanced Context Management

The context management system has been significantly improved:
- Better error handling and resource cleanup
- Support for explicit engine selection
- Simplified API for temporary cache configurations
- Automatic cleanup of resources even when exceptions occur

```python
# Example of improved context management
from twat_cache import CacheContext

# Create a context with explicit engine selection
with CacheContext(engine_name="redis", namespace="user_data") as cache:
    # Use the cache within the context
    cache.set("user:1001", {"name": "John", "role": "admin"})
    user = cache.get("user:1001")
    
    # Resources are automatically cleaned up when exiting the context
```

### Refined Backend Selection

The backend selection strategy has been further enhanced:
- More accurate data type detection for optimal backend selection
- Improved fallback mechanisms when preferred backends are unavailable
- Better handling of edge cases for various data types
- Enhanced performance for frequently accessed items

### Comprehensive Documentation

Documentation has been expanded with:
- Detailed examples for all cache backends
- Step-by-step guides for common use cases
- API reference with complete parameter descriptions
- Best practices for cache configuration

## Installation

Basic installation with just LRU caching:
```bash
pip install twat-cache
```

With all optional backends:
```bash
pip install twat-cache[all]
```

Or install specific backends:
```bash
pip install twat-cache[cachebox]     # For Rust-based high-performance cache
pip install twat-cache[cachetools]   # For flexible in-memory caching
pip install twat-cache[aiocache]     # For async-capable caching
pip install twat-cache[klepto]       # For scientific computing caching
pip install twat-cache[diskcache]    # For SQL-based disk caching
pip install twat-cache[joblib]       # For efficient array caching
pip install twat-cache[redis]        # For distributed caching with Redis
```

## Usage

### Basic Memory Caching

For simple in-memory caching with LRU eviction:

```python
from twat_cache import mcache

@mcache(maxsize=100)  # Cache up to 100 items
def expensive_function(x: int) -> int:
    # Expensive computation here
    return x * x

# First call computes
result1 = expensive_function(5)  # Computes 25

# Second call uses cache
result2 = expensive_function(5)  # Returns cached 25
```

### Disk-Based Caching

For persistent caching using SQLite:

```python
from twat_cache import bcache

@bcache(
    folder_name="my_cache",  # Cache directory name
    maxsize=1_000_000,       # Max cache size in bytes
    ttl=3600,               # Cache entries expire after 1 hour
    use_sql=True,           # Use SQLite backend
    secure=True,            # Use secure file permissions
)
def expensive_function(x: int) -> int:
    return x * x
```

### Redis Distributed Caching

For distributed caching with Redis:

```python
from twat_cache import ucache

@ucache(
    preferred_engine="redis",
    folder_name="redis_cache",  # Used as Redis namespace
    ttl=3600,                  # Cache entries expire after 1 hour
    compress=True,             # Enable compression
)
def expensive_function(x: int) -> int:
    return x * x
```

### File-Based Caching

For efficient caching of large objects like NumPy arrays:

```python
from twat_cache import fcache
import numpy as np

@fcache(
    folder_name="array_cache",
    compress=True,           # Enable compression
    secure=True,            # Use secure file permissions
)
def process_array(data: np.ndarray) -> np.ndarray:
    # Expensive array processing here
    return data * 2
```

### Async Caching

For async functions with Redis or memory backend:

```python
from twat_cache import ucache

@ucache(use_async=True)
async def fetch_data(url: str) -> dict:
    # Async web request here
    return {"data": "..."}

# First call fetches
data1 = await fetch_data("https://api.example.com")

# Second call uses cache
data2 = await fetch_data("https://api.example.com")
```

### Universal Caching

Let the library choose the best backend:

```python
from twat_cache import ucache

@ucache(
    folder_name="cache",     # Optional - uses disk cache if provided
    maxsize=1000,           # Optional - limits cache size
    ttl=3600,              # Optional - entries expire after 1 hour
    policy="lru",          # Optional - LRU eviction (default)
    use_sql=True,          # Optional - use SQL backend if available
    compress=True,         # Optional - enable compression
    secure=True,           # Optional - secure file permissions
)
def my_function(x: int) -> int:
    return x * x
```

### Smart Backend Selection

Automatically select the best backend based on data characteristics:

```python
from twat_cache import smart_cache

@smart_cache()
def process_data(data_type: str, size: int) -> Any:
    """Process different types of data with automatic backend selection."""
    if data_type == "dict":
        return {f"key_{i}": f"value_{i}" for i in range(size)}
    elif data_type == "list":
        return [i for i in range(size)]
    elif data_type == "str":
        return "x" * size
    else:
        return size
```

### Hybrid Caching

Switch backends based on result size:

```python
from twat_cache import hybrid_cache

@hybrid_cache()
def get_data(size: str) -> Union[Dict[str, Any], List[int]]:
    """Return different sized data with appropriate backend selection."""
    if size == "small":
        # Small result, will use in-memory caching
        return {"name": "Small Data", "value": 42}
    else:
        # Large result, will use disk caching
        return [i for i in range(100000)]
```

### Type-Specific Configuration

Configure caching based on data types:

```python
from twat_cache import ucache, configure_for_numpy, configure_for_json

# For NumPy arrays
@ucache(config=configure_for_numpy())
def process_array(data: np.ndarray) -> np.ndarray:
    return data * 2

# For JSON data
@ucache(config=configure_for_json())
def fetch_json_data(url: str) -> Dict[str, Any]:
    return {"data": [1, 2, 3, 4, 5], "metadata": {"source": url}}
```

### Cache Management

Clear caches and get statistics:

```python
from twat_cache import clear_cache, get_stats

# Clear all caches
clear_cache()

# Get cache statistics
stats = get_stats()
print(stats)  # Shows hits, misses, size, etc.
```

### Context Management

Use cache engines with context management:

```python
from twat_cache import CacheContext, engine_context

# Method 1: Using the CacheContext class
with CacheContext(engine_name="diskcache", folder_name="cache") as cache:
    # Use the cache
    cache.set("key", "value")
    value = cache.get("key")
    
# Method 2: Using the engine_context function
with engine_context(engine_name="redis", ttl=3600) as cache:
    # Use the cache
    cache.set("key", "value")
    value = cache.get("key")
    
# Cache is automatically closed when exiting the context
```

## Advanced Features

### TTL Support

Set time-to-live for cache entries:

```python
from twat_cache import ucache

@ucache(ttl=3600)  # Entries expire after 1 hour
def get_weather(city: str) -> dict:
    # Fetch weather data
    return {"temp": 20}
```

### Eviction Policies

Choose from different cache eviction policies:

```python
from twat_cache import ucache

# Least Recently Used (default)
@ucache(policy="lru")
def function1(x: int) -> int:
    return x * x

# Least Frequently Used
@ucache(policy="lfu")
def function2(x: int) -> int:
    return x * x

# First In, First Out
@ucache(policy="fifo")
def function3(x: int) -> int:
    return x * x
```

## Documentation

For more detailed documentation, see the following resources:

- [Context Management](docs/context_management.md)
- [Backend Selection](docs/backend_selection.md)
- [Cache Engines](docs/cache_engines.md)
- [Configuration Options](docs/configuration.md)
- [API Reference](docs/api_reference.md)

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## License

This project is licensed under the MIT License - see the LICENSE file for details.

## Rationale

Python provides several powerful caching libraries to help optimize application performance by storing and reusing expensive function results. Let's take an in-depth look at some of the most popular options, comparing their features, backends, methods, and use cases.


### Built-in functools 

Python's standard library offers basic caching functionality through the `functools` module. It provides decorators like `@lru_cache(maxsize, typed)` for simple memoization with a Least Recently Used (LRU) eviction policy. The `@cache` decorator is also available, which is equivalent to an unbounded LRU cache.

- **Backend**: In-memory Python dictionary 
- **Eviction Policy**: LRU or unbounded
- **Concurrency**: Thread-safe via internal locks, but not safe for multi-process use
- **Persistence**: No persistence, cache only exists in memory for the lifetime of the process
- **Best For**: Fast and easy caching of function results in memory with minimal setup

```pyi
### functools.pyi (Partial - Caching related parts)

import typing

_T = typing.TypeVar("_T")

@typing.overload
def lru_cache(maxsize: int | None) -> typing.Callable[[_F], _F]: ...
@typing.overload
def lru_cache(maxsize: int | None, typed: bool) -> typing.Callable[[_F], _F]: ...

class _lru_cache_wrapper(typing.Generic[_F]):
    cache: typing.Dict[typing.Tuple[typing.Any, ...], typing.Any]
    def cache_info(self) -> CacheInfo: ...
    def cache_clear(self) -> None: ...
```

```python
import functools

@functools.lru_cache(maxsize=128)
def expensive_function(x):
    return x * 2
```

### cachetools

The `cachetools` library extends the capabilities of `functools` by offering additional cache classes with various eviction policies like LFU, FIFO, TTL, and RR. It also provides `@cached` and `@cachedmethod` decorators for caching functions and methods.

- **Backend**: In-memory Python dictionary
- **Eviction Policies**: LRU, LFU, TTL, FIFO, RR, TLRU 
- **Concurrency**: Thread-safe, not designed for multi-process use
- **Persistence**: In-memory only, no persistence 
- **Customization**: Configurable `maxsize`, `ttl`, custom `getsizeof` to determine item size
- **Best For**: In-memory caching with specific eviction behavior, e.g. caching web API responses

```pyi
### cachetools.pyi (Partial - Most important classes and decorators)

import typing
from typing import Callable

KT = typing.TypeVar("KT")
VT = typing.TypeVar("VT")
DT = typing.TypeVar("DT")

class Cache(typing.MutableMapping[KT, VT]):
    def __init__(self, maxsize: int, getsizeof: Callable[[VT], int] | None = ...) -> None: ...
    @property
    def maxsize(self) -> int: ...
    @property
    def currsize(self) -> int: ...
    def get(self, key: KT, default: DT = ...) -> VT | DT: ...
    def pop(self, key: KT, default: DT = ...) -> VT | DT: ...
    def setdefault(self, key: KT, default: DT = ...) -> VT | DT: ...
    def clear(self) -> None: ...
    @staticmethod
    def getsizeof(value: Any) -> int: ...

class LRUCache(Cache[KT, VT]):
    def __init__(self, maxsize: int, getsizeof: Callable[[VT], int] | None = ...) -> None: ...
    def popitem(self) -> tuple[KT, VT]: ...

class LFUCache(Cache[KT, VT]):
    def __init__(self, maxsize: int, getsizeof: Callable[[VT], int] | None = ...) -> None: ...
    def popitem(self) -> tuple[KT, VT]: ...

class TTLCache(LRUCache[KT, VT]):
    def __init__(self, maxsize: int, ttl: int, timer: Callable[[], float] | None = ..., getsizeof: Callable[[VT], int] | None = ...) -> None: ...
    @property
    def ttl(self) -> int: ...
    def expire(self, time: float | None = ...) -> list[tuple[KT, VT]]: ...
    def popitem(self) -> tuple[KT, VT]: ...

class _CacheInfo(NamedTuple):
    hits: int
    misses: int
    maxsize: int
    currsize: int

_KeyFunc = Callable[..., typing.Hashable]
_Cache = TypeVar("_Cache", bound=Cache)

@overload
def cached(
    cache: _Cache,
    key: _KeyFunc = ...,
    lock: Any | None = ...,
    info: Literal[False] = ...,
) -> Callable[[_F], _F]: ...
@overload
def cached(
    cache: _Cache,
    key: _KeyFunc = ...,
    lock: Any | None = ...,
    info: Literal[True] = ...,
) -> Callable[[_F], _F]: ...

@overload
def cachedmethod(
    cache: Callable[[Any], _Cache],
    key: _KeyFunc = ...,
    lock: Callable[[Any], Any] | None = ...,
) -> Callable[[_F], _F]: ...
@overload
def cachedmethod(
    cache: Callable[[Any], _Cache],
    key: _KeyFunc = ...,
    lock: None = ...,
) -> Callable[[_F], _F]: ...
```

### cachebox

`cachebox` is an in-memory caching solution accelerated by a Rust backend for enhanced performance. It supports similar eviction policies as `cachetools` including a unique variable TTL cache.

- **Backend**: Rust-backed in-memory store
- **Eviction Policies**: LRU, LFU, FIFO, TTL, RR, variable TTL
- **Concurrency**: Thread-safe Rust implementation, but in-memory only
- **Decorators**: `@cached`, `@cachedmethod` support custom key generation and callbacks
- **Best For**: Performant in-memory caching with ample policy choices

```pyi
### cachebox.pyi (Partial - Most important classes)

import typing

KT = typing.TypeVar("KT")
VT = typing.TypeVar("VT")
DT = typing.TypeVar("DT")


class BaseCacheImpl(Generic[KT, VT]):

    def __init__(
        self,
        maxsize: int,
        iterable: Union[Iterable[Tuple[KT, VT]], Dict[KT, VT]] = ...,
        *,
        capacity: int = ...,
    ) -> None: ...
    @property
    def maxsize(self) -> int: ...
    def __len__(self) -> int: ...
    def __contains__(self, key: KT) -> bool: ...
    def __setitem__(self, key: KT, value: VT) -> None: ...
    def __getitem__(self, key: KT) -> VT: ...
    def __delitem__(self, key: KT) -> VT: ...
    def __iter__(self) -> typing.Iterator[KT]: ...
    def capacity(self) -> int: ...
    def is_full(self) -> bool: ...
    def is_empty(self) -> bool: ...
    def insert(self, key: KT, value: VT) -> typing.Optional[VT]: ...
    def get(self, key: KT, default: DT = None) -> typing.Union[VT, DT]: ...
    def pop(self, key: KT, default: DT = None) -> typing.Union[VT, DT]: ...
    def setdefault(
        self, key: KT, default: typing.Optional[DT] = None
    ) -> typing.Optional[VT | DT]: ...
    def popitem(self) -> typing.Tuple[KT, VT]: ...
    def drain(self, n: int) -> int: ...
    def clear(self, *, reuse: bool = False) -> None: ...
    def shrink_to_fit(self) -> None: ...
    def update(
        self, iterable: typing.Union[typing.Iterable[typing.Tuple[KT, VT]], typing.Dict[KT, VT]]
    ) -> None: ...
    def keys(self) -> typing.Iterable[KT]: ...
    def values(self) -> typing.Iterable[VT]: ...
    def items(self) -> typing.Iterable[typing.Tuple[KT, VT]]: ...

class Cache(BaseCacheImpl[KT, VT]):
    def __init__(
        self,
        maxsize: int,
        iterable: Union[Iterable[Tuple[KT, VT]], Dict[KT, VT]] = ...,
        *,
        capacity: int = ...,
    ) -> None:
        ...
    def popitem(self) -> typing.NoReturn: ...  # not implemented for this class
    def drain(self, n: int) -> typing.NoReturn: ...  # not implemented for this class


class FIFOCache(BaseCacheImpl[KT, VT]):
    def __init__(
        self,
        maxsize: int,
        iterable: Union[Iterable[Tuple[KT, VT]], Dict[KT, VT]] = ...,
        *,
        capacity: int = ...,
    ) -> None:
        ...

class RRCache(BaseCacheImpl[KT, VT]):
    def __init__(
        self,
        maxsize: int,
        iterable: Union[Iterable[Tuple[KT, VT]], Dict[KT, VT]] = ...,
        *,
        capacity: int = ...,
    ) -> None:
        ...

class TTLCache(BaseCacheImpl[KT, VT]):
    def __init__(
        self,
        maxsize: int,
        ttl: float,
        iterable: Union[Iterable[Tuple[KT, VT]], Dict[KT, VT]] = ...,
        *,
        capacity: int = ...,
    ) -> None:
        ...

class LRUCache(BaseCacheImpl[KT, VT]):
    def __init__(
        self,
        maxsize: int,
        iterable: Union[Iterable[Tuple[KT, VT]], Dict[KT, VT]] = ...,
        *,
        capacity: int = ...,
    ) -> None:
        ...

class LFUCache(BaseCacheImpl[KT, VT]):
    def __init__(
        self,
        maxsize: int,
        iterable: Union[Iterable[Tuple[KT, VT]], Dict[KT, VT]] = ...,
        *,
        capacity: int = ...,
    ) -> None:
        ...

class VTTLCache(BaseCacheImpl[KT, VT]):
    def __init__(
        self,
        maxsize: int,
        iterable: Union[Iterable[Tuple[KT, VT]], Dict[KT, VT]] = ...,
        ttl: Optional[float] = 0.0,
        *,
        capacity: int = ...,
    ) -> None:
        ...

_CacheType = TypeVar("_CacheType", bound=BaseCacheImpl)

def cached(cache: typing.Optional[_CacheType], key_maker: typing.Optional[Callable[[tuple, dict], typing.Hashable]] = ..., clear_reuse: bool = ..., callback: typing.Optional[Callable[[int, typing.Any, typing.Any], typing.Any]] = ..., copy_level: int = ..., always_copy: typing.Optional[bool] = ...) -> Callable[[_F], _F]: ...
def cachedmethod(cache: typing.Optional[_CacheType], key_maker: typing.Optional[Callable[[tuple, dict], typing.Hashable]] = ..., clear_reuse: bool = ..., callback: typing.Optional[Callable[[int, typing.Any, typing.Any], typing.Any]] = ..., copy_level: int = ..., always_copy: typing.Optional[bool] = ...) -> Callable[[_F], _F]: ...
```

### klepto

For advanced caching workflows, `klepto` provides a highly flexible solution supporting both in-memory and persistent backends like file archives, SQL databases, and HDF5 files. It allows customizing how cache keys are generated and extends the standard cache eviction policies.

- **Backends**: In-memory dict, file archives, SQL databases, HDF5 files
- **Eviction Policies**: Same as `functools` plus LFU, MRU, RR
- **Key Mapping**: `hashmap`, `stringmap`, `picklemap` algorithms to generate keys
- **Persistence**: Archives cache to disk or database for long-term storage
- **Concurrency**: Locking for thread-safety, some process-safety depending on backend
- **Best For**: Complex scenarios needing transient and persistent caching

```pyi
### key maps
class keymap:
    def __init__(self, typed: bool = False, flat: bool = True,
                 sentinel: object=...) -> None: ...
    def __call__(self, *args, **kwargs) -> object: ...

class hashmap(keymap):
    def __init__(self, algorithm: Optional[str] = None,
                 typed: bool = False, flat: bool = True,
                 sentinel: object=...) -> None: ...

class stringmap(keymap):
    def __init__(self, encoding: Optional[str] = None,
                 typed: bool = False, flat: bool = True,
                 sentinel: object=...) -> None: ...

class picklemap(keymap):
    def __init__(self, serializer: Optional[str] = None,
                 typed: bool = False, flat: bool = True,
                 sentinel: object=...) -> None: ...

### base archive
class archive(dict):
    def __init__(self, *args, **kwds) -> None: ...
    def __asdict__(self) -> dict: ...
    def __repr__(self) -> str: ...
    def copy(self, name: Optional[str] = None) -> 'archive': ...
    def load(self, *args) -> None: ...
    def dump(self, *args) -> None: ...
    def archived(self, *on) -> bool: ...
    def sync(self, clear: bool = False) -> None: ...
    def drop(self) -> None: ...
    def open(self, archive: 'archive') -> None: ...

    @property
    def archive(self) -> 'archive': ...
    @archive.setter
    def archive(self, archive: 'archive') -> None: ...
    @property
    def name(self) -> str: ...
    @name.setter
    def name(self, archive: 'archive') -> None: ...
    @property
    def state(self) -> dict: ...
    @state.setter
    def state(self, archive: 'archive') -> None: ...

class dict_archive(archive):
    def __init__(self, *args, **kwds) -> None: ...

class null_archive(archive):
    def __init__(self, *args, **kwds) -> None: ...

class file_archive(archive):
    def __init__(self, filename: Optional[str] = None, dict: Optional[dict] = None,
                 cached: bool = True, serialized: bool = True,
                 protocol: Optional[int] = None, *args, **kwds) -> None: ...

class dir_archive(archive):
    def __init__(self, dirname: Optional[str] = None, dict: Optional[dict] = None,
                 cached: bool = True, serialized: bool = True,
                 compression: int = 0, permissions: Optional[int] = None,
                 memmode: Optional[str] = None, memsize: int = 100,
                 protocol: Optional[int] = None, *args, **kwds) -> None: ...

class sqltable_archive(archive):
    def __init__(self, database: Optional[str] = None,
                 table: Optional[str] = None, dict: Optional[dict] = None,
                 cached: bool = True, serialized: bool = True,
                 protocol: Optional[int] = None, *args, **kwds) -> None: ...

class sql_archive(archive):
    def __init__(self, database: Optional[str] = None, dict: Optional[dict] = None,
                 cached: bool = True, serialized: bool = True,
                 protocol: Optional[int] = None, *args, **kwds) -> None: ...

class hdfdir_archive(archive):
    def __init__(self, dirname: Optional[str] = None, dict: Optional[dict] = None,
                 cached: bool = True, serialized: bool = True,
                 permissions: Optional[int] = None,
                 protocol: Optional[int] = None, *args, **kwds) -> None: ...

class hdf_archive(archive):
    def __init__(self, filename: Optional[str] = None, dict: Optional[dict] = None,
                 cached: bool = True, serialized: bool = True,
                 protocol: Optional[int] = None, *args, **kwds) -> None: ...
```

### diskcache

When caching large datasets that should survive process restarts, `diskcache` shines with its optimized disk-backed storage using SQLite and the filesystem. It offers a range of eviction policies, a sharded `FanoutCache`, and persistent data structures like `Deque` and `Index`.

- **Backend**: SQLite for metadata, filesystem for data
- **Eviction Policies**: Configurable, including LRU and LFU
- **Persistence**: Data persists on disk between process runs
- **Concurrency**: Thread and process-safe 
- **Added Features**: Stampede prevention, throttling, optimized I/O
- **Best For**: Persistent caching for web apps and data pipelines

```pyi
### diskcache.pyi (Partial - Most important classes)

import typing
from typing import Callable, List, Dict, Any, IO, Optional, Tuple, Union

class Cache:
    def __init__(self, directory: Optional[str] = ..., timeout: float = ..., disk: Type[Disk] = ..., **settings: Any) -> None: ...
    @property
    def directory(self) -> str: ...
    @property
    def timeout(self) -> float: ...
    @property
    def disk(self) -> Disk: ...
    def set(self, key: Any, value: Any, expire: Optional[float] = ..., read: bool = ..., tag: Optional[str] = ..., retry: bool = ...) -> bool: ...
    def get(self, key: Any, default: Optional[Any] = ..., read: bool = ..., expire_time: bool = ..., tag: bool = ..., retry: bool = ...) -> Any: ...
    def delete(self, key: Any, retry: bool = ...) -> bool: ...
    def clear(self, retry: bool = ...) -> int: ...
    def volume(self) -> int: ...
    def check(self, fix: bool = ..., retry: bool = ...) -> List[warnings.WarningMessage]: ...
    def transact(self, retry: bool = ...) -> ContextManager[Callable]: ...
    def memoize(self, name: Optional[str] = ..., typed: bool = ..., expire: Optional[float] = ..., tag: Optional[str] = ..., ignore: typing.Iterable[str] = ...) -> Callable[[_F], _F]: ...
    def close(self) -> None: ...
    def volume(self) -> int: ...
    def stats(self, enable: bool = ..., reset: bool = ...) -> Tuple[int, int]: ...
    def volume(self) -> int: ...

class FanoutCache:
    def __init__(self, directory: Optional[str] = ..., shards: int = ..., timeout: float = ..., disk: Type[Disk] = ..., **settings: Any) -> None: ...
    @property
    def directory(self) -> str: ...
    def transact(self, retry: bool = ...) -> ContextManager[Callable]: ...
    def set(self, key: Any, value: Any, expire: Optional[float] = ..., read: bool = ..., tag: Optional[str] = ..., retry: bool = ...) -> bool: ...
    def get(self, key: Any, default: Optional[Any] = ..., read: bool = ..., expire_time: bool = ..., tag: bool = ..., retry: bool = ...) -> Any: ...
    def delete(self, key: Any, retry: bool = ...) -> bool: ...
    def clear(self, retry: bool = ...) -> int: ...
    def volume(self) -> int: ...
    def stats(self, enable: bool = ..., reset: bool = ...) -> Tuple[int, int]: ...
    def memoize(self, name: Optional[str] = ..., typed: bool = ..., expire: Optional[float] = ..., tag: Optional[str] = ..., ignore: typing.Iterable[str] = ...) -> Callable[[_F], _F]: ...
    def close(self) -> None: ...
    def volume(self) -> int: ...

class Disk:
    def __init__(self, directory: str, min_file_size: int = ..., pickle_protocol: int = ...) -> None: ...
    @property
    def min_file_size(self) -> int: ...
    @property
    def pickle_protocol(self) -> int: ...
    def put(self, key: Any) -> Tuple[Union[str, sqlite3.Binary, int, float], bool]: ...
    def get(self, key: Union[str, sqlite3.Binary, int, float], raw: bool) -> Any: ...
    def store(self, value: Any, read: bool, key: Constant = ...) -> Tuple[int, int, Optional[str], Optional[Union[str, sqlite3.Binary, int, float]]]: ...
    def fetch(self, mode: int, filename: Optional[str], value: Optional[Union[str, sqlite3.Binary, int, float]], read: bool) -> Any: ...

class JSONDisk(Disk):
    def __init__(self, directory: str, compress_level: int = ..., **kwargs: Any) -> None: ...
```

### joblib

Designed for scientific computing and ML workflows, `joblib` offers transparent disk-caching for functions, with optimizations for large NumPy arrays. Results are saved to disk and only re-computed when inputs change.

- **Backend**: Filesystem
- **Persistence**: Caches results to disk for costly computations
- **Memoization**: `@memory.cache` decorator for functions
- **Serialization**: Pickle-based with optional compression
- **Concurrency**: Process-safe file locking
- **Best For**: Caching ML models, features, large NumPy arrays

```pyi
### joblib.pyi (Partial - Most important classes and functions)

import typing
from typing import Callable, List, Dict, Any, IO, Optional, Tuple, Union

_F = TypeVar("_F", bound=Callable[..., Any])

class Memory:
    def __init__(self, location: Optional[str] = ..., backend: str = ..., verbose: int = ..., bytes_limit: Optional[Union[int, str]] = ..., mmap_mode: Optional[str] = ..., compress: Union[bool, int] = ..., backend_options: Optional[Dict[str, Any]] = ...) -> None: ...
    @property
    def location(self) -> str: ...
    @property
    def backend(self) -> str: ...
    @property
    def compress(self) -> Union[bool, int]: ...
    @property
    def verbose(self) -> int: ...
    def cache(self, func: Optional[_F] = ..., ignore: Optional[List[str]] = ..., verbose: Optional[int] = ..., mmap_mode: Optional[str] = ..., cache_validation_callback: Optional[Callable[[Dict[str, Any]], bool]] = ...) -> _F: ...
    def clear(self, warn: bool = ...) -> None: ...
    def eval(self, func: _F, *args: Any, **kwargs: Any) -> Any: ...
    def __call__(self, func: _F, *args: Any, **kwargs: Any) -> Any: ...

class Parallel:
    def __init__(self, n_jobs: Optional[int] = ..., backend: Optional[str] = ..., verbose: int = ..., timeout: Optional[float] = ..., pre_dispatch: Union[str, int] = ..., batch_size: Union[str, int] = ..., temp_folder: Optional[str] = ..., max_nbytes: Optional[Union[int, str]] = ..., mmap_mode: Optional[str] = ..., prefer: Optional[str] = ..., require: Optional[str] = ...) -> None: ...
    def __call__(self, iterable: typing.Iterable) -> List[Any]: ...
    def __enter__(self) -> "Parallel": ...
    def __exit__(self, exc_type: Optional[Type[BaseException]], exc_value: Optional[BaseException], traceback: Optional[TracebackType]) -> None: ...
    def submit(self, func: _F, *args: Any, **kwargs: Any) -> concurrent.futures.Future: ...
    def map(self, func: _F, *iterables: typing.Iterable, timeout: Optional[float] = ..., chunksize: int = ...) -> typing.Iterable[Any]: ...
    def dispatch_next(self) -> None: ...
    def print_progress(self) -> None: ...
    def clear(self) -> None: ...
    def __len__(self) -> int: ...

def delayed(function: _F) -> Callable[..., Tuple[Callable, tuple, dict]]: ...
def cpu_count(only_physical_cores: bool = ...) -> int: ...
def effective_n_jobs(n_jobs: int = ...) -> int: ...
def hash(obj: Any, hash_name: str = ..., coerce_mmap: bool = ...) -> str: ...
def dump(value: Any, filename: Union[str, PathLikeStr], compress: Union[bool, int] = ..., protocol: Optional[int] = ..., cache_size: Optional[int] = ...) -> List[str]: ...
def load(filename: Union[str, PathLikeStr], mmap_mode: Optional[str] = ...) -> Any: ...
def register_parallel_backend(name: str, factory: Callable[..., ParallelBackendBase], make_default: bool = ...) -> None: ...

#### aiocache

Built for async applications using the `asyncio` framework, `aiocache` enables non-blocking caching operations. It provides both in-memory (`SimpleMemoryCache`) and distributed options (`RedisCache`, `MemcachedCache`).

- **Backends**: In-memory, Redis, Memcached
- **Async Decorators**: `@cached`, `@cached_stampede`, `@multi_cached`  
- **Serialization**: Pluggable, defaults to JSON
- **Best For**: Asynchronous web frameworks (FastAPI, Sanic), distributed caching

```pyi
### aiocache.pyi (Partial - Most important classes and decorators)

import typing
from typing import Any, Callable, Coroutine, Dict, List, Optional, Tuple, Type, Union

_F = typing.TypeVar("_F", bound=Callable[..., Any])

class BaseCache:
    NAME: str
    def __init__(self, serializer: Optional[BaseSerializer] = ..., plugins: Optional[List[BasePlugin]] = ..., namespace: Optional[str] = ..., timeout: float = ..., ttl: Optional[int] = ...) -> None: ...
    @classmethod
    def parse_uri_path(cls, path: str) -> Dict[str, str]: ...
    async def add(self, key: str, value: Any, ttl: Optional[int] = ..., dumps_fn: Optional[Callable[[Any], bytes]] = ..., namespace: Optional[str] = ..., timeout: Optional[int] = ...) -> bool: ...
    async def get(self, key: str, default: Optional[Any] = ..., loads_fn: Optional[Callable[[bytes], Any]] = ..., namespace: Optional[str] = ..., timeout: Optional[int] = ...) -> Any: ...
    async def multi_get(self, keys: List[str], loads_fn: Optional[Callable[[bytes], Any]] = ..., namespace: Optional[str] = ..., timeout: Optional[int] = ...) -> List[Any]: ...
    async def set(self, key: str, value: Any, ttl: Optional[int] = ..., dumps_fn: Optional[Callable[[Any], bytes]] = ..., namespace: Optional[str] = ..., timeout: Optional[int] = ..., _cas_token: Optional[Any] = ...) -> bool: ...
    async def multi_set(self, pairs: List[Tuple[str, Any]], ttl: Optional[int] = ..., dumps_fn: Optional[Callable[[Any], bytes]] = ..., namespace: Optional[str] = ..., timeout: Optional[int] = ...) -> bool: ...
    async def delete(self, key: str, namespace: Optional[str] = ..., timeout: Optional[int] = ...) -> int: ...
    async def exists(self, key: str, namespace: Optional[str] = ..., timeout: Optional[int] = ...) -> bool: ...
    async def increment(self, key: str, delta: int = ..., namespace: Optional[str] = ..., timeout: Optional[int] = ...) -> int: ...
    async def expire(self, key: str, ttl: Optional[int] = ..., namespace: Optional[str] = ..., timeout: Optional[int] = ...) -> bool: ...
    async def clear(self, namespace: Optional[str] = ..., timeout: Optional[int] = ...) -> bool: ...
    async def close(self, timeout: Optional[int] = ...) -> bool: ...
    async def raw(self, command: str, *args: Any, encoding: Optional[str] = ..., timeout: Optional[int] = ..., **kwargs: Any) -> Any: ...

class SimpleMemoryCache(BaseCache):
    NAME: str
    def __init__(self, serializer: Optional[BaseSerializer] = ..., **kwargs: Any) -> None: ...

class RedisCache(BaseCache):
    NAME: str
    def __init__(self, serializer: Optional[BaseSerializer] = ..., **kwargs: Any) -> None: ...

class MemcachedCache(BaseCache):
    NAME: str
    def __init__(self, serializer: Optional[BaseSerializer] = ..., **kwargs: Any) -> None: ...

class BaseSerializer:
    DEFAULT_ENCODING: Optional[str]
    def __init__(self, *args: Any, encoding: Union[str, object] = ..., **kwargs: Any) -> None: ...
    def dumps(self, value: Any) -> bytes: ...
    def loads(self, value: bytes) -> Any: ...

class JsonSerializer(BaseSerializer):
    def dumps(self, value: Any) -> str: ...
    def loads(self, value: str) -> Any: ...

class PickleSerializer(BaseSerializer):
    def dumps(self, value: Any) -> bytes: ...
    def loads(self, value: bytes) -> Any: ...

class NullSerializer(BaseSerializer):
    def dumps(self, value: Any) -> Any: ...
    def loads(self, value: Any) -> Any: ...

class BasePlugin:
    async def pre_get(self, client: BaseCache, key: str, namespace: Optional[str], **kwargs: Any) -> None: ...
    async def post_get(self, client: BaseCache, key: str, namespace: Optional[str], rv: Any, **kwargs: Any) -> None: ...
    # ... (and similar methods for other cache operations)

_Cache = TypeVar("_Cache", bound=BaseCache)

def cached(
    ttl: Optional[int] = ...,
    key: Optional[str] = ...,
    key_builder: Optional[Callable[..., str]] = ...,
    namespace: Optional[str] = ...,
    cache: Union[Type[_Cache], str] = ...,
    serializer: Optional[BaseSerializer] = ...,
    plugins: Optional[List[BasePlugin]] = ...,
    alias: Optional[str] = ...,
    noself: bool = ...,
    skip_cache_func: Optional[Callable[[Any], bool]] = ...,
    **kwargs: Any
) -> Callable[[_F], _F]: ...

def cached_stampede(
    lease: int = ...,
    ttl: Optional[int] = ...,
    key: Optional[str] = ...,
    key_builder: Optional[Callable[..., str]] = ...,
    namespace: Optional[str] = ...,
    cache: Union[Type[_Cache], str] = ...,
    serializer: Optional[BaseSerializer] = ...,
    plugins: Optional[List[BasePlugin]] = ...,
    alias: Optional[str] = ...,
    noself: bool = ...,
    skip_cache_func: Optional[Callable[[Any], bool]] = ...,
    **kwargs: Any
) -> Callable[[_F], _F]: ...

def multi_cached(
    keys_from_attr: Optional[str] = ...,
    namespace: Optional[str] = ...,
    key_builder: Optional[Callable[..., str]] = ...,
    ttl: Optional[int] = ...,
    cache: Union[Type[_Cache], str] = ...,
    serializer: Optional[BaseSerializer] = ...,
    plugins: Optional[List[BasePlugin]] = ...,
    alias: Optional[str] = ...,
    skip_cache_func: Optional[Callable[[str, Any], bool]] = ...,
    **kwargs: Any
) -> Callable[[_F], _F]: ...
```

### Summary of type hints

```pyi
### Type Stubs for Caching Libraries

from typing import (
    Any,
    Callable,
    Dict,
    List,
    Optional,
    Type,
    Iterable,
    Tuple,
    Union,
    TypeVar,
    Generic,
)
import datetime
import logging
from typing_extensions import Literal  # For older Python versions
```

*   **Generics:** For generic types like `Cache`, `FIFOCache`, `LFUCache`, etc. (in `cachetools`, `cachebox`, `klepto`) , I've used `TypeVar` to represent the key (`KT`) and value (`VT`) types.  This gives better type hinting.
*   **`object` as Default:** Where the original code uses implicit dynamic typing (no explicit type) or uses an internal implementation detail, I've often used `object` as the most general type hint.  This avoids creating stubs for internal classes.
*   **`NotImplementedError` and `typing.NoReturn`:** I've used `typing.NoReturn` for methods that are *not implemented* in a base class.  This is more precise than raising `NotImplementedError`, which would imply that a subclass *should* implement it.
*   **`defaultdict`:**  I've handled cases where `defaultdict` is used internally, providing a default factory function.
*   **`_Conn`:**  This internal helper class from `aiocache` is fully defined as all its methods are based on exported API.
* **`_DISTUTILS_PATCH`, `VIRTUALENV_PATCH_FILE`, etc.:** These internal constants/classes, used for patching `distutils`, are omitted.
*   **`kwargs` Handling:** In many cases, I've used `**kwargs` to represent arbitrary keyword arguments, especially when those arguments are passed directly to another function (like `open` or a backend-specific constructor).
* **`functools`:** The return type of decorators (`lru_cache`, `cache`) is `Callable[[Callable[..., R]], Callable[..., R]]`.  I've used a `TypeVar` named `_R` to make the relationship between the input and output types clear.
* **`klepto` classes**: Added the API for the `archive`, `dir_archive`, `sql_archive`, `sqltable_archive` based on the source code.
* **`joblib` classes**: Added the classes in the public API, `Memory`, `MemorizedFunc`, `NotMemorizedFunc`.

#### Recommendations

The best caching library depends on your specific needs - whether you require persistence, have large datasets, need multi-process safety, or are using an async framework. By understanding the strengths of each library, you can make an informed choice to optimize your application's performance.

================
File: TODO.md
================
---
this_file: TODO.md
---

# TODO

## Phase 1

- [ ] Add more defensive programming with better input validation
  - Validate function arguments more thoroughly
  - Add type checking for critical parameters
  - Implement graceful fallbacks for invalid inputs
- [ ] Refactor complex functions in decorators.py to improve readability
  - Break down large functions into smaller, focused ones
  - Improve naming for better self-documentation
  - Add explanatory comments for complex logic
- [ ] Increase overall test coverage to 90%
  - [ ] Add more unit tests for edge cases
  - [ ] Implement integration tests for all backends
  - [ ] Add performance regression tests

## Medium Priority

- [ ] Add support for asynchronous cache operations in all engines
  - Implement async versions of all cache operations
  - Add async context managers
  - Ensure compatibility with asyncio event loops
- [ ] Implement a unified configuration system with environment variable support
  - Add support for loading config from environment variables
  - Create a configuration hierarchy (env vars > config files > defaults)
  - Add validation for configuration values
- [ ] Create a more intuitive API for cache management operations
  - Simplify the interface for common operations
  - Add helper functions for frequent use cases
  - Improve error messages and feedback
- [ ] Add decorator factories with more customizable options
  - Support for custom key generation functions
  - Add options for cache invalidation strategies
  - Implement conditional caching based on arguments
- [ ] Create simpler API for cache statistics and monitoring
  - Add hit/miss ratio tracking
  - Implement cache efficiency metrics
  - Add visualization tools for cache performance

## Performance Optimizations

- [ ] Optimize key generation for better performance
  - Implement faster hashing algorithms for keys
  - Add support for custom key generation functions
  - Optimize key storage and lookup
- [ ] Implement smart TTL handling with refreshing strategies
  - Add background refresh for frequently accessed items
  - Implement sliding TTL windows
  - Add support for TTL based on access patterns
- [ ] Add memory usage monitoring to prevent cache bloat
  - Implement memory usage tracking
  - Add automatic pruning based on memory pressure
  - Create alerts for excessive memory usage
- [ ] Optimize serialization/deserialization for common data types
  - Add specialized serializers for numpy arrays and pandas DataFrames
  - Implement compression for large objects
  - Add support for incremental serialization

## Compatibility and Integration

- [ ] Ensure compatibility with Python 3.12+
  - Test with latest Python versions
  - Update type hints to use latest typing features
  - Address any deprecation warnings
- [ ] Add support for integration with popular frameworks
  - Create adapters for Flask, FastAPI, Django
  - Add middleware for web frameworks
  - Implement integration examples
- [ ] Ensure compatibility with container environments
  - Test in Docker and Kubernetes environments
  - Add configuration for containerized deployments
  - Document best practices for containers

## Documentation and Examples

- [ ] Create a comprehensive API reference with examples
  - Document all public classes and functions
  - Add usage examples for each feature
  - Include performance considerations
- [ ] Add doctest examples in function docstrings
  - Add executable examples in docstrings
  - Ensure examples are tested in CI
  - Keep examples up-to-date with API changes
- [ ] Create tutorials for advanced use cases
  - Add step-by-step guides for common scenarios
  - Create examples for complex configurations
  - Document performance optimization strategies

================
File: VERSION.txt
================
v2.6.2



================================================================
End of Codebase
================================================================

================
File: twat-mp.txt
================
This file is a merged representation of a subset of the codebase, containing files not matching ignore patterns, combined into a single document by Repomix. The content has been processed where empty lines have been removed.

================================================================
File Summary
================================================================

Purpose:
--------
This file contains a packed representation of the entire repository's contents.
It is designed to be easily consumable by AI systems for analysis, code review,
or other automated processes.

File Format:
------------
The content is organized as follows:
1. This summary section
2. Repository information
3. Directory structure
4. Multiple file entries, each consisting of:
  a. A separator line (================)
  b. The file path (File: path/to/file)
  c. Another separator line
  d. The full contents of the file
  e. A blank line

Usage Guidelines:
-----------------
- This file should be treated as read-only. Any changes should be made to the
  original repository files, not this packed version.
- When processing this file, use the file path to distinguish
  between different files in the repository.
- Be aware that this file may contain sensitive information. Handle it with
  the same level of security as you would the original repository.

Notes:
------
- Some files may have been excluded based on .gitignore rules and Repomix's configuration
- Binary files are not included in this packed representation. Please refer to the Repository Structure section for a complete list of file paths, including binary files
- Files matching these patterns are excluded: .specstory/**/*.md, .venv/**, _private/**, CLEANUP.txt, **/*.json, *.lock
- Files matching patterns in .gitignore are excluded
- Files matching default ignore patterns are excluded
- Empty lines have been removed from all files

Additional Info:
----------------

================================================================
Directory Structure
================================================================
.cursor/
  rules/
    0project.mdc
    cleanup.mdc
    filetree.mdc
    quality.mdc
.github/
  workflows/
    push.yml
    release.yml
docs/
  architecture.md
src/
  twat_mp/
    __init__.py
    async_mp.py
    mp.py
tests/
  test_async_mp.py
  test_benchmark.py
  test_twat_mp.py
.gitignore
.pre-commit-config.yaml
API_REFERENCE.md
CHANGELOG.md
cleanup.py
LICENSE
LOG.md
pyproject.toml
README.md
TODO.md
VERSION.txt

================================================================
Files
================================================================

================
File: .cursor/rules/0project.mdc
================
---
description: About this project
globs:
---
# About this project

`twat-fs` is a file system utility library focused on robust and extensible file upload capabilities with multiple provider support. It provides:

- Multi-provider upload system with smart fallback (catbox.moe default, plus Dropbox, S3, etc.)
- Automatic retry for temporary failures, fallback for permanent ones
- URL validation and clean developer experience with type hints
- Simple CLI: `python -m twat_fs upload_file path/to/file.txt`
- Easy installation: `uv pip install twat-fs` (basic) or `uv pip install 'twat-fs[all,dev]'` (all features)

## Development Notes
- Uses `uv` for Python package management
- Quality tools: ruff, mypy, pytest
- Clear provider protocol for adding new storage backends
- Strong typing and runtime checks throughout

================
File: .cursor/rules/cleanup.mdc
================
---
description: Run `cleanup.py` script before and after changes
globs:
---
Before you do any changes or if I say "cleanup", run the `cleanup.py update` script in the main folder. Analyze the results, describe recent changes in @LOG.md and edit @TODO.md to update priorities and plan next changes. PERFORM THE CHANGES, then run the `cleanup.py status` script and react to the results.

When you edit @TODO.md, lead in lines with empty GFM checkboxes if things aren't done (`- [ ] `) vs. filled (`- [x] `) if done.

================
File: .cursor/rules/filetree.mdc
================
---
description: File tree of the project
globs: 
---
[ 928]  .
├── [  96]  .cursor
│   └── [ 224]  rules
│       ├── [ 821]  0project.mdc
│       ├── [ 515]  cleanup.mdc
│       ├── [1.9K]  filetree.mdc
│       └── [2.0K]  quality.mdc
├── [ 128]  .github
│   └── [ 128]  workflows
│       ├── [2.9K]  push.yml
│       └── [1.4K]  release.yml
├── [3.0K]  .gitignore
├── [ 470]  .pre-commit-config.yaml
├── [  96]  .specstory
│   └── [ 320]  history
│       ├── [2.7K]  .what-is-this.md
│       ├── [ 58K]  2025-03-04_03-11-implementing-todo-md-plan.md
│       ├── [ 69K]  2025-03-04_03-23-codebase-analysis-and-todo-list-creation.md
│       ├── [ 15K]  2025-03-04_06-17-fixing-hatch-configuration-error.md
│       ├── [103K]  2025-03-04_07-16-implementing-todo-md-phases-1-and-2.md
│       ├── [ 644]  2025-03-04_07-54-untitled.md
│       ├── [1.2K]  2025-03-04_07-59-project-maintenance-and-documentation-update.md
│       └── [134K]  2025-03-04_08-26-ruff-linting-errors-and-fixes.md
├── [ 288]  .venv
├── [6.0K]  API_REFERENCE.md
├── [3.8K]  CHANGELOG.md
├── [ 939]  CLEANUP.txt
├── [1.0K]  LICENSE
├── [1.3K]  LOG.md
├── [ 13K]  README.md
├── [ 78K]  REPO_CONTENT.txt
├── [2.2K]  TODO.md
├── [   7]  VERSION.txt
├── [ 13K]  cleanup.py
├── [ 160]  dist
├── [  96]  docs
│   └── [6.4K]  architecture.md
├── [5.3K]  pyproject.toml
├── [ 128]  src
│   └── [ 256]  twat_mp
│       ├── [ 572]  __init__.py
│       ├── [ 17K]  async_mp.py
│       └── [ 25K]  mp.py
└── [ 224]  tests
    ├── [2.8K]  test_async_mp.py
    ├── [6.3K]  test_benchmark.py
    └── [ 13K]  test_twat_mp.py

13 directories, 34 files

================
File: .cursor/rules/quality.mdc
================
---
description: Quality
globs:
---
- **Verify Information**: Always verify information before presenting it. Do not make assumptions or speculate without clear evidence.
- **No Apologies**: Never use apologies.
- **No Whitespace Suggestions**: Don't suggest whitespace changes.
- **No Inventions**: Don't invent major changes other than what's explicitly requested.
- **No Unnecessary Confirmations**: Don't ask for confirmation of information already provided in the context.
- **Preserve Existing Code**: Don't remove unrelated code or functionalities. Pay attention to preserving existing structures.
- **No Implementation Checks**: Don't ask the user to verify implementations that are visible in the provided context.
- **No Unnecessary Updates**: Don't suggest updates or changes to files when there are no actual modifications needed.
- **No Current Implementation**: Don't show or discuss the current implementation unless specifically requested.
- **Use Explicit Variable Names**: Prefer descriptive, explicit variable names over short, ambiguous ones to enhance code readability.
- **Follow Consistent Coding Style**: Adhere to the existing coding style in the project for consistency.
- **Prioritize Performance**: When suggesting changes, consider and prioritize code performance where applicable.
- **Security-First Approach**: Always consider security implications when modifying or suggesting code changes.
- **Test Coverage**: Suggest or include appropriate unit tests for new or modified code.
- **Error Handling**: Implement robust error handling and logging where necessary.
- **Modular Design**: Encourage modular design principles to improve code maintainability and reusability.
- **Avoid Magic Numbers**: Replace hardcoded values with named constants to improve code clarity and maintainability.
- **Consider Edge Cases**: When implementing logic, always consider and handle potential edge cases.
- **Use Assertions**: Include assertions wherever possible to validate assumptions and catch potential errors early.

================
File: .github/workflows/push.yml
================
name: Build & Test
on:
  push:
    branches: [main]
    tags-ignore: ["v*"]
  pull_request:
    branches: [main]
  workflow_dispatch:
permissions:
  contents: write
  id-token: write
# Ensure that only one run per branch/commit is active at once.
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
jobs:
  # === QUALITY JOB: Lint and format checks ===
  quality:
    name: Code Quality
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Run Ruff lint
        uses: astral-sh/ruff-action@v3
        with:
          version: "latest"
          args: "check --output-format=github"
      - name: Run Ruff Format
        uses: astral-sh/ruff-action@v3
        with:
          version: "latest"
          args: "format --check --respect-gitignore"
  # === TEST JOB: Run tests ===
  test:
    name: Run Tests
    needs: quality
    strategy:
      matrix:
        python-version: ["3.12"]
        os: [ubuntu-latest]
      fail-fast: true
    runs-on: ${{ matrix.os }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install UV
        uses: astral-sh/setup-uv@v5
        with:
          version: "latest"
          python-version: ${{ matrix.python-version }}
          enable-cache: true
          cache-suffix: ${{ matrix.os }}-${{ matrix.python-version }}
      - name: Install test dependencies
        run: |
          uv pip install --system --upgrade pip
          uv pip install --system ".[test]"
      - name: Run tests with Pytest
        run: uv run pytest -n auto --maxfail=1 --disable-warnings --cov-report=xml --cov-config=pyproject.toml --cov=src/twat_mp --cov=tests tests/
      - name: Upload coverage report
        uses: actions/upload-artifact@v4
        with:
          name: coverage-${{ matrix.python-version }}-${{ matrix.os }}
          path: coverage.xml
  # === BUILD JOB: Create distribution artifacts ===
  build:
    name: Build Distribution
    needs: test
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install UV
        uses: astral-sh/setup-uv@v5
        with:
          version: "latest"
          python-version: "3.12"
          enable-cache: true
      - name: Install build tools
        run: uv pip install build hatchling hatch-vcs
      - name: Build distributions
        run: uv run python -m build --outdir dist
      - name: Upload distribution artifacts
        uses: actions/upload-artifact@v4
        with:
          name: dist-files
          path: dist/
          retention-days: 5

================
File: .github/workflows/release.yml
================
name: Release
on:
  push:
    tags: ["v*"]
permissions:
  contents: write
  id-token: write
jobs:
  release:
    name: Release to PyPI
    runs-on: ubuntu-latest
    environment:
      name: pypi
      url: https://pypi.org/p/twat-mp
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install UV
        uses: astral-sh/setup-uv@v5
        with:
          version: "latest"
          python-version: "3.12"
          enable-cache: true
      - name: Install build tools
        run: uv pip install build hatchling hatch-vcs
      - name: Build distributions
        run: uv run python -m build --outdir dist
      - name: Verify distribution files
        run: |
          ls -la dist/
          test -n "$(find dist -name '*.whl')" || (echo "Wheel file missing" && exit 1)
          test -n "$(find dist -name '*.tar.gz')" || (echo "Source distribution missing" && exit 1)
      - name: Publish to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          password: ${{ secrets.PYPI_TOKEN }}
      - name: Create GitHub Release
        uses: softprops/action-gh-release@v1
        with:
          files: dist/*
          generate_release_notes: true
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

================
File: docs/architecture.md
================
# twat-mp Architecture

This document explains the architecture of the `twat-mp` package using simple diagrams.

## Component Overview

```
+---------------------------+
|       twat-mp Package     |
+---------------------------+
           |
           |
           v
+---------------------------+
|      Core Components      |
+---------------------------+
           |
           +----------------+----------------+
           |                |                |
           v                v                v
+------------------+ +---------------+ +---------------+
|  Process-based   | |  Thread-based | |  Async-based  |
|  Parallelism     | |  Parallelism  | |  Parallelism  |
+------------------+ +---------------+ +---------------+
|                  | |               | |               |
| - MultiPool      | | - ThreadPool  | | - AsyncMultiPool |
| - ProcessPool    | |               | |               |
| - pmap, imap,    | |               | | - apmap       |
|   amap decorators| |               | |               |
+------------------+ +---------------+ +---------------+
           |                |                |
           |                |                |
           v                v                v
+---------------------------+---------------------------+
|                 Underlying Libraries                  |
+---------------------------+---------------------------+
|                                                       |
|  - pathos (ProcessPool, ThreadPool)                   |
|  - aiomultiprocess (AsyncMultiPool)                   |
|                                                       |
+-------------------------------------------------------+
```

## Execution Flow

### Process/Thread Pool Execution Flow

```
+-------------+     +-------------+     +----------------+
| User Code   |     | Pool        |     | Worker         |
| with        |---->| Creation    |---->| Processes      |
| ProcessPool |     | (Context)   |     | or Threads     |
+-------------+     +-------------+     +----------------+
      |                   |                    |
      |                   |                    |
      v                   v                    v
+-------------+     +-------------+     +----------------+
| Function    |     | Task        |     | Parallel       |
| to Execute  |---->| Distribution|---->| Execution      |
+-------------+     +-------------+     +----------------+
                          |                    |
                          |                    |
                          v                    v
                    +-------------+     +----------------+
                    | Result      |<----| Results        |
                    | Collection  |     | from Workers   |
                    +-------------+     +----------------+
                          |
                          |
                          v
                    +-------------+
                    | Return      |
                    | to User     |
                    +-------------+
```

### Async Pool Execution Flow

```
+-------------+     +-------------+     +----------------+
| Async       |     | AsyncMulti  |     | Worker         |
| User Code   |---->| Pool        |---->| Processes      |
+-------------+     | Creation    |     |                |
      |             +-------------+     +----------------+
      |                   |                    |
      v                   v                    v
+-------------+     +-------------+     +----------------+
| Async       |     | Task        |     | Parallel       |
| Function    |---->| Distribution|---->| Execution of   |
+-------------+     +-------------+     | Async Functions|
                          |             +----------------+
                          |                    |
                          v                    v
                    +-------------+     +----------------+
                    | Await       |<----| Async Results  |
                    | Results     |     | from Workers   |
                    +-------------+     +----------------+
                          |
                          |
                          v
                    +-------------+
                    | Return      |
                    | to User     |
                    +-------------+
```

## Decorator Pattern

```
+-------------+     +-------------+     +----------------+
| Function    |     | Decorator   |     | Wrapped        |
| Definition  |---->| Application |---->| Function       |
| @pmap       |     | (mmap)      |     |                |
+-------------+     +-------------+     +----------------+
                          |                    |
                          |                    |
                          v                    v
                    +-------------+     +----------------+
                    | Function    |     | MultiPool      |
                    | Call with   |---->| Creation &     |
                    | Iterable    |     | Management     |
                    +-------------+     +----------------+
                                               |
                                               |
                                               v
                                        +----------------+
                                        | Parallel       |
                                        | Execution &    |
                                        | Result Return  |
                                        +----------------+
```

## Class Hierarchy

```
                  +-------------+
                  |  MultiPool  |
                  +-------------+
                        ^
                        |
          +-------------+-------------+
          |                           |
+-----------------+         +-----------------+
|  ProcessPool    |         |   ThreadPool    |
+-----------------+         +-----------------+


+------------------+
| AsyncMultiPool   |
+------------------+
```

## Decorator Relationships

```
                  +-------------+
                  |    mmap     |
                  | (factory)   |
                  +-------------+
                        |
                        v
          +-------------+-------------+-------------+
          |             |             |             |
+-----------------+  +------+  +------+  +---------+
|      pmap       |  | imap |  | amap |  |  apmap  |
| (eager eval)    |  | (lazy)|  |(async)|  |(async) |
+-----------------+  +------+  +------+  +---------+
```

================
File: src/twat_mp/__init__.py
================


================
File: src/twat_mp/async_mp.py
================
logger = logging.getLogger(__name__)
    class AioPool:  # type: ignore
T = TypeVar("T")
U = TypeVar("U")
V = TypeVar("V")
def _check_aiomultiprocess() -> None:
        raise ImportError(
class AsyncMultiPool:
    def __init__(
        _check_aiomultiprocess()
    async def __aenter__(self) -> "AsyncMultiPool":
                self.pool = aiomultiprocess.Pool(
                logger.debug(f"Created AsyncMultiPool with {self.processes} processes")
                logger.error(f"Failed to create AsyncMultiPool: {e}")
                raise RuntimeError(f"Failed to create AsyncMultiPool: {e}") from e
    async def __aexit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None:
            logger.debug("No pool to clean up in __aexit__")
            logger.warning("Cleanup already attempted, skipping")
            logger.warning(
            logger.debug("Attempting graceful pool shutdown with close()")
            self.pool.close()
            await self.pool.join()
            logger.debug("Pool gracefully closed and joined")
            logger.error(f"Error during graceful pool shutdown: {e}")
            logger.debug(f"Traceback: {traceback.format_exc()}")
                logger.debug("Attempting forceful pool termination")
                self.pool.terminate()
                logger.debug("Pool forcefully terminated and joined")
                logger.error(f"Error during forceful pool termination: {e2}")
            logger.debug("Clearing pool reference")
            logger.error(error_msg)
            raise RuntimeError(error_msg) from cleanup_error
    async def map(
            raise RuntimeError("Pool not initialized. Use 'async with' statement.")
            return await self.pool.map(func, iterable)
            logger.error(f"Error during parallel map operation: {e}")
            raise RuntimeError(f"Error during parallel map operation: {e}") from e
    async def starmap(
            return await self.pool.starmap(func, iterable)
            logger.error(f"Error during parallel starmap operation: {e}")
            raise RuntimeError(f"Error during parallel starmap operation: {e}") from e
    async def imap(
            async for result in self.pool.imap(func, iterable):
            logger.error(f"Error during parallel imap operation: {e}")
            raise RuntimeError(f"Error during parallel imap operation: {e}") from e
def apmap(
    @wraps(func)
    async def wrapper(iterable: Iterable[T]) -> list[U]:
            async with AsyncMultiPool() as pool:
                return await pool.map(func, iterable)
            logger.error(f"Error in apmap decorator: {e}")
            raise RuntimeError(f"Error in parallel processing: {e}") from e

================
File: src/twat_mp/mp.py
================
logger = logging.getLogger(__name__)
T = TypeVar("T")
U = TypeVar("U")
DEBUG_MODE = os.environ.get("TWAT_MP_DEBUG", "0").lower() in ("1", "true", "yes", "on")
class WorkerError(Exception):
    def __init__(
                f": {type(original_exception).__name__}: {original_exception}"
        super().__init__(detailed_message)
def set_debug_mode(enabled: bool = True) -> None:
        logging.basicConfig(
            handlers=[logging.StreamHandler(sys.stdout)],
        logger.setLevel(logging.DEBUG)
        logger.debug("Debug mode enabled for twat_mp")
        logger.setLevel(logging.INFO)
        logger.info("Debug mode disabled for twat_mp")
def _worker_wrapper(func: Callable[[T], U], item: T, worker_id: int | None = None) -> U:
        return func(item)
        tb_str = traceback.format_exc()
        logger.error(
        logger.debug(f"Traceback: {tb_str}")
        raise WorkerError(
class MultiPool:
        self.nodes: int = nodes if nodes is not None else mp.cpu_count()  # type: ignore
            logger.debug(
    def __enter__(self) -> PathosPool:
            logger.debug(f"Creating pool with {self.nodes} nodes")
            self.pool = self.pool_class(nodes=self.nodes)
                logger.debug(f"Pool created successfully: {self.pool}")
                logger.error(msg)
                raise RuntimeError(msg)
            def enhanced_map(func: Callable[[T], U], iterable: Iterator[T]) -> Any:
                def wrapped_func(item, idx=None):
                    return _worker_wrapper(func, item, idx)
                    return original_map(wrapped_func, iterable)
                    if isinstance(e, WorkerError):
                    logger.error(f"Error during parallel map operation: {e}")
                        logger.debug(f"Traceback: {traceback.format_exc()}")
                    raise RuntimeError(
            logger.error(error_msg)
            raise RuntimeError(error_msg) from e
    def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> Literal[False]:
                    self.pool.terminate()
                    self.pool.join()
                        logger.debug("Closing pool gracefully")
                    self.pool.close()
                        logger.debug("Pool closed and joined successfully")
                        logger.debug("Pool terminated and joined after cleanup error")
                        logger.debug(f"Failed to terminate pool: {e2}")
                        logger.debug("Clearing pool resources")
                    self.pool.clear()
                        logger.debug("Pool resources cleared")
                        logger.debug(f"Error clearing pool resources: {e}")
                    logger.debug("Pool reference set to None")
            logger.debug("No pool to clean up")
class ProcessPool(MultiPool):
    def __init__(self, nodes: int | None = None, debug: bool | None = None) -> None:
        super().__init__(pool_class=PathosProcessPool, nodes=nodes, debug=debug)
class ThreadPool(MultiPool):
        super().__init__(pool_class=PathosThreadPool, nodes=nodes, debug=debug)
def mmap(
        raise ValueError(
    def decorator(func: Callable[[T], U]) -> Callable[[Iterator[T]], Iterator[U]]:
        @wraps(func)
        def wrapper(iterable: Iterator[T], *args: Any, **kwargs: Any) -> Any:
                    f"Executing {func.__name__} with {how} on {type(iterable).__name__}"
                with MultiPool(debug=use_debug) as pool:
                        mapping_method = getattr(pool, how)
                            logger.debug(f"Using pool method: {how}")
                        raise ValueError(error_msg) from err
                            logger.debug(f"Starting parallel execution with {how}")
                        result = mapping_method(func, iterable)
                                logger.debug("Getting results from async operation")
                            result = result.get()
                            logger.debug("Parallel execution completed successfully")
                                isinstance(e, WorkerError)
                            logger.debug("KeyboardInterrupt detected during execution")
def pmap(func: Callable[[T], U]) -> Callable[[Iterator[T]], Iterator[U]]:
    return mmap("map")(func)
def imap(func: Callable[[T], U]) -> Callable[[Iterator[T]], Iterator[U]]:
    return mmap("imap")(func)
def amap(func: Callable[[T], U]) -> Callable[[Iterator[T]], Any]:
    return mmap("amap", get_result=True)(func)

================
File: tests/test_async_mp.py
================
async def async_double(x: int) -> int:
    await asyncio.sleep(0.01)  # Simulate some async work
async def async_raise_error(x: Any) -> Any:
    raise ValueError(f"Error processing {x}")
async def test_async_multi_pool_map() -> None:
    async with AsyncMultiPool() as pool:
        results = await pool.map(async_double, range(5))
async def test_async_multi_pool_empty() -> None:
        results = await pool.map(async_double, [])
async def test_async_multi_pool_error() -> None:
    with pytest.raises(ValueError):
            await pool.map(async_raise_error, range(5))
async def test_async_multi_pool_imap() -> None:
        async for result in pool.imap(async_double, range(5)):
            results.append(result)
async def test_async_multi_pool_starmap() -> None:
    async def async_sum(*args: int) -> int:
        await asyncio.sleep(0.01)
        return sum(args)
        results = await pool.starmap(async_sum, [(1, 2), (3, 4), (5, 6)])
async def test_apmap_decorator() -> None:
    async def double(x: int) -> int:
    results = await double(range(5))
async def test_pool_not_initialized() -> None:
    pool = AsyncMultiPool()
    with pytest.raises(RuntimeError, match="Pool not initialized"):
        await pool.map(async_double, range(5))
async def test_pool_cleanup() -> None:

================
File: tests/test_benchmark.py
================
def _compute_intensive(x: int) -> int:
    for _ in range(1000):  # Simulate CPU-intensive work
def _io_intensive(x: int) -> int:
    time.sleep(0.001)  # Simulate I/O wait
def generate_data(size: int) -> list[int]:
    return list(range(size))
def small_data() -> list[int]:
    return generate_data(100)
def medium_data() -> list[int]:
    return generate_data(1000)
def large_data() -> list[int]:
    return generate_data(10000)
def run_parallel_operation(
    parallel_func = parallel_impl(func)
    return list(parallel_func(data))
class TestComputeIntensiveBenchmarks:
    def test_sequential_vs_process_pool(self, benchmark, medium_data):
        def sequential() -> list[int]:
            return list(map(_compute_intensive, medium_data))
        def parallel() -> list[int]:
            with ProcessPool() as pool:
                return list(pool.map(_compute_intensive, medium_data))
        result = sequential()  # Run once to get result
        assert parallel() == result  # Verify results match
        def run_both() -> tuple[list[int], list[int]]:
            return sequential(), parallel()
        benchmark(run_both)
    @pytest.mark.parametrize("data_size", [100, 1000, 10000])
    def test_parallel_implementations(self, benchmark, data_size):
        data = generate_data(data_size)
        def process_map(
            return mmap(how="map")(f)
        def thread_map(
            def wrapper(iterable: Any) -> Iterator[Any]:
                with ThreadPool() as pool:
                    return pool.map(f, iterable)
            "amap": lambda f: amap(f),
            "imap": lambda f: imap(f),
            "pmap": lambda f: pmap(f),
        reference_result = run_parallel_operation(
        for name, impl in implementations.items():
            result = run_parallel_operation(_compute_intensive, data, impl)
        def run_all() -> dict[str, list[int]]:
                name: run_parallel_operation(_compute_intensive, data, impl)
                for name, impl in implementations.items()
        benchmark(run_all)
class TestIOIntensiveBenchmarks:
    def test_thread_vs_process_pool(self, benchmark, medium_data):
        def process_pool() -> list[int]:
                return list(pool.map(_io_intensive, medium_data))
        def thread_pool() -> list[int]:
        result = process_pool()  # Run once to get result
        assert thread_pool() == result  # Verify results match
            return process_pool(), thread_pool()
class TestScalabilityBenchmarks:
    @pytest.mark.parametrize("nodes", [2, 4, 8, 16])
    def test_worker_scaling(self, benchmark, medium_data, nodes):
        def run_with_workers() -> list[int]:
            with ProcessPool(nodes=nodes) as pool:
        benchmark(run_with_workers)
class TestCompositionBenchmarks:
    def test_chained_operations(self, benchmark, medium_data):
        def sequential_chain() -> list[int]:
            return [_io_intensive(_compute_intensive(x)) for x in medium_data]
        def parallel_chain() -> list[int]:
            compute = amap(_compute_intensive)
            io_op = amap(_io_intensive)
            return list(io_op(compute(medium_data)))
        result = sequential_chain()  # Run once to get result
        assert parallel_chain() == result  # Verify results match
            return sequential_chain(), parallel_chain()

================
File: tests/test_twat_mp.py
================
T = TypeVar("T")
U = TypeVar("U")
def test_version():
def _square(x: int) -> int:
def _subs(x: int) -> int:
isquare = amap(_square)
isubs = amap(_subs)
def test_process_pool_context():
    with ProcessPool() as pool:
        result = list(pool.map(_square, iter(range(5))))
def test_thread_pool_context():
    with ThreadPool() as pool:
def test_amap_decorator():
    result = isquare(iter(range(5)))
def test_pmap_decorator():
    def square(x: int) -> int:
    result = list(square(iter(range(5))))
def test_imap_decorator():
    iterator = square(iter(range(3)))
    assert next(iterator) == 0
    assert next(iterator) == 1
    assert next(iterator) == 4
def test_composed_operations():
    result = list(isubs(isquare(iter(range(5)))))
def test_pool_nodes_specification():
    with ProcessPool(nodes=TEST_PROCESS_POOL_SIZE) as pool:
    with ThreadPool(nodes=TEST_THREAD_POOL_SIZE) as pool:
def test_parallel_vs_sequential_performance():
    def slow_square(x: int) -> int:
        time.sleep(0.01)  # Simulate CPU work
    start_time = time.time()
    sequential_result = [slow_square(x) for x in range(20)]
    sequential_time = time.time() - start_time
        parallel_result = list(pool.map(slow_square, iter(range(20))))
    parallel_time = time.time() - start_time
    print(f"Sequential: {sequential_time:.4f}s, Parallel: {parallel_time:.4f}s")
def test_mmap_decorator_variants():
    map_decorator = mmap("map")
    def square1(x: int) -> int:
    assert list(square1(iter(range(5)))) == [0, 1, 4, 9, 16]
    imap_decorator = mmap("imap")
    def square2(x: int) -> int:
    assert list(square2(iter(range(5)))) == [0, 1, 4, 9, 16]
    amap_decorator = mmap("amap", get_result=True)
    def square3(x: int) -> int:
    assert square3(iter(range(5))) == [0, 1, 4, 9, 16]
def test_empty_iterable():
        result = list(pool.map(_square, iter([])))
    assert list(square(iter([]))) == []
    def square_imap(x: int) -> int:
    assert list(square_imap(iter([]))) == []
    def square_amap(x: int) -> int:
    assert square_amap(iter([])) == []
def test_error_propagation():
    def error_func(x: int) -> int:
            raise ValueError("Test error")
    with pytest.raises(WorkerError) as excinfo:
            list(pool.map(error_func, iter(range(5))))
    assert "ValueError" in str(excinfo.value)
    assert "Test error" in str(excinfo.value)
    assert "3" in str(excinfo.value)  # Check that the input item is mentioned
    def error_map(x: int) -> int:
            raise ValueError("Test error in pmap")
    with pytest.raises(RuntimeError, match="Failed to create or use pool"):
        list(error_map(iter(range(5))))
    def error_imap(x: int) -> int:
            raise ValueError("Test error in imap")
    with pytest.raises(ValueError):
        list(error_imap(iter(range(5))))
    def error_amap(x: int) -> int:
            raise ValueError("Test error in amap")
        error_amap(iter(range(5)))
def test_debug_mode():
    with patch("logging.Logger.debug") as mock_debug:
        set_debug_mode(True)
            list(pool.map(_square, iter(range(3))))
        set_debug_mode(False)
def test_invalid_mapping_method():
    with pytest.raises(ValueError, match="Invalid mapping method"):
        mmap("invalid_method")
    with patch("twat_mp.mp.MultiPool.__enter__") as mock_enter:
        mock_pool = MagicMock()
        delattr(mock_pool, valid_method)  # Remove the valid method from the mock
        decorator = mmap(valid_method)
        def test_func(x: int) -> int:
            test_func(iter(range(5)))
def test_pool_creation_failure():
    with patch(
        side_effect=RuntimeError("Test pool creation error"),
        with pytest.raises(RuntimeError, match="Test pool creation error"):
            with ProcessPool():
def test_resource_cleanup_after_exception():
    class TestError(Exception):
    with patch("twat_mp.mp.MultiPool.__exit__") as mock_exit:
                raise TestError("Test exception")
def test_keyboard_interrupt_handling():
    with patch("pathos.pools.ProcessPool.terminate") as mock_terminate:
        with patch("pathos.pools.ProcessPool.join") as mock_join:
                    raise KeyboardInterrupt()
def test_large_data_handling():
    large_data = list(range(1000))
        result = list(pool.map(_square, iter(large_data)))
        assert len(result) == 1000
def test_nested_pools():
    def nested_pool_func(x: int) -> List[int]:
        with ThreadPool(nodes=2) as inner_pool:
            return list(inner_pool.map(_square, iter(range(x + 1))))
    with ProcessPool(nodes=2) as outer_pool:
        results = list(outer_pool.map(nested_pool_func, iter(range(3))))
def test_pool_reuse_failure():
    with ProcessPool() as p:
        list(p.map(_square, iter(range(5))))
            result = list(pool.map(_square, iter(range(3))))
            print(f"Pool raised exception after context exit: {type(e).__name__}: {e}")
def test_custom_exception_handling():
    class CustomError(Exception):
    def raise_custom_error(x: int) -> int:
            raise CustomError(f"Value {x} is too large")
            list(pool.map(raise_custom_error, iter(range(5))))
    assert "CustomError" in str(excinfo.value)
    assert "Value 3 is too large" in str(excinfo.value)
    def decorated_error_func(x: int) -> int:
            raise CustomError(f"Decorated value {x} is too large")
        list(decorated_error_func(iter(range(5))))

================
File: .gitignore
================
*_autogen/
.DS_Store
__version__.py
__pycache__/
_Chutzpah*
_deps
_NCrunch_*
_pkginfo.txt
_Pvt_Extensions
_ReSharper*/
_TeamCity*
_UpgradeReport_Files/
!?*.[Cc]ache/
!.axoCover/settings.json
!.vscode/extensions.json
!.vscode/launch.json
!.vscode/settings.json
!.vscode/tasks.json
!**/[Pp]ackages/build/
!Directory.Build.rsp
.*crunch*.local.xml
.axoCover/*
.builds
.cr/personal
.fake/
.history/
.ionide/
.localhistory/
.mfractor/
.ntvs_analysis.dat
.paket/paket.exe
.sass-cache/
.vs/
.vscode
.vscode/*
.vshistory/
[Aa][Rr][Mm]/
[Aa][Rr][Mm]64/
[Bb]in/
[Bb]uild[Ll]og.*
[Dd]ebug/
[Dd]ebugPS/
[Dd]ebugPublic/
[Ee]xpress/
[Ll]og/
[Ll]ogs/
[Oo]bj/
[Rr]elease/
[Rr]eleasePS/
[Rr]eleases/
[Tt]est[Rr]esult*/
[Ww][Ii][Nn]32/
*_h.h
*_i.c
*_p.c
*_wpftmp.csproj
*- [Bb]ackup ([0-9]).rdl
*- [Bb]ackup ([0-9][0-9]).rdl
*- [Bb]ackup.rdl
*.[Cc]ache
*.[Pp]ublish.xml
*.[Rr]e[Ss]harper
*.a
*.app
*.appx
*.appxbundle
*.appxupload
*.aps
*.azurePubxml
*.bim_*.settings
*.bim.layout
*.binlog
*.btm.cs
*.btp.cs
*.build.csdef
*.cab
*.cachefile
*.code-workspace
*.coverage
*.coveragexml
*.d
*.dbmdl
*.dbproj.schemaview
*.dll
*.dotCover
*.DotSettings.user
*.dsp
*.dsw
*.dylib
*.e2e
*.exe
*.gch
*.GhostDoc.xml
*.gpState
*.ilk
*.iobj
*.ipdb
*.jfm
*.jmconfig
*.la
*.lai
*.ldf
*.lib
*.lo
*.log
*.mdf
*.meta
*.mm.*
*.mod
*.msi
*.msix
*.msm
*.msp
*.ncb
*.ndf
*.nuget.props
*.nuget.targets
*.nupkg
*.nvuser
*.o
*.obj
*.odx.cs
*.opendb
*.opensdf
*.opt
*.out
*.pch
*.pdb
*.pfx
*.pgc
*.pgd
*.pidb
*.plg
*.psess
*.publishproj
*.publishsettings
*.pubxml
*.pyc
*.rdl.data
*.rptproj.bak
*.rptproj.rsuser
*.rsp
*.rsuser
*.sap
*.sbr
*.scc
*.sdf
*.sln.docstates
*.sln.iml
*.slo
*.smod
*.snupkg
*.so
*.suo
*.svclog
*.tlb
*.tlh
*.tli
*.tlog
*.tmp
*.tmp_proj
*.tss
*.user
*.userosscache
*.userprefs
*.vbp
*.vbw
*.VC.db
*.VC.VC.opendb
*.VisualState.xml
*.vsp
*.vspscc
*.vspx
*.vssscc
*.xsd.cs
**/[Pp]ackages/*
**/*.DesktopClient/GeneratedArtifacts
**/*.DesktopClient/ModelManifest.xml
**/*.HTMLClient/GeneratedArtifacts
**/*.Server/GeneratedArtifacts
**/*.Server/ModelManifest.xml
*~
~$*
$tf/
AppPackages/
artifacts/
ASALocalRun/
AutoTest.Net/
Backup*/
BenchmarkDotNet.Artifacts/
bld/
BundleArtifacts/
ClientBin/
cmake_install.cmake
CMakeCache.txt
CMakeFiles
CMakeLists.txt.user
CMakeScripts
CMakeUserPresets.json
compile_commands.json
coverage*.info
coverage*.json
coverage*.xml
csx/
CTestTestfile.cmake
dlldata.c
DocProject/buildhelp/
DocProject/Help/*.hhc
DocProject/Help/*.hhk
DocProject/Help/*.hhp
DocProject/Help/*.HxC
DocProject/Help/*.HxT
DocProject/Help/html
DocProject/Help/Html2
ecf/
FakesAssemblies/
FodyWeavers.xsd
Generated_Code/
Generated\ Files/
healthchecksdb
install_manifest.txt
ipch/
Makefile
MigrationBackup/
mono_crash.*
nCrunchTemp_*
node_modules/
nunit-*.xml
OpenCover/
orleans.codegen.cs
Package.StoreAssociation.xml
paket-files/
project.fragment.lock.json
project.lock.json
publish/
PublishScripts/
rcf/
ScaffoldingReadMe.txt
ServiceFabricBackup/
StyleCopReport.xml
Testing
TestResult.xml
UpgradeLog*.htm
UpgradeLog*.XML
x64/
x86/
# Python coverage
.coverage
.coverage.*
htmlcov/
coverage.xml
.pytest_cache/
.benchmarks/

_private

================
File: .pre-commit-config.yaml
================
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.3.4
    hooks:
      - id: ruff
        args: [--fix]
      - id: ruff-format
        args: [--respect-gitignore]
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: check-yaml
      - id: check-toml
      - id: check-added-large-files
      - id: debug-statements
      - id: check-case-conflict
      - id: mixed-line-ending
        args: [--fix=lf]

================
File: API_REFERENCE.md
================
# twat-mp API Reference

This document provides a comprehensive reference for the `twat-mp` package's API.

## Table of Contents

- [Core Classes](#core-classes)
  - [MultiPool](#multipool)
  - [ProcessPool](#processpool)
  - [ThreadPool](#threadpool)
  - [AsyncMultiPool](#asyncmultipool)
- [Decorators](#decorators)
  - [pmap](#pmap)
  - [imap](#imap)
  - [amap](#amap)
  - [apmap](#apmap)
- [Usage Patterns](#usage-patterns)
  - [Choosing the Right Pool](#choosing-the-right-pool)
  - [Error Handling](#error-handling)
  - [Resource Management](#resource-management)

## Core Classes

### MultiPool

```python
class MultiPool:
    def __init__(self, pool_class=PathosProcessPool, nodes=None):
        ...
```

Base class for managing Pathos parallel processing pools. This class abstracts the creation and cleanup of a parallel processing pool, automatically choosing the number of nodes (processes or threads) based on the CPU count if not provided.

**Parameters:**
- `pool_class`: The Pathos pool class to instantiate (default: `PathosProcessPool`)
- `nodes`: The number of processes/threads to use (default: CPU count)

**Methods:**
- `__enter__()`: Enter the runtime context and create the pool
- `__exit__(exc_type, exc_value, traceback)`: Exit the runtime context, ensuring the pool is properly closed and resources are freed

**Example:**
```python
with MultiPool(pool_class=PathosProcessPool) as pool:
    results = pool.map(lambda x: x * 2, range(5))
print(list(results))
```

### ProcessPool

```python
class ProcessPool(MultiPool):
    def __init__(self, nodes=None):
        ...
```

Context manager specifically for creating a process-based pool. This subclass of MultiPool defaults to using the ProcessPool from Pathos.

**Parameters:**
- `nodes`: Number of processes to use (default: CPU count)

**Example:**
```python
with ProcessPool() as pool:
    results = pool.map(lambda x: x * 2, range(10))
```

### ThreadPool

```python
class ThreadPool(MultiPool):
    def __init__(self, nodes=None):
        ...
```

Context manager specifically for creating a thread-based pool. This subclass of MultiPool defaults to using the ThreadPool from Pathos.

**Parameters:**
- `nodes`: Number of threads to use (default: CPU count)

**Example:**
```python
with ThreadPool() as pool:
    results = pool.map(lambda x: x * 2, range(10))
```

### AsyncMultiPool

```python
class AsyncMultiPool:
    def __init__(self, processes=None, initializer=None, initargs=None, **kwargs):
        ...
```

A context manager for managing an aiomultiprocess.Pool. Provides high-level interface for parallel processing with async/await support.

**Parameters:**
- `processes`: Number of processes to use (default: CPU count)
- `initializer`: Optional callable to initialize worker processes
- `initargs`: Arguments to pass to the initializer
- `**kwargs`: Additional keyword arguments passed to aiomultiprocess.Pool

**Methods:**
- `__aenter__()`: Enter the async context, creating and starting the pool
- `__aexit__(exc_type, exc_val, exc_tb)`: Exit the async context, closing and joining the pool
- `map(func, iterable)`: Apply the function to each item in the iterable in parallel
- `starmap(func, iterable)`: Like map() but unpacks arguments from the iterable
- `imap(func, iterable)`: Async iterator version of map()

**Example:**
```python
async def process_items():
    async with AsyncMultiPool() as pool:
        async def work(x):
            await asyncio.sleep(0.1)  # Some async work
            return x * 2

        results = await pool.map(work, range(10))
        return results
```

## Decorators

### pmap

```python
@pmap
def func(x):
    ...
```

Standard parallel map decorator (eager evaluation). Wraps a function so that when it is called with an iterable, the function is applied in parallel using a ProcessPool.

**Example:**
```python
@pmap
def square(x):
    return x * x

results = list(square(range(10)))
```

### imap

```python
@imap
def func(x):
    ...
```

Lazy parallel map decorator that returns an iterator. Results are yielded as they become available.

**Example:**
```python
@imap
def cube(x):
    return x * x * x

for result in cube(range(5)):
    print(result)  # Prints results as they become available
```

### amap

```python
@amap
def func(x):
    ...
```

Asynchronous parallel map decorator with automatic result retrieval. Uses the 'amap' method of Pathos pools.

**Example:**
```python
@amap
def double(x):
    return x * 2

results = list(double(range(10)))
```

### apmap

```python
@apmap
async def func(x):
    ...
```

Decorator for async functions to run in parallel using AsyncMultiPool. Requires the 'aio' extra to be installed.

**Example:**
```python
@apmap
async def double(x):
    await asyncio.sleep(0.1)  # Some async work
    return x * 2

async def main():
    results = await double(range(10))
    print(results)

asyncio.run(main())
```

## Usage Patterns

### Choosing the Right Pool

- **ProcessPool**: Best for CPU-intensive tasks that benefit from parallel execution across multiple cores
- **ThreadPool**: Best for I/O-bound tasks where threads can efficiently wait for I/O operations
- **AsyncMultiPool**: Best for combining async/await with multiprocessing, particularly useful for mixed workloads

### Error Handling

All pools provide proper error propagation. Exceptions raised in worker processes/threads are propagated to the main process:

```python
try:
    with ProcessPool() as pool:
        results = list(pool.map(potentially_failing_function, data))
except Exception as e:
    print(f"An error occurred: {e}")
```

### Resource Management

The context manager pattern ensures proper cleanup of resources:

```python
# Resources are automatically cleaned up when exiting the context
with ProcessPool() as pool:
    # Use the pool
    results = pool.map(func, data)

# Pool is now closed and resources are freed
```

For async pools:

```python
async def main():
    async with AsyncMultiPool() as pool:
        # Use the pool
        results = await pool.map(async_func, data)

    # Pool is now closed and resources are freed

asyncio.run(main())
```

================
File: CHANGELOG.md
================
# Changelog

All notable changes to the `twat-mp` project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added

- Created comprehensive API reference documentation
- Added architecture diagrams explaining component relationships and workflows
- Added real-world examples for image processing, data processing, web scraping, and file operations
- Implemented better error handling with descriptive error messages and suggestions
- Added interactive examples in Jupyter notebooks
- Implemented debug mode with detailed logging

### Changed

- Improved resource cleanup in AsyncMultiPool.__aexit__ using close() instead of terminate()
- Enhanced error handling during pool cleanup to ensure proper resource management
- Updated docstrings with more examples and clearer explanations
- Improved compatibility with Python 3.12+ async features
- Enhanced exception propagation from worker processes
- Fixed build system configuration by adding proper `tool.hatch.version` settings
- Renamed `WorkerException` to `WorkerError` for consistency with Python naming conventions
- Updated tests to reflect the new error handling behavior
- Fixed `test_pool_reuse_failure` test to document current behavior
- Improved code quality by removing unused variables and lambda assignments

### Fixed

- Fixed AttributeError handling in exception handling code
- Improved graceful shutdown mechanism with better signal handling
- Fixed keyboard interrupt handling during parallel execution
- Addressed linter warnings and improved code quality
- Fixed package build error by adding missing version configuration in pyproject.toml
- Fixed inconsistent exception naming across the codebase
- Fixed test assertions to properly check for error messages in `WorkerError`

## [2.5.3] - 2025-03-04

### Added

- Added proper version configuration in pyproject.toml using hatch-vcs
- Improved build system configuration for better package distribution
- Created LOG.md file to track development progress (now merged into CHANGELOG.md)
- Updated TODO.md with prioritized tasks

### Fixed

- Fixed package build error related to missing version configuration
- Ensured proper version extraction from Git tags

### Changed

- Updated CHANGELOG.md with recent changes
- Updated README.md with information about recent updates

## [2.0.0] - 2025-02-20

### Added

- Added async support via `aiomultiprocess` integration
- New `AsyncMultiPool` class for combining async/await with multiprocessing
- New `apmap` decorator for easy async parallel processing
- Comprehensive test suite for async functionality
- Documentation for async features

### Changed

- Updated project dependencies to include optional `aiomultiprocess` support
- Enhanced type hints and error handling across the codebase
- Changed version handling to use static version instead of VCS-based versioning

## [1.7.5] - 2025-02-15

### Added

- Implemented AsyncMultiPool class for combining async/await with multiprocessing
- Added apmap decorator for easy async parallel processing
- Created comprehensive test suite for async functionality

### Changed

- Optimized CI/CD pipeline with improved GitHub Actions workflow
- Enhanced stability and performance optimizations in core multiprocessing functions
- Updated documentation and README with clearer usage examples
- Updated project dependencies to include optional aiomultiprocess support
- Enhanced type hints and error handling across the codebase

[Unreleased]: https://github.com/twardoch/twat-mp/compare/v2.5.3...HEAD
[2.5.3]: https://github.com/twardoch/twat-mp/compare/v2.0.0...v2.5.3
[2.0.0]: https://github.com/twardoch/twat-mp/compare/v1.7.5...v2.0.0
[1.7.5]: https://github.com/twardoch/twat-mp/compare/v1.7.3...v1.7.5
// ... existing references ...

================
File: cleanup.py
================
LOG_FILE = Path("CLEANUP.txt")
os.chdir(Path(__file__).parent)
def new() -> None:
    if LOG_FILE.exists():
        LOG_FILE.unlink()
def prefix() -> None:
    readme = Path(".cursor/rules/0project.mdc")
    if readme.exists():
        log_message("\n=== PROJECT STATEMENT ===")
        content = readme.read_text()
        log_message(content)
def suffix() -> None:
    todo = Path("TODO.md")
    if todo.exists():
        log_message("\n=== TODO.md ===")
        content = todo.read_text()
def log_message(message: str) -> None:
    timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
    with LOG_FILE.open("a") as f:
        f.write(log_line)
def run_command(cmd: list[str], check: bool = True) -> subprocess.CompletedProcess:
        result = subprocess.run(
            log_message(result.stdout)
        log_message(f"Command failed: {' '.join(cmd)}")
        log_message(f"Error: {e.stderr}")
        return subprocess.CompletedProcess(cmd, 1, "", str(e))
def check_command_exists(cmd: str) -> bool:
        return which(cmd) is not None
class Cleanup:
    def __init__(self) -> None:
        self.workspace = Path.cwd()
    def _print_header(self, message: str) -> None:
        log_message(f"\n=== {message} ===")
    def _check_required_files(self) -> bool:
            if not (self.workspace / file).exists():
                log_message(f"Error: {file} is missing")
    def _generate_tree(self) -> None:
        if not check_command_exists("tree"):
            log_message("Warning: 'tree' command not found. Skipping tree generation.")
            rules_dir = Path(".cursor/rules")
            rules_dir.mkdir(parents=True, exist_ok=True)
            tree_result = run_command(
            with open(rules_dir / "filetree.mdc", "w") as f:
                f.write("---\ndescription: File tree of the project\nglobs: \n---\n")
                f.write(tree_text)
            log_message("\nProject structure:")
            log_message(tree_text)
            log_message(f"Failed to generate tree: {e}")
    def _git_status(self) -> bool:
        result = run_command(["git", "status", "--porcelain"], check=False)
        return bool(result.stdout.strip())
    def _venv(self) -> None:
        log_message("Setting up virtual environment")
            run_command(["uv", "venv"])
            if venv_path.exists():
                os.environ["VIRTUAL_ENV"] = str(self.workspace / ".venv")
                log_message("Virtual environment created and activated")
                log_message("Virtual environment created but activation failed")
            log_message(f"Failed to create virtual environment: {e}")
    def _install(self) -> None:
        log_message("Installing package with all extras")
            self._venv()
            run_command(["uv", "pip", "install", "-e", ".[test,dev]"])
            log_message("Package installed successfully")
            log_message(f"Failed to install package: {e}")
    def _run_checks(self) -> None:
        log_message("Running code quality checks")
            log_message(">>> Running code fixes...")
            run_command(
            log_message(">>>Running type checks...")
            run_command(["python", "-m", "mypy", "src", "tests"], check=False)
            log_message(">>> Running tests...")
            run_command(["python", "-m", "pytest", "tests"], check=False)
            log_message("All checks completed")
            log_message(f"Failed during checks: {e}")
    def status(self) -> None:
        prefix()  # Add README.md content at start
        self._print_header("Current Status")
        self._check_required_files()
        self._generate_tree()
        result = run_command(["git", "status"], check=False)
        self._print_header("Environment Status")
        self._install()
        self._run_checks()
        suffix()  # Add TODO.md content at end
    def venv(self) -> None:
        self._print_header("Virtual Environment Setup")
    def install(self) -> None:
        self._print_header("Package Installation")
    def update(self) -> None:
        self.status()
        if self._git_status():
            log_message("Changes detected in repository")
                run_command(["git", "add", "."])
                run_command(["git", "commit", "-m", commit_msg])
                log_message("Changes committed successfully")
                log_message(f"Failed to commit changes: {e}")
            log_message("No changes to commit")
    def push(self) -> None:
        self._print_header("Pushing Changes")
            run_command(["git", "push"])
            log_message("Changes pushed successfully")
            log_message(f"Failed to push changes: {e}")
def repomix(
            cmd.append("--compress")
            cmd.append("--remove-empty-lines")
            cmd.append("-i")
            cmd.append(ignore_patterns)
        cmd.extend(["-o", output_file])
        run_command(cmd)
        log_message(f"Repository content mixed into {output_file}")
        log_message(f"Failed to mix repository: {e}")
def print_usage() -> None:
    log_message("Usage:")
    log_message("  cleanup.py status   # Show current status and run all checks")
    log_message("  cleanup.py venv     # Create virtual environment")
    log_message("  cleanup.py install  # Install package with all extras")
    log_message("  cleanup.py update   # Update and commit changes")
    log_message("  cleanup.py push     # Push changes to remote")
def main() -> NoReturn:
    new()  # Clear log file
    if len(sys.argv) < 2:
        print_usage()
        sys.exit(1)
    cleanup = Cleanup()
            cleanup.status()
            cleanup.venv()
            cleanup.install()
            cleanup.update()
            cleanup.push()
        log_message(f"Error: {e}")
    repomix()
    sys.stdout.write(Path("CLEANUP.txt").read_text())
    sys.exit(0)  # Ensure we exit with a status code
    main()

================
File: LICENSE
================
MIT License

Copyright (c) 2025 Adam Twardoch

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

================
File: LOG.md
================
# Development Log

This file tracks the development progress of the `twat-mp` project. For a more structured changelog, see [CHANGELOG.md](CHANGELOG.md).

## 2025-03-04

### Linting and Test Fixes

- Renamed `WorkerException` to `WorkerError` for consistency with Python naming conventions
- Updated tests to reflect the new error handling behavior
- Fixed `test_pool_reuse_failure` test to document current behavior
- Ensured all tests pass with the updated error handling
- Removed unused variables and improved code quality
- Updated TODO.md to mark completed tasks
- Updated CHANGELOG.md with recent changes

### Code Quality Improvements

- Fixed inconsistent exception naming across the codebase
- Fixed test assertions to properly check for error messages in `WorkerError`
- Removed lambda assignments in favor of direct function references
- Improved error handling with more descriptive error messages

### Documentation Updates

- Added "Recently Completed" section to TODO.md
- Updated CHANGELOG.md with detailed descriptions of recent changes
- Created LOG.md file to track development progress

## Next Steps

- Continue implementing tasks from TODO.md, focusing on Phase 1 priorities
- Consider adding a CLI interface for common operations
- Implement support for cancellation of running tasks
- Add progress tracking for long-running parallel operations

================
File: pyproject.toml
================
# this_file: twat_mp/pyproject.toml
# Build System Configuration
# -------------------------
# Specifies the build system and its requirements for packaging the project
# Specifies the build backend and its requirements for building the package
[build-system]
requires = [
    "hatchling>=1.27.0", # Core build backend for Hatch
    "hatch-vcs>=0.4.0", # Version Control System plugin for Hatch

]
build-backend = "hatchling.build" # Use Hatchling as the build backend

# Project Metadata Configuration
# ------------------------------
# Comprehensive project description, requirements, and compatibility information
[project]
name = "twat-mp"
dynamic = ["version"]
description = "Parallel processing utilities using Pathos mpprocessing library"
readme = "README.md"
requires-python = ">=3.10" # Minimum Python version required
license = "MIT"
keywords = ["parallel", "mpprocessing", "pathos", "map", "pool"]
classifiers = [
    "Development Status :: 4 - Beta",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
    "Programming Language :: Python :: Implementation :: CPython",
    "Programming Language :: Python :: Implementation :: PyPy",
]
# Runtime Dependencies
# -------------------
# External packages required for the project to function
dependencies = [
    "pathos>=0.3.3", # Parallel processing library
    "twat>=1.8.1",
    "aiomultiprocess>=0.9.1"
    # Main twat package
]

[[project.authors]]
name = "Adam Twardoch"
email = "adam+github@twardoch.com"

[project.urls]
Documentation = "https://github.com/twardoch/twat-mp#readme"
Issues = "https://github.com/twardoch/twat-mp/issues"
Source = "https://github.com/twardoch/twat-mp"

[project.entry-points."twat.plugins"]
mp = "twat_mp"

[tool.hatch.build.targets.wheel]
packages = ["src/twat_mp"]

[tool.hatch.envs.default]
dependencies = ["mypy>=1.0.0", "ruff>=0.1.0"]

[project.optional-dependencies]
test = [
    "pytest>=7.0.0",
    "pytest-cov>=4.0.0",
    "pytest-benchmark[histogram]>=4.0.0",
    "pytest-xdist>=3.5.0", # For parallel test execution
    "pandas>=2.0.0", # Required by some test dependencies
    "numpy>=1.24.0", # Required by pandas
    "matplotlib>=3.7.0", # For benchmark visualization

]
dev = ["pre-commit>=3.6.0"]
all = [
    "twat>=1.0.0",
    "pathos>=0.3.3",
    "aiomultiprocess>=0.9.1"
    # Include aio in all
]

[tool.hatch.envs.test]
dependencies = [".[test]"]

[tool.hatch.envs.test.scripts]
# Regular tests can run in parallel
test = "python -m pytest -n auto {args:tests}"
test-cov = "python -m pytest -n auto --cov-report=term-missing --cov-config=pyproject.toml --cov=src/twat_mp --cov=tests {args:tests}"
# Benchmarks must run sequentially
bench = "python -m pytest -v -p no:briefcase tests/test_benchmark.py --benchmark-only"
bench-save = "python -m pytest -v -p no:briefcase tests/test_benchmark.py --benchmark-only --benchmark-json=benchmark/results.json"
bench-hist = "python -m pytest -v -p no:briefcase tests/test_benchmark.py --benchmark-only --benchmark-histogram=benchmark/hist"
bench-compare = "python -m pytest-benchmark compare benchmark/results.json --sort fullname --group-by func"

[tool.hatch.envs.lint]
detached = true
dependencies = ["black>=23.1.0", "mypy>=1.0.0", "ruff>=0.1.0"]

[tool.hatch.envs.lint.scripts]
typing = "mypy --install-types --non-interactive {args:src/twat_mp tests}"
style = ["ruff check {args:.}", "ruff format {args:.}"]
fmt = ["ruff format {args:.}", "ruff check --fix {args:.}"]
all = ["style", "typing"]

[tool.ruff]
target-version = "py310"
line-length = 88
lint.extend-select = [
    "I", # isort
    "N", # pep8-naming
    "B", # flake8-bugbear
    "RUF", # Ruff-specific rules

]
lint.ignore = [
    "ARG001", # Unused function argument
    "E501", # Line too long
    "I001",
]

[tool.ruff.format]
quote-style = "double"
indent-style = "space"
skip-magic-trailing-comma = false
line-ending = "lf"

[tool.ruff.lint.per-file-ignores]
"tests/*" = ["S101"]

[tool.coverage.run]
source_pkgs = ["twat_mp", "tests"]
branch = true
parallel = true
omit = ["src/twat_mp/__about__.py"]

[tool.coverage.paths]
twat_mp = ["src/twat_mp", "*/twat-mp/src/twat_mp"]
tests = ["tests", "*/twat-mp/tests"]

[tool.coverage.report]
exclude_lines = ["no cov", "if __name__ == .__main__.:", "if TYPE_CHECKING:"]

[tool.mypy]
python_version = "3.10"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
check_untyped_defs = true
disallow_untyped_decorators = true
no_implicit_optional = true
warn_redundant_casts = true
warn_unused_ignores = true
warn_no_return = true
warn_unreachable = true

[tool.pytest.ini_options]
markers = ["benchmark: marks tests as benchmarks (select with '-m benchmark')"]
addopts = "-v -p no:briefcase"
testpaths = ["tests"]
python_files = ["test_*.py"]
filterwarnings = ["ignore::DeprecationWarning", "ignore::UserWarning"]
asyncio_mode = "auto"
asyncio_default_fixture_loop_scope = "function"

[tool.pytest-benchmark]
min_rounds = 100
min_time = 0.1
histogram = true
storage = "file"
save-data = true
compare = [
    "min", # Minimum time
    "max", # Maximum time
    "mean", # Mean time
    "stddev", # Standard deviation
    "median", # Median time
    "iqr", # Inter-quartile range
    "ops", # Operations per second
    "rounds", # Number of rounds

]

[tool.hatch.version]
source = "vcs"

================
File: README.md
================
# twat-mp

Parallel processing utilities using Pathos and aiomultiprocess libraries. This package provides convenient context managers and decorators for parallel processing, with process-based, thread-based, and async-based pools.

## Features

* Multiple parallel processing options:
  + `ProcessPool`: For CPU-intensive parallel processing using Pathos
  + `ThreadPool`: For I/O-bound parallel processing using Pathos
  + `AsyncMultiPool`: For combined async/await with multiprocessing using aiomultiprocess
* Decorators for common parallel mapping operations:
  + `pmap`: Standard parallel map (eager evaluation)
  + `imap`: Lazy parallel map returning an iterator
  + `amap`: Asynchronous map with automatic result retrieval
  + `apmap`: Async parallel map for use with async/await functions
* Automatic CPU core detection for optimal pool sizing
* Clean resource management with context managers
* Full type hints and modern Python features
* Flexible pool configuration with customizable worker count
* Graceful error handling and resource cleanup
* Enhanced exception propagation with detailed context
* Debug mode with comprehensive logging
* Optional dependencies to reduce installation footprint
* Version control system (VCS) based versioning using hatch-vcs

## Recent Updates

* Added debug mode with detailed logging via `set_debug_mode()`
* Enhanced error handling with `WorkerException` for better context
* Improved exception propagation from worker processes
* Added comprehensive docstrings to all public functions and classes
* Fixed build system configuration with proper version handling
* Enhanced error handling and resource cleanup
* Improved compatibility with Python 3.12+ async features
* Added comprehensive API reference documentation
* Added real-world examples for various use cases

## Installation

Basic installation:
```bash
pip install twat-mp
```

With async support:
```bash
pip install 'twat-mp[aio]'
```

With all extras and development tools:
```bash
pip install 'twat-mp[all,dev]'
```

## Usage

### Basic Usage

```python
from twat_mp import ProcessPool, pmap

# Using the pool directly
with ProcessPool() as pool:
    results = pool.map(lambda x: x * 2, range(10))

# Using the decorator
@pmap
def double(x):
    return x * 2

results = double(range(10))
```

### Async Support

The package provides async support through `aiomultiprocess`, allowing you to combine the benefits of async/await with multiprocessing:

```python
import asyncio
from twat_mp import AsyncMultiPool, apmap

# Using the pool directly
async def process_items():
    async with AsyncMultiPool() as pool:
        async def work(x):
            await asyncio.sleep(0.1)  # Some async work
            return x * 2

        results = await pool.map(work, range(10))
        return results

# Using the decorator
@apmap
async def double(x):
    await asyncio.sleep(0.1)  # Some async work
    return x * 2

async def main():
    results = await double(range(10))
    print(results)  # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

asyncio.run(main())
```

The async support is particularly useful when you need to:
- Perform CPU-intensive tasks in parallel
- Handle many concurrent I/O operations
- Combine async/await with true multiprocessing
- Process results from async APIs in parallel

### Advanced Async Features

The `AsyncMultiPool` provides additional methods for different mapping strategies:

```python
import asyncio
from twat_mp import AsyncMultiPool

async def main():
    # Using starmap for unpacking arguments
    async def sum_values(a, b):
        await asyncio.sleep(0.01)
        return a + b

    async with AsyncMultiPool() as pool:
        # Regular map
        double_results = await pool.map(
            lambda x: x * 2,
            range(5)
        )
        print(double_results)  # [0, 2, 4, 6, 8]

        # Starmap unpacks arguments
        sum_results = await pool.starmap(
            sum_values,
            [(1, 2), (3, 4), (5, 6)]
        )
        print(sum_results)  # [3, 7, 11]

        # imap returns an async iterator
        async for result in pool.imap(sum_values, [(1, 1), (2, 2), (3, 3)]):
            print(result)  # Prints 2, 4, 6 as they complete

asyncio.run(main())
```

### Using Process and Thread Pools

The package provides dedicated context managers for both process and thread pools:

```python
from twat_mp import ProcessPool, ThreadPool

# For CPU-intensive operations
with ProcessPool() as pool:
    results = pool.map(lambda x: x * x, range(10))
    print(list(results))  # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

# For I/O-bound operations
with ThreadPool() as pool:
    results = pool.map(lambda x: x * 2, range(10))
    print(list(results))  # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

# Custom number of workers
with ProcessPool(nodes=4) as pool:
    results = pool.map(lambda x: x * x, range(10))
```

### Using Map Decorators

The package provides three decorators for different mapping strategies:

```python
from twat_mp import amap, imap, pmap

# Standard parallel map (eager evaluation)
@pmap
def square(x: int) -> int:
    return x * x

results = list(square(range(10)))
print(results)  # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

# Lazy parallel map (returns iterator)
@imap
def cube(x: int) -> int:
    return x * x * x

for result in cube(range(5)):
    print(result)  # Prints results as they become available

# Asynchronous parallel map with automatic result retrieval
@amap
def double(x: int) -> int:
    return x * 2

results = list(double(range(10)))
print(results)  # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
```

### Function Composition

Decorators can be composed for complex parallel operations:

```python
from twat_mp import amap

@amap
def compute_intensive(x: int) -> int:
    result = x
    for _ in range(1000):  # Simulate CPU-intensive work
        result = (result * x + x) % 10000
    return result

@amap
def io_intensive(x: int) -> int:
    import time
    time.sleep(0.001)  # Simulate I/O wait
    return x * 2

# Chain parallel operations
results = list(io_intensive(compute_intensive(range(100))))
```

### Debug Mode and Error Handling

The package provides a debug mode for detailed logging and enhanced error handling:

```python
from twat_mp import ProcessPool, set_debug_mode
import time
import random

def process_item(x):
    """Process an item with random delay and potential errors."""
    # Simulate random processing time
    time.sleep(random.random() * 0.5)
    
    # Randomly fail for demonstration
    if random.random() < 0.2:  # 20% chance of failure
        raise ValueError(f"Random failure processing item {x}")
        
    return x * 10

# Enable debug mode for detailed logging
set_debug_mode(True)

try:
    with ProcessPool() as pool:
        results = list(pool.map(process_item, range(10)))
        print(f"Processed results: {results}")
except Exception as e:
    print(f"Caught exception: {e}")
    # The exception will include details about which worker and input item caused the error
finally:
    # Disable debug mode when done
    set_debug_mode(False)
```

The enhanced error handling provides detailed context about failures:

```python
from twat_mp import ProcessPool

def risky_function(x):
    if x == 5:
        raise ValueError("Cannot process item 5")
    return x * 2

try:
    with ProcessPool() as pool:
        results = list(pool.map(risky_function, range(10)))
except ValueError as e:
    # The exception will include the worker ID and input item that caused the error
    print(f"Caught error: {e}")
    # Handle the error appropriately
```

## Real-World Examples

### Image Processing

Processing images in parallel can significantly speed up operations like resizing, filtering, or format conversion:

```python
from twat_mp import ProcessPool
from PIL import Image
import os

def resize_image(file_path):
    """Resize an image to 50% of its original size."""
    try:
        with Image.open(file_path) as img:
            # Get the original size
            width, height = img.size
            # Resize to 50%
            resized = img.resize((width // 2, height // 2))
            # Save with '_resized' suffix
            output_path = os.path.splitext(file_path)[0] + '_resized' + os.path.splitext(file_path)[1]
            resized.save(output_path)
            return output_path
    except Exception as e:
        return f"Error processing {file_path}: {e}"

# Get all image files in a directory
image_files = [f for f in os.listdir('images') if f.lower().endswith(('.png', '.jpg', '.jpeg'))]
image_paths = [os.path.join('images', f) for f in image_files]

# Process images in parallel
with ProcessPool() as pool:
    results = list(pool.map(resize_image, image_paths))

print(f"Processed {len(results)} images")
```

### Web Scraping

Thread pools are ideal for I/O-bound operations like web scraping:

```python
import requests
from bs4 import BeautifulSoup
from twat_mp import ThreadPool

def fetch_page_title(url):
    """Fetch the title of a webpage."""
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, 'html.parser')
        title = soup.title.string if soup.title else "No title found"
        return {"url": url, "title": title, "status": response.status_code}
    except Exception as e:
        return {"url": url, "error": str(e), "status": None}

# List of URLs to scrape
urls = [
    "https://www.python.org",
    "https://www.github.com",
    "https://www.stackoverflow.com",
    "https://www.wikipedia.org",
    "https://www.reddit.com"
]

# Use ThreadPool for I/O-bound operations
with ThreadPool() as pool:
    results = list(pool.map(fetch_page_title, urls))

# Print results
for result in results:
    if "error" in result:
        print(f"Error fetching {result['url']}: {result['error']}")
    else:
        print(f"{result['url']} - {result['title']} (Status: {result['status']})")
```

### Data Processing with Pandas

Process large datasets in parallel chunks:

```python
import pandas as pd
import numpy as np
from twat_mp import ProcessPool

def process_chunk(chunk_data):
    """Process a chunk of data."""
    # Simulate some data processing
    chunk_data['processed'] = chunk_data['value'] * 2 + np.random.randn(len(chunk_data))
    chunk_data['category'] = pd.cut(chunk_data['processed'], 
                                    bins=[-np.inf, 0, 10, np.inf], 
                                    labels=['low', 'medium', 'high'])
    # Calculate some statistics
    result = {
        'chunk_id': chunk_data['chunk_id'].iloc[0],
        'mean': chunk_data['processed'].mean(),
        'median': chunk_data['processed'].median(),
        'std': chunk_data['processed'].std(),
        'count': len(chunk_data),
        'categories': chunk_data['category'].value_counts().to_dict()
    }
    return result

# Create a large DataFrame
n_rows = 1_000_000
df = pd.DataFrame({
    'value': np.random.randn(n_rows),
    'group': np.random.choice(['A', 'B', 'C', 'D'], n_rows)
})

# Split into chunks for parallel processing
chunk_size = 100_000
chunks = []
for i, chunk_start in enumerate(range(0, n_rows, chunk_size)):
    chunk_end = min(chunk_start + chunk_size, n_rows)
    chunk = df.iloc[chunk_start:chunk_end].copy()
    chunk['chunk_id'] = i
    chunks.append(chunk)

# Process chunks in parallel
with ProcessPool() as pool:
    results = list(pool.map(process_chunk, chunks))

# Combine results
summary = pd.DataFrame(results)
print(summary)
```

### Async File Processing

Combine async I/O with parallel processing:

```python
import asyncio
import aiofiles
import os
from twat_mp import AsyncMultiPool

async def count_words(filename):
    """Count words in a file asynchronously."""
    try:
        async with aiofiles.open(filename, 'r') as f:
            content = await f.read()
            word_count = len(content.split())
            return {"filename": filename, "word_count": word_count}
    except Exception as e:
        return {"filename": filename, "error": str(e)}

async def main():
    # Get all text files in a directory
    files = [os.path.join("documents", f) for f in os.listdir("documents") 
             if f.endswith(".txt")]
    
    # Process files in parallel
    async with AsyncMultiPool() as pool:
        results = await pool.map(count_words, files)
    
    # Calculate total word count
    total_words = sum(r.get("word_count", 0) for r in results)
    
    # Print results
    for result in results:
        if "error" in result:
            print(f"Error processing {result['filename']}: {result['error']}")
        else:
            print(f"{result['filename']}: {result['word_count']} words")
    
    print(f"Total word count: {total_words}")

# Run the async main function
asyncio.run(main())
```

## API Reference

For detailed API documentation, see the [API Reference](API_REFERENCE.md).

## License

MIT

================
File: TODO.md
================
---
this_file: TODO.md
---

# TODO

Tip: Periodically run `python ./cleanup.py status` to see results of lints and tests. Use `uv pip ...` not `pip ...`


## Phase 1

- [ ] Add a CLI interface for common operations
- [ ] Add support for cancellation of running tasks
- [ ] Implement progress tracking for long-running parallel operations
- [ ] Implement backpressure mechanisms for memory-intensive operations
- [ ] Add support for process affinity and priority settings
- [ ] Implement a unified interface for all pool types
- [ ] Add support for custom serialization/deserialization
- [ ] Add context manager for automatic resource cleanup across all parallel operations
- [ ] Implement adaptive pool sizing based on workload and system resources
- [ ] Add support for task prioritization and scheduling

## Phase 2

- [ ] Increase test coverage to at least 90%
- [ ] Implement stress tests for high concurrency scenarios
- [ ] Add tests for resource leaks and memory usage
- [ ] Add tests for different Python versions and platforms
- [ ] Implement property-based testing for robustness

## Phase 3

- [ ] Optimize serialization/deserialization for better performance
- [ ] Optimize memory usage for large datasets
- [ ] Implement caching mechanisms for repeated operations
- [ ] Profile and optimize critical code paths
- [ ] Add performance comparison with native multiprocessing
- [ ] Implement workload-specific optimization strategies

## Phase 4

- [ ] Implement compatibility with other multiprocessing libraries
- [ ] Add support for distributed processing across multiple machines
- [ ] Add support for GPU acceleration where applicable
- [ ] Implement integration with dask and other distributed computing frameworks
- [ ] Add support for cloud-based parallel processing

## User Experience

- [ ] Implement automatic pool selection based on workload characteristics
- [ ] Add progress bars and status reporting for long-running tasks
- [ ] Create wizard-like interfaces for common operations

## Package Management

- [ ] Create a conda package for easier installation
- [ ] Implement automated release process with GitHub Actions
- [ ] Add support for Windows-specific optimizations
- [ ] Create development documentation for contributors

================
File: VERSION.txt
================
v2.6.2



================================================================
End of Codebase
================================================================



================================================================
End of Codebase
================================================================
