# Undetecta Python SDK Architecture Design

## Overview

The `undetecta` Python package provides a type-safe, ergonomic interface for the Undetecta API. It mirrors the JavaScript SDK's API surface while following Python best practices (PEP 517, PEP 621, type hints).

## Design Principles

1. **Type Safety**: Full type hints using Python's `typing` module with Pydantic for runtime validation
2. **Ergonomics**: Simple, intuitive API that follows common Python client patterns (requests, httpx)
3. **Async Support**: First-class async support with httpx for modern Python applications
4. **Error Handling**: Structured exception hierarchy for predictable error handling
5. **Modern Packaging**: `pyproject.toml` with PEP 517/621 compliance
6. **Minimal Dependencies**: Only essential dependencies (httpx, pydantic)

## API Surface Analysis

The Undetecta API has the following public endpoints:

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/v1/scrape` | Scrape a URL (sync response) |
| POST | `/v1/search` | Web search (sync response) |
| GET | `/v1/health` | Health check |

**Note**: Admin endpoints (users, API keys, proxies) are internal and not exposed in the public client.

### Request/Response Pattern

All responses follow the `result()` wrapper pattern:

```python
# Success response
{"success": True, "data": T}

# Error response
{"success": False, "error": {"code": str, "message": str}}
```

### Authentication

Authentication is via `x-api-key` header.

## Package Structure

```
undetecta/
├── pyproject.toml         # PEP 621 package configuration
├── README.md              # Package documentation
├── LICENSE                # MIT License
├── src/
│   └── undetecta/
│       ├── __init__.py            # Main entry point, exports
│       ├── _client.py             # UndetectaClient class
│       ├── _types.py              # Type definitions and Pydantic models
│       ├── _errors.py             # Exception classes
│       ├── _scrape.py             # Scrape API methods
│       ├── _search.py             # Search API methods
│       └── _transport/
│           ├── __init__.py
│           ├── _http.py           # HTTP transport using httpx
│           └── _retry.py          # Retry logic with exponential backoff
├── tests/
│   ├── __init__.py
│   ├── test_client.py
│   ├── test_scrape.py
│   ├── test_search.py
│   └── test_errors.py
├── .gitignore
├── .python-version          # Python version (3.10+)
└── CHANGELOG.md
```

**Note**: Leading underscores on modules (`_client.py`) indicate private/internal modules. The public API is exported through `__init__.py`.

## Client Architecture

### Main Client Class

```python
class UndetectaClient:
    """Undetecta API client."""

    def __init__(
        self,
        *,
        api_key: str,
        base_url: str = "https://api.undetecta.com",
        timeout: float = 60.0,
        max_retries: int = 3,
    ) -> None: ...

    # Core methods
    async def scrape(self, request: ScrapeRequest) -> ScrapeResponse: ...
    async def search(self, request: SearchRequest) -> SearchResponse: ...
    async def health(self) -> HealthResponse: ...

    # Sync variants (optional, convenience)
    def scrape_sync(self, request: ScrapeRequest) -> ScrapeResponse: ...
    def search_sync(self, request: SearchRequest) -> SearchResponse: ...

    # Configuration (read-only)
    @property
    def api_key(self) -> str: ...
    @property
    def base_url(self) -> str: ...
    @property
    def timeout(self) -> float: ...
    @property
    def max_retries(self) -> int: ...
```

### Synchronous Support

For users who prefer synchronous code, the SDK provides blocking methods via `httpx.Client`:

```python
# Using async (default)
client = UndetectaClient(api_key="sk_...")
result = await client.scrape(ScrapeRequest(url="https://example.com"))

# Using sync convenience methods
result = client.scrape_sync(ScrapeRequest(url="https://example.com"))
```

### Type System

All types use Python's `typing` module with Pydantic for validation:

```python
# Enums (using StrEnum for Python 3.11+)
class ScrapeFormat(StrEnum):
    HTML = "html"
    RAW_HTML = "rawHtml"
    MARKDOWN = "markdown"
    LINKS = "links"
    SCREENSHOT = "screenshot"
    BRANDING = "branding"

class WaitUntil(StrEnum):
    LOAD = "load"
    DOM_CONTENT_LOADED = "domcontentloaded"
    NETWORK_IDLE = "networkidle"

class JobStatus(StrEnum):
    PENDING = "pending"
    RUNNING = "running"
    COMPLETED = "completed"
    FAILED = "failed"
    STOPPED = "stopped"

# Request models (Pydantic)
class ScrapeRequest(BaseModel):
    url: str
    formats: list[ScrapeFormat] = [ScrapeFormat.MARKDOWN]
    timeout: int = 30000
    wait_until: WaitUntil = WaitUntil.DOM_CONTENT_LOADED
    # ... 30+ other optional fields

# Response models (Pydantic)
class ScrapeMetadata(BaseModel):
    status_code: int | None = None
    url: str | None = None
    title: str | None = None
    description: str | None = None
    # ...

class ScrapeResponse(BaseModel):
    id: str
    status: JobStatus
    created_at: str  # ISO 8601 datetime
    completed_at: str | None = None
    metadata: ScrapeMetadata | None = None
    markdown: str | None = None
    html: str | None = None
    # ...
```

### Export Structure

```python
# src/undetecta/__init__.py
"""Undetecta API Client."""

from undetecta._client import UndetectaClient
from undetecta._types import (
    ScrapeFormat,
    ScrapeRequest,
    ScrapeResponse,
    SearchRequest,
    SearchResponse,
    WaitUntil,
    # ... other types
)
from undetecta._errors import (
    UndetectaError,
    ApiKeyError,
    RateLimitError,
    ValidationError,
    NetworkError,
    TimeoutError,
    NotFoundError,
    ServerError,
)

__all__ = [
    # Client
    "UndetectaClient",
    # Types
    "ScrapeFormat",
    "ScrapeRequest",
    "ScrapeResponse",
    "SearchRequest",
    "SearchResponse",
    "WaitUntil",
    # ...
    # Errors
    "UndetectaError",
    "ApiKeyError",
    "RateLimitError",
    "ValidationError",
    "NetworkError",
    "TimeoutError",
    "NotFoundError",
    "ServerError",
]
```

## Error Handling Strategy

### Exception Hierarchy

```python
class UndetectaError(Exception):
    """Base class for all Undetecta errors."""

    def __init__(self, code: str, message: str, status_code: int | None = None) -> None: ...

class ApiKeyError(UndetectaError):
    """Raised when API key is invalid or missing (401)."""

class RateLimitError(UndetectaError):
    """Raised when rate limit is exceeded (429)."""

class ValidationError(UndetectaError):
    """Raised when request validation fails (400)."""

class NotFoundError(UndetectaError):
    """Raised when a resource is not found (404)."""

class ServerError(UndetectaError):
    """Raised when the server returns a 5xx error."""

class NetworkError(UndetectaError):
    """Raised when network request fails."""

class TimeoutError(UndetectaError):
    """Raised when request times out."""
```

### Error Handling Pattern

```python
from undetecta import UndetectaClient, ApiKeyError, RateLimitError

client = UndetectaClient(api_key="sk_...")

try:
    result = await client.scrape(ScrapeRequest(url="https://example.com"))
except ApiKeyError as e:
    print(f"Invalid API key: {e.message}")
except RateLimitError as e:
    print(f"Rate limited, retry later: {e.message}")
except ValidationError as e:
    print(f"Invalid request: {e.message}")
except UndetectaError as e:
    print(f"API error: {e.code} - {e.message}")
```

## HTTP Transport

### Using httpx

The SDK uses `httpx` for HTTP requests:

1. **Async-first**: Primary API uses `httpx.AsyncClient`
2. **Sync fallback**: Sync variants use `httpx.Client`
3. **Automatic retries**: Exponential backoff for 5xx and network errors
4. **Timeout handling**: Per-request timeout with `httpx.Timeout`
5. **Headers injection**: `x-api-key`, `Content-Type`, `User-Agent`

```python
import httpx

class _HttpTransport:
    """Internal HTTP transport with retry logic."""

    async def _request(
        self,
        method: str,
        url: str,
        *,
        json: dict[str, Any] | None = None,
        max_retries: int = 3,
    ) -> dict[str, Any]: ...
```

### Retry Strategy

- **Retryable status codes**: 408, 429, 500-599
- **Backoff**: Exponential with jitter (base: 1s, max: 30s)
- **Max retries**: Configurable (default: 3)
- **Non-retryable**: 4xx errors (except 429)

## Configuration

### pyproject.toml

```toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "undetecta"
version = "0.1.0"
description = "Python SDK for the Undetecta API"
readme = "README.md"
requires-python = ">=3.10"
license = { text = "MIT" }
authors = [
    { name = "Undetecta", email = "sdk@undetecta.com" }
]
keywords = ["scraping", "automation", "anti-detection"]
classifiers = [
    "Development Status :: 4 - Beta",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
]

dependencies = [
    "httpx>=0.27.0",
    "pydantic>=2.0.0",
]

[project.optional-dependencies]
dev = [
    "pytest>=8.0.0",
    "pytest-asyncio>=0.24.0",
    "pytest-cov>=5.0.0",
    "ruff>=0.8.0",
    "mypy>=1.0.0",
]

[project.urls]
Homepage = "https://undetecta.com"
Documentation = "https://github.com/undetecta/sdk-python"
Repository = "https://github.com/undetecta/sdk-python"
Issues = "https://github.com/undetecta/sdk-python/issues"

[tool.hatch.build.targets.wheel]
packages = ["src/undetecta"]

[tool.ruff]
line-length = 100
target-version = "py310"

[tool.ruff.lint]
select = ["E", "F", "I", "N", "W", "UP"]
ignore = ["E501"]

[tool.mypy]
python_version = "3.10"
strict = true
warn_return_any = true
warn_unused_ignores = true
```

## Usage Examples

### Basic Scraping

```python
import asyncio
from undetecta import UndetectaClient, ScrapeRequest, ScrapeFormat

async def main():
    client = UndetectaClient(api_key="sk_...")

    result = await client.scrape(ScrapeRequest(
        url="https://example.com",
        formats=[ScrapeFormat.MARKDOWN]
    ))

    print(result.markdown)

asyncio.run(main())
```

### Synchronous Usage

```python
from undetecta import UndetectaClient, ScrapeRequest

client = UndetectaClient(api_key="sk_...")
result = client.scrape_sync(ScrapeRequest(url="https://example.com"))
print(result.markdown)
```

### Advanced Scraping

```python
result = await client.scrape(ScrapeRequest(
    url="https://example.com",
    formats=[ScrapeFormat.MARKDOWN, ScrapeFormat.SCREENSHOT, ScrapeFormat.LINKS],
    screenshot_options=ScreenshotOptions(
        full_page=True,
        format="png"
    ),
    wait_for_selector=".main-content",
    actions=[
        Action(type="click", selector=".cookie-accept")
    ]
))
```

### Search

```python
result = await client.search(SearchRequest(
    query="python web scraping",
    limit=10,
    sources=[SearchSource.WEB],
    scrape_options=SearchScrapeOptions(
        formats=[ScrapeFormat.MARKDOWN]
    )
))

for item in result.web or []:
    print(f"{item.title}: {item.url}")
    if item.markdown:
        print(item.markdown[:200])
```

### Custom Configuration

```python
client = UndetectaClient(
    api_key="sk_...",
    base_url="https://api.undetecta.loc:6363",  # Local development
    timeout=120.0,
    max_retries=5
)
```

## Dependencies

### Runtime
- `httpx>=0.27.0`: Async HTTP client with sync support
- `pydantic>=2.0.0`: Runtime validation and type definitions

### Development
- `pytest>=8.0.0`: Testing framework
- `pytest-asyncio>=0.24.0`: Async test support
- `pytest-cov>=5.0.0`: Coverage reporting
- `ruff>=0.8.0`: Fast Python linter
- `mypy>=1.0.0`: Static type checking

## Type Definitions Mapping

The Python SDK types mirror the JavaScript SDK and Zod schemas:

| JavaScript/TS | Python | Pydantic |
|---------------|--------|----------|
| `type ScrapeFormat = ...` | `class ScrapeFormat(StrEnum)` | N/A |
| `interface ScrapeOptions` | `class ScrapeRequest(BaseModel)` | `BaseModel` |
| `interface ScrapeJobResponse` | `class ScrapeResponse(BaseModel)` | `BaseModel` |
| `enum WaitUntil` | `class WaitUntil(StrEnum)` | N/A |
| `type JobStatus = ...` | `class JobStatus(StrEnum)` | N/A |

### Naming Conventions

- **Request types**: `{Operation}Request` (e.g., `ScrapeRequest`)
- **Response types**: `{Operation}Response` (e.g., `ScrapeResponse`)
- **Enums**: `PascalCase` with `StrEnum` for value serialization
- **Fields**: `snake_case` (Python convention)
  - JS `waitForSelector` → Python `wait_for_selector`
  - JS `onlyMainContent` → Python `only_main_content`

## Publishing to PyPI

### Release Process

1. **Version bump**: Update `version` in `pyproject.toml`
2. **Build**: `python -m build`
3. **Check**: `twine check dist/*`
4. **Upload**: `twine upload dist/*`

### Trusted Publishers (Recommended)

Configure PyPI to use trusted publishers (no tokens needed):

```toml
[tool.hatch.publish.index]
disable = true

# Or via PyPI dashboard: GitHub Actions -> OIDC
```

### CI/CD

```yaml
# .github/workflows/release.yml
name: Release to PyPI

on:
  release:
    types: [published]

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      id-token: write  # Required for trusted publishing
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install build
      - run: python -m build
      - uses: pypa/gh-action-pypi-publish@release/v1
```

## Testing Strategy

### Test Structure

```python
# tests/test_client.py
import pytest
from undetecta import UndetectaClient, ScrapeRequest
from unittest.mock import AsyncMock, patch

@pytest.mark.asyncio
async def test_scrape_success():
    client = UndetectaClient(api_key="test-key")
    # Mock HTTP response
    # Assert response structure
```

### Test Coverage

- Unit tests for each module
- HTTP client mocking
- Error handling tests
- Type validation tests

## Implementation Order

1. **Package setup**: `pyproject.toml`, directory structure
2. **Type definitions**: `_types.py` with Pydantic models
3. **Error classes**: `_errors.py`
4. **HTTP transport**: `_transport/_http.py`, `_transport/_retry.py`
5. **Client**: `_client.py`
6. **API methods**: `_scrape.py`, `_search.py`
7. **Entry point**: `__init__.py` with exports
8. **Tests**: Unit tests for each module
9. **Documentation**: README with examples

## Future Enhancements (Out of Scope for MVP)

1. Webhook support for async jobs
2. Streaming responses
3. Batch operations
4. Context manager support
5. OpenTelemetry integration
6. Sphinx documentation
