# Task Recon: Implement retry logic with tenacity for 429 and 5xx handling

**Task ID:** 7.2
**Status:** pending
**Generated:** 2026-01-28T23:50:00Z
**Agent:** claude-opus-4-5-20251101

---

## Task Summary

Add retry logic using tenacity library with exponential backoff for transient failures (429, 500, 502, 503, 504), proper Retry-After header parsing for 429 responses, and 401 token refresh with single retry.

### Parent Task Context

**Parent Task:** Implement GraphClient with Retry Logic (ID: 7)
**Parent Goal:** Create the HTTP client for Microsoft Graph API with retry/backoff, throttling handling, and Pydantic validation.
**This Subtask's Role:** Enhance the existing `_request` method with tenacity retry decorator to handle transient failures automatically, making the GraphClient resilient to API throttling and server errors.

### Dependencies
- Depends on: 7.1 (Implement httpx async client with auth header injection) - **DONE**

### Acceptance Criteria
1. Retry on status codes: 429, 500, 502, 503, 504
2. Use Retry-After header for 429 responses when present
3. Exponential backoff: 1s, 2s, 4s (max 3 retries)
4. Refresh token on 401 and retry once
5. Raise RateLimitError after max retries on 429
6. Raise TeamsPhoneError after max retries on 5xx
7. Log retry attempts with before_sleep callback

---

## Insights from Previous Tasks

### Dependency Task Notes
| Task ID | Title | Key Insight |
|---------|-------|-------------|
| 7.1 | Implement httpx async client with auth header injection | GraphClient base implementation complete with `_request`, `get`, `post`, context manager |

### Reusable Patterns from Dependencies
- **Async context manager pattern**: Implemented in `src/teams_phone/infrastructure/graph_client.py:136-164` during Task 7.1
- **Header injection pattern**: `_get_headers()` returns dict with Bearer token at line 56-66
- **respx mocking pattern**: Used throughout `tests/unit/test_graph_client.py` for HTTP mocking

### Warnings and Gotchas
- **Token acquisition is sync**: `auth_service.get_token()` is synchronous - called from `_get_headers()` which is also sync. For 401 retry, use `auth_service.refresh_token()` instead.
- **raise_for_status()**: Both `get()` and `post()` call `response.raise_for_status()` - error mapping will be added in 7.4. Retry logic should happen in `_request` before response is returned.
- **respx for HTTP mocking**: Use respx library (already in project) instead of responses for httpx compatibility.

### Open Items Inherited
- [ ] Subtask 7.3: Add pagination support with `get_paginated()` method
- [ ] Subtask 7.4: Add error mapping to exception hierarchy

---

## Architecture Context

### Relevant Design Decisions
- **Retry Policy** (from `.taskmaster/docs/architecture/11-error-handling-strategy.md:46-53`):
  | HTTP Status | Retry? | Strategy |
  |-------------|--------|----------|
  | 429 (Throttled) | Yes | Use Retry-After header, max 3 retries |
  | 500, 502, 503, 504 | Yes | Exponential backoff: 1s, 2s, 4s |
  | 401 (Unauthorized) | Once | Refresh token, retry once |
  | 400, 403, 404 | No | Fail immediately |

- **GraphClient Key Behaviors** (from `.taskmaster/docs/architecture/5-components.md:124-129`):
  - Automatic retry on 429 (throttled) and 503 (service unavailable)
  - Exponential backoff: 3 retries with 1s, 2s, 4s delays
  - Request timeout: 30 seconds default
  - Automatic token refresh on 401

### Module Structure
```
src/teams_phone/infrastructure/
    graph_client.py  <- Modify this file to add retry logic
```

### Conventions to Follow
- Use constants from `teams_phone.constants` (MAX_RETRIES=3, DEFAULT_TIMEOUT=30)
- Raise exceptions from `teams_phone.exceptions` hierarchy
- All retry logic should be in `_request` method
- Use type hints throughout (mypy strict mode)
- Follow existing docstring conventions

---

## Related Code References

### Primary References (Must Read)
| File | Lines | Purpose |
|------|-------|---------|
| `src/teams_phone/infrastructure/graph_client.py` | 68-93 | Current `_request` method to enhance with retry |
| `src/teams_phone/exceptions.py` | 98-110 | RateLimitError to raise after 429 retry exhaustion |
| `src/teams_phone/exceptions.py` | 10-45 | TeamsPhoneError base class with remediation pattern |
| `src/teams_phone/constants.py` | 41-42 | MAX_RETRIES=3 constant to use |
| `src/teams_phone/services/auth_service.py` | 282-333 | `refresh_token()` method for 401 handling |

### Secondary References (Helpful Context)
| File | Lines | Purpose |
|------|-------|---------|
| `.taskmaster/docs/architecture/11-error-handling-strategy.md` | 45-53 | Retry policy specification |
| `.taskmaster/docs/architecture/7-core-workflows.md` | 183-204 | Error handling flow (API throttling) diagram |
| `.taskmaster/docs/architecture/6-external-apis.md` | 9 | Graph API rate limits return 429 with Retry-After header |
| `tests/unit/test_graph_client.py` | 159-193 | Existing test patterns for `_request` |

### Code Patterns to Follow

```python
# Pattern from graph_client.py:68-93 - current _request method to enhance
async def _request(
    self,
    method: str,
    url: str,
    **kwargs: Any,
) -> httpx.Response:
    """Make an async HTTP request.

    Args:
        method: HTTP method (GET, POST, etc.).
        url: Full URL for the request.
        **kwargs: Additional arguments passed to httpx.AsyncClient.request().

    Returns:
        The httpx.Response object.

    Raises:
        RuntimeError: If client is not initialized (not in async context).
    """
    if self._client is None:
        raise RuntimeError(
            "GraphClient must be used as async context manager. "
            "Use 'async with GraphClient(...) as client:'"
        )
    headers = self._get_headers()
    return await self._client.request(method, url, headers=headers, **kwargs)
```

```python
# Exception pattern from exceptions.py:98-110
class RateLimitError(TeamsPhoneError):
    """Rate limit exceeded (exit code 6).

    Raised when Graph API returns HTTP 429 (throttled):
    - Too many requests in a time window
    - Tenant-level throttling limits reached

    Note: The CLI will automatically retry with exponential backoff
    before raising this exception.
    """

    exit_code: int = ExitCode.RATE_LIMIT
```

```python
# Test pattern from test_graph_client.py:175-192
@pytest.mark.unit
@pytest.mark.asyncio
async def test_makes_request_with_headers(
    self, mock_auth_service: MagicMock
) -> None:
    """_request includes authentication headers."""
    with respx.mock:
        route = respx.get(f"{GRAPH_API_BASE}/users").mock(
            return_value=httpx.Response(200, json={"value": []})
        )

        async with GraphClient(mock_auth_service) as client:
            response = await client._request("GET", f"{GRAPH_API_BASE}/users")

        assert route.called
        assert response.status_code == 200
```

---

## Dependencies

### Internal Modules
- `teams_phone.constants` - MAX_RETRIES (3), provides retry count
- `teams_phone.exceptions` - RateLimitError, AuthenticationError, TeamsPhoneError for error raising
- `teams_phone.services.auth_service.AuthService` - `refresh_token()` method for 401 handling

### External Packages
- `httpx>=0.25.0` - HTTP client (already installed)
- `tenacity` - **NOT INSTALLED** - Must be added to pyproject.toml

### New Dependencies Needed
- `tenacity>=8.0.0` - Add to `pyproject.toml` dependencies:
  ```toml
  dependencies = [
      ...
      "tenacity>=8.0.0",
  ]
  ```

---

## Testing Guidance

### Test File Location
`tests/unit/test_graph_client.py` - Add new test class for retry logic

### Test Patterns in Use
```python
# Pattern from test_graph_client.py:15-20 - Mock AuthService fixture
@pytest.fixture
def mock_auth_service(tmp_path: Path) -> MagicMock:
    """Create a mock AuthService that returns a test token."""
    mock_service = MagicMock(spec=AuthService)
    mock_service.get_token.return_value = "test-access-token-12345"
    return mock_service
```

```python
# Pattern for testing HTTP responses with respx
@pytest.mark.unit
@pytest.mark.asyncio
async def test_get_raises_on_error(self, mock_auth_service: MagicMock) -> None:
    """get method raises HTTPStatusError on non-2xx response."""
    with respx.mock:
        respx.get(f"{GRAPH_API_BASE}/users").mock(
            return_value=httpx.Response(404, json={"error": {"message": "Not found"}})
        )

        async with GraphClient(mock_auth_service) as client:
            with pytest.raises(httpx.HTTPStatusError):
                await client.get("/users")
```

### Fixtures Available
- `mock_auth_service` - MagicMock of AuthService with `get_token()` returning test token
- `tmp_path` - pytest built-in for temporary directory

### Test Cases to Implement
1. Test retry on 429 with Retry-After header (integer seconds)
2. Test retry on 429 with Retry-After header (HTTP-date format)
3. Test retry on 429 without Retry-After header (use exponential backoff)
4. Test retry on each 5xx code (500, 502, 503, 504)
5. Test exponential backoff timing (1s, 2s, 4s)
6. Test 401 token refresh flow - refresh called, retry succeeds
7. Test 401 double failure - raises AuthenticationError
8. Test max retry exhaustion on 429 - raises RateLimitError
9. Test max retry exhaustion on 5xx - raises TeamsPhoneError
10. Test no retry on 400, 403, 404

### Test Utilities
- `tests/conftest.py` - Contains marker registration (unit, integration, contract)
- `respx` - For mocking httpx async client responses

---

## Risks and Blockers

### Technical Risks
- [ ] **tenacity not installed**: Must add `tenacity>=8.0.0` to `pyproject.toml` and run `uv pip install -e ".[dev]"` before implementation
- [ ] **HTTP-date parsing**: Retry-After header can be in HTTP-date format (e.g., "Wed, 21 Oct 2015 07:28:00 GMT"). Need to handle both integer seconds and HTTP-date.
- [ ] **Async compatibility**: tenacity works with async functions via `@retry` decorator, but need to ensure proper async handling in custom wait callback

### Dependency Risks
- [ ] Task 7.4 (error mapping) depends on this work being done correctly - retry exhaustion must raise appropriate exceptions

### Open Questions
- [ ] Should retry logging use Python's `logging` module or be silent? Architecture doc mentions "Log warning, wait X seconds" in workflow diagram
- [ ] Should httpx connection/timeout errors also trigger retries? Task details mention `_should_retry_exception` for httpx errors

---

## Implementation Recommendations

### Suggested Approach
1. **Add tenacity dependency** to pyproject.toml and install
2. **Create helper functions** at module level (before class):
   - `_is_retryable_status(response: httpx.Response) -> bool`
   - `_get_retry_after(response: httpx.Response) -> float | None`
3. **Create custom wait callback** that checks Retry-After header
4. **Create before_sleep callback** for logging retry attempts
5. **Wrap _request with retry logic** - use tenacity decorator or manual retry loop
6. **Handle 401 separately** - not via tenacity, use explicit single-retry logic with token refresh
7. **Add comprehensive tests** for all retry scenarios

### Files to Create/Modify
| Action | File | Reason |
|--------|------|--------|
| Modify | `pyproject.toml` | Add tenacity dependency |
| Modify | `src/teams_phone/infrastructure/graph_client.py` | Add retry logic to _request |
| Modify | `tests/unit/test_graph_client.py` | Add retry tests |

### Implementation Strategy

```python
# Recommended structure for retry logic in graph_client.py

import logging
from email.utils import parsedate_to_datetime
from tenacity import (
    retry,
    retry_if_result,
    stop_after_attempt,
    wait_exponential,
    before_sleep_log,
    RetryCallState,
)

from teams_phone.constants import MAX_RETRIES
from teams_phone.exceptions import AuthenticationError, RateLimitError, TeamsPhoneError

logger = logging.getLogger(__name__)

# Retryable status codes
RETRYABLE_STATUS_CODES = {429, 500, 502, 503, 504}


def _is_retryable_status(response: httpx.Response) -> bool:
    """Check if response status code should trigger a retry."""
    return response.status_code in RETRYABLE_STATUS_CODES


def _get_retry_after(response: httpx.Response) -> float | None:
    """Parse Retry-After header value.

    Handles both integer seconds and HTTP-date formats.
    Returns None if header is missing or unparseable.
    """
    retry_after = response.headers.get("Retry-After")
    if retry_after is None:
        return None

    # Try integer seconds first
    try:
        return float(retry_after)
    except ValueError:
        pass

    # Try HTTP-date format
    try:
        retry_date = parsedate_to_datetime(retry_after)
        from datetime import datetime, timezone
        now = datetime.now(timezone.utc)
        delta = (retry_date - now).total_seconds()
        return max(0.0, delta)
    except (ValueError, TypeError):
        return None


def _custom_wait(retry_state: RetryCallState) -> float:
    """Custom wait that uses Retry-After header if available."""
    outcome = retry_state.outcome
    if outcome and not outcome.failed:
        response = outcome.result()
        if response.status_code == 429:
            retry_after = _get_retry_after(response)
            if retry_after is not None:
                return retry_after

    # Fallback to exponential backoff: 1s, 2s, 4s
    return min(4.0, 2 ** (retry_state.attempt_number - 1))
```

### Quick Wins
- Start by adding tenacity to pyproject.toml and verifying import works
- Implement `_get_retry_after` helper first - it's self-contained and testable
- Use `before_sleep_log(logger, logging.WARNING)` from tenacity for logging

---

## Cheat Sheet

### Key Imports
```python
from tenacity import (
    retry,
    retry_if_result,
    stop_after_attempt,
    wait_exponential,
    before_sleep_log,
    RetryCallState,
)
from email.utils import parsedate_to_datetime
import logging

from teams_phone.constants import MAX_RETRIES
from teams_phone.exceptions import AuthenticationError, RateLimitError, TeamsPhoneError
```

### Key Functions to Use
- `response.headers.get("Retry-After")` - Get retry header value
- `auth_service.refresh_token()` - Refresh token on 401 (synchronous)
- `parsedate_to_datetime(date_string)` - Parse HTTP-date format

### Common Patterns
- **Retry decorator with custom condition**:
  ```python
  @retry(retry=retry_if_result(_is_retryable_status), stop=stop_after_attempt(MAX_RETRIES))
  ```
- **Custom wait using callback**:
  ```python
  @retry(wait=_custom_wait)
  ```
- **Logging retries**:
  ```python
  @retry(before_sleep=before_sleep_log(logger, logging.WARNING))
  ```

### Tenacity Key Concepts
- `retry_if_result(predicate)` - Retry if predicate returns True for result
- `stop_after_attempt(n)` - Stop after n attempts
- `wait_exponential(multiplier=1, min=1, max=4)` - Exponential backoff
- `before_sleep(callback)` - Called before each retry sleep
- `retry_error_callback(callback)` - Called when all retries exhausted

### Exception Mapping After Retry Exhaustion
```python
# After max retries on 429:
raise RateLimitError(
    "Rate limit exceeded after 3 retries",
    remediation="Wait and retry, or reduce request rate"
)

# After max retries on 5xx:
raise TeamsPhoneError(
    f"Graph API unavailable (HTTP {status_code}) after 3 retries"
)

# After 401 double failure:
raise AuthenticationError(
    "Authentication failed after token refresh",
    remediation="Run `teams-phone auth login` to re-authenticate"
)
```

---

*Generated by task-recon (claude-opus-4-5-20251101) | Ready for implementation*
