# Gmail 429 Fix — Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Eliminate Gmail API 429 errors on listings by chunking batch requests and using cache to skip already-known messages.

**Architecture:** Modify `GmailAdapter.list_messages()` to (1) check the message cache before batching, (2) split remaining IDs into chunks of 25, (3) pause 200ms between chunks, (4) retry any 429'd messages once after 500ms backoff. No changes to output format or calling code.

**Tech Stack:** Existing `ts4k.state.cache` module, `asyncio.sleep` for pauses, `googleapiclient.errors.HttpError` for 429 detection.

---

### Task 1: Add cache-aware batch fetching

**Files:**
- Modify: `src/ts4k/adapters/gmail.py:320-388` (`list_messages` method)

**Step 1: Write the failing test — cache hits skip batch fetch**

Add to `tests/test_gmail_adapter.py`:

```python
class TestGmailAdapterCacheAwareListing:
    """Test that list_messages uses cache to skip API calls."""

    @pytest.mark.asyncio
    async def test_cached_messages_skip_batch_fetch(self, tmp_path, monkeypatch):
        """Messages already in cache should not be re-fetched via batch."""
        adapter = _make_adapter()

        # Seed cache with msg1's header.
        import ts4k.state.cache as cache_mod
        monkeypatch.setattr(cache_mod, "_INDEX_FILE", tmp_path / "index.json")
        monkeypatch.setattr(cache_mod, "_BODIES_DIR", tmp_path / "bodies")
        monkeypatch.setattr(cache_mod, "_CACHE_DIR", tmp_path)
        (tmp_path / "bodies").mkdir()

        cached_header = {
            "id": "g:msg1",
            "raw_id": "msg1",
            "thread_id": "g:t1",
            "raw_thread_id": "t1",
            "from": "Cached Sender <cached@test.com>",
            "subject": "Cached Subject",
            "date": "2026-02-20T09:15:00Z",
            "snippet": "cached snippet",
            "source": "g",
        }
        cache_mod.store_header("g:msg1", cached_header)

        # Mock messages.list to return 2 IDs (msg1 cached, msg2 not).
        list_result = {"messages": [{"id": "msg1"}, {"id": "msg2"}]}
        adapter._service.users().messages().list.return_value.execute = MagicMock(
            return_value=list_result
        )

        # Mock batch — should only fetch msg2.
        fetched_ids = []

        class MockBatch:
            def __init__(self, callback):
                self.callback = callback
                self.requests = []
            def add(self, request):
                self.requests.append(request)
            def execute(self):
                for i, _req in enumerate(self.requests):
                    fetched_ids.append("msg2")  # only msg2 should appear
                    resp = _make_api_message(
                        msg_id="msg2", subject="Fetched Subject",
                        internal_date=EPOCH_MS_2026_02_19_14_30, format="metadata",
                    )
                    self.callback(str(i), resp, None)

        adapter._service.new_batch_http_request = lambda callback: MockBatch(callback)

        results = await adapter.list_messages("newer_than:1d")

        assert len(results) == 2
        assert len(fetched_ids) == 1  # Only msg2 was batch-fetched
        # Cached msg should appear in results with cached data.
        cached_result = [r for r in results if r["raw_id"] == "msg1"][0]
        assert cached_result["from"] == "Cached Sender <cached@test.com>"
```

**Step 2: Run test to verify it fails**

Run: `uv run pytest tests/test_gmail_adapter.py::TestGmailAdapterCacheAwareListing::test_cached_messages_skip_batch_fetch -v`
Expected: FAIL — `list_messages` doesn't check cache yet.

**Step 3: Write the failing test — chunked batching**

Add to the same class:

```python
    @pytest.mark.asyncio
    async def test_large_listing_chunked(self, tmp_path, monkeypatch):
        """50 messages should be fetched in 2 chunks of 25, not one batch of 50."""
        import ts4k.state.cache as cache_mod
        monkeypatch.setattr(cache_mod, "_INDEX_FILE", tmp_path / "index.json")
        monkeypatch.setattr(cache_mod, "_BODIES_DIR", tmp_path / "bodies")
        monkeypatch.setattr(cache_mod, "_CACHE_DIR", tmp_path)
        (tmp_path / "bodies").mkdir()

        adapter = _make_adapter()

        # 50 message IDs.
        list_result = {"messages": [{"id": f"msg{i}"} for i in range(50)]}
        adapter._service.users().messages().list.return_value.execute = MagicMock(
            return_value=list_result
        )

        batch_call_count = [0]
        batch_sizes = []

        class MockBatch:
            def __init__(self, callback):
                self.callback = callback
                self.requests = []
            def add(self, request):
                self.requests.append(request)
            def execute(self):
                batch_call_count[0] += 1
                batch_sizes.append(len(self.requests))
                for i, _req in enumerate(self.requests):
                    idx = sum(batch_sizes[:-1]) + i
                    resp = _make_api_message(
                        msg_id=f"msg{idx}",
                        subject=f"Subject {idx}",
                        internal_date=str(1771578900000 - idx * 60000),
                        format="metadata",
                    )
                    self.callback(str(i), resp, None)

        adapter._service.new_batch_http_request = lambda callback: MockBatch(callback)

        results = await adapter.list_messages("newer_than:7d", count=50)

        assert len(results) == 50
        assert batch_call_count[0] == 2  # 2 chunks
        assert batch_sizes == [25, 25]
```

**Step 4: Run test to verify it fails**

Run: `uv run pytest tests/test_gmail_adapter.py::TestGmailAdapterCacheAwareListing::test_large_listing_chunked -v`
Expected: FAIL — `list_messages` sends one batch of 50.

**Step 5: Write the failing test — retry on 429**

```python
    @pytest.mark.asyncio
    async def test_429_errors_retried(self, tmp_path, monkeypatch):
        """Messages that get 429'd in the first batch should be retried."""
        import ts4k.state.cache as cache_mod
        monkeypatch.setattr(cache_mod, "_INDEX_FILE", tmp_path / "index.json")
        monkeypatch.setattr(cache_mod, "_BODIES_DIR", tmp_path / "bodies")
        monkeypatch.setattr(cache_mod, "_CACHE_DIR", tmp_path)
        (tmp_path / "bodies").mkdir()

        adapter = _make_adapter()

        list_result = {"messages": [{"id": "msg1"}, {"id": "msg2"}, {"id": "msg3"}]}
        adapter._service.users().messages().list.return_value.execute = MagicMock(
            return_value=list_result
        )

        from googleapiclient.errors import HttpError
        mock_resp = MagicMock()
        mock_resp.status = 429
        mock_resp.reason = "Too Many Requests"
        error_429 = HttpError(mock_resp, b'{"error": {"message": "Rate limit"}}')

        call_count = [0]

        class MockBatch:
            def __init__(self, callback):
                self.callback = callback
                self.requests = []
            def add(self, request):
                self.requests.append(request)
            def execute(self):
                call_count[0] += 1
                for i, _req in enumerate(self.requests):
                    if call_count[0] == 1 and i == 1:
                        # msg2 gets 429 on first attempt
                        self.callback(str(i), None, error_429)
                    else:
                        idx = i if call_count[0] == 1 else 1  # retry batch has msg2
                        msg_id = ["msg1", "msg2", "msg3"][idx] if call_count[0] == 1 else "msg2"
                        resp = _make_api_message(
                            msg_id=msg_id,
                            subject=f"Subject {msg_id}",
                            internal_date=str(1771578900000 - i * 60000),
                            format="metadata",
                        )
                        self.callback(str(i), resp, None)

        adapter._service.new_batch_http_request = lambda callback: MockBatch(callback)

        results = await adapter.list_messages("newer_than:1d")

        assert len(results) == 3  # All 3 returned, none silently dropped
        assert call_count[0] == 2  # Original batch + 1 retry batch
        result_ids = {r["raw_id"] for r in results}
        assert result_ids == {"msg1", "msg2", "msg3"}
```

**Step 6: Run test to verify it fails**

Run: `uv run pytest tests/test_gmail_adapter.py::TestGmailAdapterCacheAwareListing::test_429_errors_retried -v`
Expected: FAIL — no retry logic yet.

**Step 7: Implement cache-aware chunked batch with retry**

In `src/ts4k/adapters/gmail.py`, replace `list_messages` (lines 320-388) with:

```python
    _BATCH_CHUNK_SIZE = 25
    _BATCH_PAUSE_SECS = 0.2
    _RETRY_PAUSE_SECS = 0.5

    async def list_messages(
        self,
        query: str | None = None,
        count: int = 20,
        page_token: str | None = None,
    ) -> list[dict]:
        """Search Gmail and return a list of message-header dicts.

        Efficient three-layer fetch:
        1. messages.list() -> IDs only
        2. Cache check — return cached headers for known IDs
        3. Chunked batch messages.get(format=metadata) for cache misses,
           with retry on 429 errors
        """
        service = self._require_service()

        # Step 1: Get message IDs.
        list_args: dict[str, Any] = {
            "userId": "me",
            "q": query or "in:inbox",
            "maxResults": count,
        }
        if page_token:
            list_args["pageToken"] = page_token

        list_result = await asyncio.to_thread(
            lambda: service.users().messages().list(**list_args).execute()
        )

        message_ids = [m["id"] for m in list_result.get("messages", [])]
        if not message_ids:
            return []

        # Step 2: Check cache for known headers.
        from ts4k.state import cache

        results: list[dict] = []
        uncached_ids: list[str] = []

        for msg_id in message_ids:
            prefixed = f"{self._prefix}:{msg_id}"
            cached = cache.get_header(prefixed)
            if cached:
                results.append(cached)
            else:
                uncached_ids.append(msg_id)

        # Step 3: Batch-fetch uncached IDs in chunks.
        if uncached_ids:
            fetched = await self._chunked_batch_fetch(service, uncached_ids)
            results.extend(fetched)

        # Sort by date descending.
        results.sort(key=lambda m: m.get("date", ""), reverse=True)

        # Attach pagination token.
        next_token = list_result.get("nextPageToken")
        if next_token and results:
            results[-1]["_next_page_token"] = next_token

        return results

    async def _chunked_batch_fetch(
        self, service, msg_ids: list[str]
    ) -> list[dict]:
        """Batch-fetch message metadata in chunks, with 429 retry."""
        all_results: list[dict] = []
        chunks = [
            msg_ids[i : i + self._BATCH_CHUNK_SIZE]
            for i in range(0, len(msg_ids), self._BATCH_CHUNK_SIZE)
        ]

        for chunk_idx, chunk in enumerate(chunks):
            if chunk_idx > 0:
                await asyncio.sleep(self._BATCH_PAUSE_SECS)

            fetched, failed_ids = await self._batch_fetch_chunk(service, chunk)
            all_results.extend(fetched)

            # Retry 429'd messages once after backoff.
            if failed_ids:
                logger.info(
                    "Retrying %d messages after 429 rate limit", len(failed_ids)
                )
                await asyncio.sleep(self._RETRY_PAUSE_SECS)
                retried, still_failed = await self._batch_fetch_chunk(
                    service, failed_ids
                )
                all_results.extend(retried)
                if still_failed:
                    logger.warning(
                        "Failed to fetch %d messages after retry: %s",
                        len(still_failed),
                        still_failed,
                    )

        return all_results

    async def _batch_fetch_chunk(
        self, service, msg_ids: list[str]
    ) -> tuple[list[dict], list[str]]:
        """Fetch one chunk of message metadata via batch API.

        Returns (results, failed_ids) where failed_ids are messages
        that got 429 errors and should be retried.
        """
        results: list[dict] = []
        failed_ids: list[str] = []
        # Map request_id -> msg_id so we can identify failures.
        id_map: dict[str, str] = {}

        def _batch_callback(request_id, response, exception):
            msg_id = id_map.get(request_id, request_id)
            if exception is not None:
                status = getattr(getattr(exception, "resp", None), "status", 0)
                if status == 429:
                    failed_ids.append(msg_id)
                else:
                    logger.warning("Batch metadata fetch error for %s: %s", msg_id, exception)
            else:
                results.append(response)

        batch = service.new_batch_http_request(callback=_batch_callback)
        for i, msg_id in enumerate(msg_ids):
            request_id = str(i)
            id_map[request_id] = msg_id
            batch.add(
                service.users().messages().get(
                    userId="me",
                    id=msg_id,
                    format="metadata",
                    metadataHeaders=["Subject", "From", "Date", "To"],
                ),
                request_id=request_id,
            )
        await asyncio.to_thread(batch.execute)

        # Convert to header dicts.
        header_dicts = [_msg_to_headers(msg, self._prefix) for msg in results]
        return header_dicts, failed_ids
```

**Step 8: Run all three new tests**

Run: `uv run pytest tests/test_gmail_adapter.py::TestGmailAdapterCacheAwareListing -v`
Expected: All 3 PASS.

**Step 9: Run the full test suite**

Run: `uv run pytest tests/ -x -q`
Expected: All pass, no regressions. The existing `TestGmailAdapterListMessages` tests should still pass since they use <=2 messages (under the chunk size).

**Step 10: Commit**

```bash
git add src/ts4k/adapters/gmail.py tests/test_gmail_adapter.py
git commit -m "Fix Gmail 429: chunked batch fetch with cache check and retry

Closes #12"
```
