# Sprint Change Proposal: Epic-005 Sync Integration

**Date:** 2025-11-24
**Generated By:** Correct Course Workflow
**Trigger:** STORY-037 (Data Serving Layer) - Sync integration gap discovered
**Scope:** Minor - Direct implementation by development team
**Estimated Effort:** 3-4 hours

---

## Executive Summary

Epic-005 implemented Features and Users tables with fully functional repositories, but **sync orchestration was not implemented**. Features table remains empty because nothing populates it during background sync. This proposal adds **STORY-038: Feature Sync Integration** to complete Epic-005 using staleness-based refresh (1-hour TTL).

---

## 1. Issue Summary

### Problem Statement

Epic-005 implemented Features and Users tables with fully functional repositories (`FeatureRepository`, `UserRepository`), but **did not integrate them into the sync orchestration layer** (background sync + CLI sync). As a result:

- **Features table remains empty** - No product catalog visibility (core epic goal unmet)
- **Users table incompletely populated** - Only extracted during bug refresh for mutable tests
- **MCP tools return empty results** - `list_features`, `list_users` have no data to serve
- **Epic-005 cannot be validated** - Infrastructure complete but deliverables untestable

### Discovery Context

Issue discovered during STORY-037 review (Data Serving Layer). While implementing MCP tools to expose features and users, we found that:
1. ✅ Repositories exist and are functionally correct
2. ✅ Sync methods (`FeatureRepository.sync_features()`) work when called directly
3. ❌ Nothing in `cache.py` orchestration calls these sync methods
4. ❌ Background sync only syncs: Products → Tests → Bugs (Features and Users omitted)

---

## 2. Impact Analysis

### Epic Impact

**Epic-005 Status:** Incomplete - requires one additional story

**Current State:**
- ✅ STORY-035A: Features Repository & Sync (repository functional, not wired)
- ✅ STORY-036: User Metadata Extraction (extraction logic works, not comprehensive)
- ✅ STORY-037: Data Serving Layer (MCP tools exist, return empty data)
- ❌ **Missing:** Sync orchestration integration

**Required Additions:**
- **STORY-038: Feature Sync Integration** (3-4 hours)
- Priority: **HIGH** (blocks Epic-005 completion)

### Artifact Changes

**Documentation:**
- ✅ `docs/epics/epic-005-data-enhancement-and-serving.md` - Updated with STORY-038
- ✅ `docs/architecture/adrs/ADR-015-feature-staleness-and-sync-strategy.md` - Created
- ✅ `CLAUDE.md` - Updated with 3-phase sync flow
- ✅ `.env.example` - Added `FEATURE_CACHE_TTL_SECONDS`

**Implementation (Pending):**
- Database: Add `features_synced_at` column to `products` table
- Config: Add `FEATURE_CACHE_TTL_SECONDS` to `config.py`
- Sync: Implement `PersistentCache.refresh_features()` method
- Sync: Integrate Phase 3 into `run_background_refresh()`
- Tools: Update `list_features` with staleness check
- Repository: Add `update_features_last_synced()` helper
- Testing: Unit + integration tests

---

## 3. Recommended Approach

### Selected Path: Direct Adjustment

**Approach:** Add STORY-038 to complete Epic-005 sync integration

**Why This Approach:**

✅ **Minimal Effort** (3-4 hours total)
- Clear implementation pattern (follows bug staleness from STORY-024)
- No refactoring of existing code
- Well-understood architectural approach

✅ **Low Risk**
- Pattern already proven with bugs
- FeatureRepository already tested and working
- No breaking changes to existing APIs or tools

✅ **Positive Momentum**
- Framing: "Completing Epic-005" not "Fixing mistakes"
- Builds on successful work
- Quick win to full epic delivery

✅ **High Value**
- Unblocks Epic-005 validation
- Makes MCP tools functional
- No scope reduction or timeline impact

---

## 4. Technical Strategy

### Sync Pattern: Staleness-Based with 1-Hour TTL

**Two refresh paths:**

1. **Background Sync** (Automatic, every 15 min)
   - Phase 1: Discover new tests
   - Phase 2: Refresh mutable tests + bugs
   - Phase 3 (NEW): **Refresh features if stale**
     - Check `products.features_synced_at`
     - If stale (> `FEATURE_CACHE_TTL_SECONDS`) OR NULL → Refresh from API
     - If fresh → Skip (already current)

2. **Tool Calls** (On-demand)
   - User calls `list_features(product_id)`
   - Check `products.features_synced_at`
   - If stale OR NULL → Refresh from API
   - If fresh → Return from cache
   - `force_refresh=True` bypasses staleness check

### Configuration

```bash
FEATURE_CACHE_TTL_SECONDS=3600  # 1 hour (same as bugs)
```

**Rationale for 1-hour TTL:**
- Features change infrequently (vs bugs during active testing)
- Balances freshness vs API efficiency
- Predictable API load (1 call per product per hour)
- Users can force refresh if needed

---

## 5. Implementation Details

### STORY-038: Feature Sync Integration

**Acceptance Criteria:**

**Schema:**
1. [ ] Add `features_synced_at` column to `products` table (nullable datetime)
2. [ ] Alembic migration generated and tested

**Configuration:**
3. [ ] `FEATURE_CACHE_TTL_SECONDS` added to `config.py` (default: 3600)
4. [ ] `FEATURE_CACHE_TTL_SECONDS` added to `.env.example` ✅ (completed)

**Background Sync:**
5. [ ] `PersistentCache.refresh_features()` method implemented
6. [ ] Checks staleness using configurable TTL (not hardcoded 1 hour)
7. [ ] Integrated as Phase 3 in `run_background_refresh()`
8. [ ] Skips refresh if features fresh (< TTL)
9. [ ] Error handling (log errors, continue with next product)

**Tool Integration:**
10. [ ] `list_features` checks staleness before returning cached data
11. [ ] `force_refresh=True` parameter bypasses staleness check
12. [ ] Updates `products.features_synced_at` after refresh

**Repository Helper:**
13. [ ] `ProductRepository.update_features_last_synced()` method

**Testing:**
14. [ ] Unit tests: Staleness logic (mock scenarios)
15. [ ] Integration tests: Background sync refreshes features
16. [ ] Integration tests: Tool staleness check behavior
17. [ ] Integration tests: Force refresh bypasses cache

**Documentation:**
18. [x] ADR-015 created ✅
19. [x] CLAUDE.md updated ✅
20. [x] Epic-005 updated ✅

**Validation:**
21. [ ] Background sync logs show features being refreshed
22. [ ] `list_features` tool returns populated data

---

## 6. Code Changes Preview

### Change 1: Background Sync Method

```python
# src/testio_mcp/database/cache.py

async def refresh_features(self, product_id: int) -> dict[str, Any]:
    """Refresh features for product if stale.

    Checks products.features_synced_at timestamp. If stale or NULL,
    refreshes from API. Otherwise skips (already fresh).
    """
    from testio_mcp.config import get_settings

    settings = get_settings()

    # Check staleness
    product = await product_repo.get_product(product_id)
    if product and product.features_synced_at:
        seconds_since_sync = (now - product.features_synced_at).total_seconds()

        # ✅ Use configurable TTL, not hardcoded
        if seconds_since_sync < settings.FEATURE_CACHE_TTL_SECONDS:
            logger.debug(f"Features fresh for product {product_id}, skipping")
            return {"created": 0, "updated": 0, "total": 0, "skipped": True}

    # Stale or NULL - refresh from API
    await feature_repo.sync_features(product_id)
    await product_repo.update_features_last_synced(product_id)

    return {**result, "skipped": False}
```

### Change 2: Background Sync Integration

```python
# src/testio_mcp/database/cache.py - run_background_refresh()

# Phase 3 (NEW): Refresh features
total_features_synced = 0
for product in products_info:
    try:
        features_result = await self.refresh_features(product_id)
        if not features_result["skipped"]:
            total_features_synced += features_result["total"]
    except Exception as e:
        logger.error(f"Failed to refresh features for product {product_id}: {e}")
        # Continue with next product
```

### Change 3: Tool Staleness Check

```python
# src/testio_mcp/tools/list_features_tool.py

@mcp.tool()
async def list_features(
    product_id: int,
    section_id: int | None = None,
    force_refresh: bool = False,
    ctx: Context = None,
) -> dict:
    """List features with staleness check."""
    from testio_mcp.config import get_settings

    settings = get_settings()
    cache = ctx.server_context["cache"]

    # Check staleness (unless force_refresh)
    if not force_refresh:
        product = await product_repo.get_product(product_id)
        if product and product.features_synced_at:
            seconds_since_sync = (now - product.features_synced_at).total_seconds()

            # ✅ Use configurable TTL
            if seconds_since_sync < settings.FEATURE_CACHE_TTL_SECONDS:
                logger.debug("Using cached features")
            else:
                await cache.refresh_features(product_id)
        else:
            await cache.refresh_features(product_id)
    else:
        await cache.refresh_features(product_id)

    # Return from SQLite
    async with get_service_context(ctx, FeatureService) as service:
        return await service.list_features(product_id, section_id)
```

---

## 7. Success Criteria

**Functional:**
- ✅ Background sync logs show features refreshing when stale
- ✅ Background sync skips features when fresh (< 1 hour)
- ✅ `list_features` returns populated data
- ✅ Tool respects staleness (cache hits when fresh)
- ✅ `force_refresh=True` bypasses cache

**Testing:**
- ✅ All unit tests pass
- ✅ All integration tests pass
- ✅ Manual MCP Inspector verification

**Validation:**
- ✅ Epic-005 goals achieved (catalog visibility)
- ✅ No regressions (existing tests pass)

---

## 8. Implementation Handoff

**Change Scope:** Minor - Direct implementation by development team

**Timeline:** 3-4 hours
- Implementation: 2 hours
- Testing: 1-1.5 hours
- Documentation: 30 minutes (completed)

**No Dependencies:** Work can proceed immediately

**Implementation Sequence:**
1. Schema migration (`alembic revision --autogenerate`)
2. Configuration (`FEATURE_CACHE_TTL_SECONDS` in `config.py`)
3. Background sync (`refresh_features()` + Phase 3 integration)
4. Tool staleness (`list_features` update)
5. Repository helper (`update_features_last_synced()`)
6. Testing (unit + integration)

**Validation:**
- Run background sync, verify features refresh in logs
- Call `list_features` tool via MCP Inspector, verify data returned
- Verify staleness behavior (cache hit vs API call)

---

## 9. References

- **Epic-005:** `docs/epics/epic-005-data-enhancement-and-serving.md`
- **ADR-013:** User Story Embedding Strategy
- **ADR-015:** Feature Staleness and Sync Strategy (created)
- **STORY-024:** Intelligent Bug Caching (pattern precedent)
- **Correct Course Checklist:** `.bmad/bmm/workflows/4-implementation/correct-course/checklist.md`

---

**✅ Documentation Updates Complete**

All documentation changes have been applied:
- ✅ Epic-005 updated with STORY-038
- ✅ ADR-015 created
- ✅ CLAUDE.md updated with sync flow
- ✅ .env.example updated with FEATURE_CACHE_TTL_SECONDS

**Ready for implementation of STORY-038.**
