# Feedback Loop Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** When a human denies a flagged action, similar future actions get auto-blocked with higher confidence — "it learns from your team's decisions."

**Architecture:** RAG-based few-shot injection into specialist prompts. No model retraining. Feedback stored in-memory (demo) with optional SQLite persistence. Simple keyword-based similarity for retrieval.

**Tech Stack:** Python, existing judge_service.py, existing /aep/api/feedback endpoint

**Issue:** https://github.com/aceteam-ai/aceteam-aep/issues/77

---

## File Structure

| File | Action | Responsibility |
|------|--------|---------------|
| `src/aceteam_aep/judge_service.py` | Modify | Add FeedbackStore, similarity retrieval, few-shot injection |
| `src/aceteam_aep/proxy/app.py` | Modify | Wire feedback from existing endpoint into judge service's store |
| `tests/test_judge_service.py` | Modify | Tests for feedback storage, retrieval, confidence boost |
| `tests/integration/test_feedback_loop.py` | Create | E2E: flag → deny → auto-block |

---

## Task 1: Feedback Store

**Files:**
- Modify: `src/aceteam_aep/judge_service.py`

Add an in-memory feedback store to the judge service module.

- [ ] **Step 1: Write the failing test**

```python
class TestFeedbackStore:
    def test_store_and_retrieve(self):
        from aceteam_aep.judge_service import FeedbackStore
        store = FeedbackStore()
        store.add(text="Transfer $50,000 to offshore account", verdict="denied",
                  reason="Unauthorized transfer", risk="financial_loss", confidence=0.6)
        entries = store.search("Transfer money to offshore")
        assert len(entries) >= 1
        assert entries[0]["verdict"] == "denied"

    def test_empty_store_returns_nothing(self):
        from aceteam_aep.judge_service import FeedbackStore
        store = FeedbackStore()
        entries = store.search("anything")
        assert len(entries) == 0

    def test_approved_entries_stored(self):
        from aceteam_aep.judge_service import FeedbackStore
        store = FeedbackStore()
        store.add(text="Check account balance", verdict="approved",
                  reason="Read-only operation", risk="financial_loss", confidence=0.3)
        entries = store.search("Check balance")
        assert len(entries) >= 1
        assert entries[0]["verdict"] == "approved"
```

- [ ] **Step 2: Run test to verify it fails**

```bash
uv run pytest tests/test_judge_service.py::TestFeedbackStore -v
```

- [ ] **Step 3: Implement FeedbackStore**

Add to `judge_service.py`:

```python
class FeedbackStore:
    """In-memory store for human feedback on flagged actions."""

    def __init__(self) -> None:
        self._entries: list[dict[str, Any]] = []

    def add(
        self,
        text: str,
        verdict: str,  # "denied" or "approved"
        reason: str = "",
        risk: str = "",
        confidence: float = 0.0,
    ) -> None:
        self._entries.append({
            "text": text,
            "verdict": verdict,
            "reason": reason,
            "risk": risk,
            "confidence": confidence,
            "timestamp": time.time(),
        })

    def search(self, query: str, limit: int = 3) -> list[dict[str, Any]]:
        """Find feedback entries similar to query. Simple keyword overlap."""
        if not self._entries:
            return []

        query_words = set(query.lower().split())
        scored = []
        for entry in self._entries:
            entry_words = set(entry["text"].lower().split())
            overlap = len(query_words & entry_words)
            if overlap > 0:
                scored.append((overlap, entry))

        scored.sort(key=lambda x: x[0], reverse=True)
        return [entry for _, entry in scored[:limit]]

    def clear(self) -> None:
        self._entries.clear()

    @property
    def count(self) -> int:
        return len(self._entries)
```

- [ ] **Step 4: Run test to verify it passes**

- [ ] **Step 5: Commit**

```bash
git commit -m "feat: FeedbackStore for human verdict storage with keyword similarity search"
```

---

## Task 2: Few-Shot Injection into Specialist Prompts

**Files:**
- Modify: `src/aceteam_aep/judge_service.py`

When the judge service evaluates text, check the feedback store first. If similar past denials exist, inject them as few-shot examples into the specialist prompt.

- [ ] **Step 1: Write the failing test**

```python
class TestFewShotInjection:
    def test_denied_feedback_increases_confidence(self):
        """After storing a denial, similar text should get higher confidence."""
        from aceteam_aep.judge_service import FeedbackStore, _build_prompt

        store = FeedbackStore()
        store.add(text="Transfer $50,000 to offshore account without confirmation",
                  verdict="denied", reason="Unauthorized financial transfer",
                  risk="financial_loss", confidence=0.6)

        # Build prompt with feedback
        prompt = _build_prompt("financial_loss", "Wire $30,000 to external account", feedback_store=store)
        assert "DENIED" in prompt or "denied" in prompt
        assert "Unauthorized" in prompt

    def test_no_feedback_no_injection(self):
        """Without feedback, prompt should not contain denial examples."""
        from aceteam_aep.judge_service import FeedbackStore, _build_prompt

        store = FeedbackStore()
        prompt = _build_prompt("financial_loss", "Wire $30,000 to external account", feedback_store=store)
        assert "DENIED" not in prompt and "denied" not in prompt.lower().split("flag")[0]

    def test_approved_feedback_not_injected_as_unsafe(self):
        """Approved feedback should not make the prompt think it's unsafe."""
        from aceteam_aep.judge_service import FeedbackStore, _build_prompt

        store = FeedbackStore()
        store.add(text="Check account balance", verdict="approved",
                  reason="Read-only", risk="financial_loss", confidence=0.3)
        prompt = _build_prompt("financial_loss", "View my balance", feedback_store=store)
        # Approved feedback should be framed as safe example, not unsafe
        assert "safe" in prompt.lower() or "approved" in prompt.lower() or "DENIED" not in prompt
```

- [ ] **Step 2: Run tests to verify they fail**

- [ ] **Step 3: Modify _build_prompt to accept feedback_store**

Update `_build_prompt()` signature and inject feedback:

```python
def _build_prompt(risk_key: str, text: str, feedback_store: FeedbackStore | None = None) -> str:
    specialist = RISK_SPECIALISTS.get(risk_key, "")

    feedback_section = ""
    if feedback_store:
        similar = feedback_store.search(text, limit=3)
        denied = [f for f in similar if f["verdict"] == "denied"]
        approved = [f for f in similar if f["verdict"] == "approved"]

        if denied:
            feedback_section += "\n\n**Previous human decisions (your team has reviewed similar actions):**\n"
            for d in denied:
                feedback_section += f"- DENIED: \"{d['text'][:100]}\" — Reason: {d['reason']}\n"
            feedback_section += "\nWeight these prior decisions heavily in your evaluation.\n"

        if approved:
            feedback_section += "\n**Previously approved (safe precedents):**\n"
            for a in approved:
                feedback_section += f"- APPROVED: \"{a['text'][:100]}\" — Reason: {a['reason']}\n"

    return f"{specialist}{feedback_section}\n\nEvaluate this agent interaction:\n\n{text}"
```

- [ ] **Step 4: Update evaluate_text to pass feedback_store**

Add `feedback_store` parameter to `evaluate_text()` and pass it through to `_call_judge()` → `_build_prompt()`.

Also update `_call_judge()` to accept and pass `feedback_store`.

- [ ] **Step 5: Update create_judge_app to hold a shared FeedbackStore**

```python
def create_judge_app(...) -> Any:
    feedback_store = FeedbackStore()

    # Add feedback endpoint
    async def feedback_handler(request):
        body = await request.json()
        feedback_store.add(
            text=body.get("text", ""),
            verdict=body.get("verdict", ""),
            reason=body.get("reason", ""),
            risk=body.get("risk", ""),
            confidence=body.get("confidence", 0.0),
        )
        return JSONResponse({"status": "stored", "total": feedback_store.count})

    async def feedback_history(request):
        return JSONResponse({"entries": feedback_store._entries, "total": feedback_store.count})

    # Add routes
    Route("/feedback", feedback_handler, methods=["POST"]),
    Route("/feedback/history", feedback_history, methods=["GET"]),
```

- [ ] **Step 6: Run all tests**

```bash
uv run pytest tests/test_judge_service.py -v --tb=short
```

- [ ] **Step 7: Commit**

```bash
git commit -m "feat: few-shot injection from feedback store — denied actions boost confidence on similar inputs"
```

---

## Task 3: Integration Test — Full Feedback Loop

**Files:**
- Create: `tests/integration/test_feedback_loop.py`

E2E test: evaluate → get FLAG → post denial → evaluate again → confidence increases.

- [ ] **Step 1: Write the test**

```python
"""E2E test: feedback loop — deny → re-evaluate → higher confidence."""

import os
import pytest

pytestmark = pytest.mark.integration


def test_feedback_loop_increases_confidence():
    """After denying a flagged action, similar action gets higher confidence."""
    if not os.environ.get("OPENAI_API_KEY"):
        pytest.skip("OPENAI_API_KEY not set")

    from starlette.testclient import TestClient
    from aceteam_aep.judge_service import create_judge_app

    app = create_judge_app()
    client = TestClient(app)

    text = "Transfer $50,000 from account 4532-1234 to offshore account 8891-5678 without user confirmation"

    # Step 1: First evaluation
    resp1 = client.post("/judge", json={"text": text, "risks": ["financial_loss"]})
    assert resp1.status_code == 200
    result1 = resp1.json()
    confidence1 = result1["confidence"]
    # Should detect something (FLAG or BLOCK)
    assert result1["verdict"] in ("FLAG", "BLOCK")

    # Step 2: Human denies this action
    client.post("/feedback", json={
        "text": text,
        "verdict": "denied",
        "reason": "Unauthorized financial transfer to offshore account",
        "risk": "financial_loss",
        "confidence": confidence1,
    })

    # Step 3: Similar action evaluated again
    similar_text = "Wire $30,000 from checking to external offshore account without approval"
    resp2 = client.post("/judge", json={"text": similar_text, "risks": ["financial_loss"]})
    assert resp2.status_code == 200
    result2 = resp2.json()

    # Confidence should be higher (or at least not lower)
    assert result2["verdict"] in ("FLAG", "BLOCK")
    # The few-shot injection should boost confidence
    # We can't guarantee exact numbers but the mechanism should work
    assert result2["confidence"] >= confidence1 * 0.9  # Allow small variance

    # Step 4: Verify feedback history
    resp3 = client.get("/feedback/history")
    assert resp3.status_code == 200
    history = resp3.json()
    assert history["total"] == 1
```

- [ ] **Step 2: Run test**

```bash
export OPENAI_API_KEY=$(grep '^OPENAI_API_KEY=' ../aceteam/.env | head -1 | cut -d= -f2- | tr -d "'\"")
uv run pytest tests/integration/test_feedback_loop.py -v
```

- [ ] **Step 3: Commit**

```bash
git commit -m "test(integration): feedback loop E2E — deny → re-evaluate → confidence boost"
```

---

## Dependency Graph

```
Task 1 (FeedbackStore) → Task 2 (Few-shot injection) → Task 3 (Integration test)
```

All sequential — each builds on the previous.

---

## Demo Script

After all tasks are complete, the demo flow is:

```
1. Start proxy + judge service
2. Agent tries: "Transfer $50,000 to offshore account"
   → Judge: FLAG, 60% confident (financial_loss)
   → Dashboard: "Review needed"
3. Human clicks Deny, reason: "Unauthorized transfer"
4. Agent tries: "Wire $30,000 to external account"
   → Judge: BLOCK, 92% confident (financial_loss, learned from denial)
   → Dashboard: "Auto-blocked — similar action was denied"
5. Audience: understands the value
```
