# Directive: From Insights to Enablers — What to Build Next

## The Problem with What We Have

We analyzed 318 AI coding sessions across 3 Pratilipi developers. The qualitative analysis produced insights like "missing_context is 40% of failures" and "Zzjjaayy has a broken tool integration." The VP feedback was:

1. **"What does this tell about the developer?"** → It doesn't. A developer might know exactly what they're doing but be lazy to prompt well. The analysis can't distinguish skill from effort.
2. **"How does this bring value to AI agents?"** → Right now it doesn't. It's a retrospective report. Another dashboard won't help. What's needed is something that **changes behavior in real-time**.

The core realization: **insights alone are worthless. The value is in the enabler that acts on them.**

---

## Ideation: What's Actually Actionable from Our Data

### What we know (from 150 flagged sessions, 318 total):

| Insight | Data Source | Actionable? |
|---------|------------|-------------|
| Missing context is #1 failure cause (36-40%) | session_narratives | **YES** — we can detect this pattern and intervene |
| `error_paste_and_fix` prompts score 0.25 avg trouble | prompt_buckets | **YES** — we can teach/enforce this pattern |
| Multi-task prompts score 8.67 avg trouble | prompt_buckets | **YES** — we can detect and warn |
| AI hallucinating file paths/commands | session_narratives | **YES** — we can detect and flag |
| Broken tool integration (null results) | session_streams | **YES** — we can detect and alert ops |
| `code_review` single-file prompts work best (2.53) | prompt_buckets | **YES** — we can recommend this shape |
| Scope creep from broad prompts | session_narratives | **YES** — we can detect and split |
| 54-68% sessions abandoned | resolution patterns | Metric only — symptom, not lever |
| Developer X has Y% of root cause Z | patterns.json | Metric only — retrospective |

### What's NOT actionable:
- Per-developer comparisons (VP said "this doesn't tell me about the developer")
- Historical dashboards (nobody looks at them after day 1)
- Root cause pie charts (interesting once, then forgotten)

### What IS actionable — the 3 intervention points:

**1. BEFORE the session: Prompt shaping**
- We know which prompt patterns work (error_paste_and_fix: 0.25 score) and which fail (multi_task: 8.67)
- A pre-flight check that grades the prompt BEFORE the AI runs would prevent waste
- Quantifiable: reduction in trouble_score for prompted sessions vs unprompted

**2. DURING the session: Live detection**
- We can detect "null result spirals" (Zzjjaayy's #1 issue) in real-time
- We can detect scope creep (AI exploring >10 files without editing any)
- We can detect error cascades (3+ consecutive shell errors)
- Quantifiable: sessions interrupted early vs sessions that spiral to score 50+

**3. AFTER the session: Feedback loop**
- Session-level scoring already works (trouble_score)
- Weekly digest: "your 3 highest-scoring sessions this week, here's what went wrong and what to do differently"
- Quantifiable: week-over-week change in avg trouble score

---

## The Direction: Real-Time Session Intelligence

**Not another dashboard. An agent/middleware that sits in the session flow.**

The data we have proves specific patterns cause failures. Instead of reporting on failures after they happen, we should build a system that:

1. **Scores prompts before they run** — using the prompt_bucket classifier + trouble_score correlations we already built
2. **Monitors sessions in near-real-time** — the daemon already pushes data to our DB; we can tail it
3. **Generates per-session recommendations** — reusing the LLM narrative pipeline but focused on "next time, do X"

### Why this is quantifiable:
- **Baseline**: We have trouble_score distributions per developer per week (c02_team_trends already computes weekly aggregations)
- **Intervention**: Deploy the enabler
- **Measurement**: Compare trouble_score distributions before/after
- **Additional metrics**: Abandonment rate, sessions with score >10, avg score per prompt bucket

---

## Research Directive for Next Agent

### Objective
Design a **session intelligence system** that uses qc-trace's existing data pipeline to provide real-time and post-session interventions. This is NOT a dashboard — it's an enabler that makes developers' interactions with AI tools more effective, with quantifiable before/after metrics.

### What to Research

#### 1. Real-Time Intervention Feasibility
- **How does data flow today?** The daemon (`qc_trace/daemon/`) collects session data and pushes to the DB. Can we tap into this flow to detect patterns as they happen?
- Read: `qc_trace/daemon/collector.py`, `qc_trace/server/handlers.py`, `qc_trace/db/writer.py`
- Key question: Is there a webhook/event system, or do we need to poll? What's the latency from session event → DB write?

#### 2. Prompt Pre-Flight Scoring
- We have `_classify_prompt()` in `analysis-feb2026/qualitative/qualitative_analysis.py` (cell 4) that buckets prompts
- We have trouble_score correlations per bucket
- Research: Can we build a lightweight scorer that takes a prompt string and returns a risk score + recommendation?
- The scorer should use the bucket classifier + the avg trouble scores we computed:
  ```
  error_paste_and_fix: 0.25 (best)
  code_review: 2.24-2.53
  detailed_spec: 1.07-2.48
  general: 3.0-6.72
  exploration: 1.0-7.14
  multi_task: 8.67 (worst)
  vague_directive: 17.0 (worst)
  ```
- Could this be an MCP tool that devs add to their Claude Code config? Or a git hook? Or a CLI wrapper?

#### 3. Session Monitor — Pattern Detection
- Using the data in our DB, can we build detectors for:
  - **Null spiral**: >3 consecutive tool_results with null/empty output → alert: "your tool integration may be broken"
  - **Error cascade**: >3 consecutive shell errors (from `is_shell_error()`) → alert: "AI is stuck, consider intervening"
  - **Scope creep**: >10 explore-type tool calls without any edit-type calls → alert: "AI is exploring without acting"
  - **Hallucination**: tool_call references a file path → tool_result says "No such file" → flag
- Read: `analysis-feb2026/analyzers/helpers.py` for `is_shell_error()`, `categorize_tool()`, `has_error_cascade()`
- Key question: What's the delivery mechanism? Slack bot? CLI notification? MCP tool?

#### 4. Post-Session Digest
- Can we automate the qualitative analysis pipeline to run daily/weekly?
- Generate a per-developer weekly email/Slack message: "This week: X sessions, Y flagged. Top issue: Z. Recommendation: ..."
- This reuses: `qualitative_analysis.py` cells 2 (scoring), 4 (bucketing), 5 (LLM narratives), 6 (patterns)
- Quantifiable: Does avg trouble_score decrease over time after receiving digests?

#### 5. Delivery Mechanism Research
- **MCP Server approach**: Build a custom MCP server that Claude Code/Cursor can call. Tools like `check_prompt(prompt_text)` → returns risk score and suggestion. `session_health()` → returns current session's health based on tool call patterns.
- **Slack bot approach**: Post weekly digests and real-time alerts to a team channel
- **CLI wrapper approach**: Wrap the AI CLI invocation with a pre/post hook
- **CLAUDE.md injection**: Auto-generate per-repo CLAUDE.md files with prompting best practices based on that repo's failure patterns
- Which approach has the lowest friction for developer adoption?

#### 6. Quantification Framework
- Define the metrics that prove this works:
  - Primary: avg trouble_score per developer per week (already computed in c02_team_trends)
  - Secondary: abandonment_rate, sessions_with_score_gt_10, prompt_bucket_distribution_shift
  - Baseline period: Jan-Feb 2026 data (what we have)
  - Measurement period: after enabler deployment
- Can we A/B test? (enable for some devs, not others)

### Data Available for Research
- **DB schema**: sessions (id, user_email, org, repo_name, cwd, git_branch, source, first_seen), messages, tool_calls, tool_results, token_usage
- **Existing analyzers**: a01-a10 (heuristic), b01-b04 (LLM), c01-c02 (aggregate)
- **Qualitative pipeline**: prompt classifier, trouble scorer, LLM narratives, pattern detector
- **Daemon code**: `qc_trace/daemon/` — the collection pipeline that feeds the DB
- **MCP infrastructure**: The project already has MCP server concepts (`qc_trace/server/`)

### Constraints
- Must work with existing qc-trace data pipeline (no new data collection needed)
- Must be quantifiable (before/after metrics)
- Must be an enabler (makes dev life easier), not a surveillance tool (tracks dev performance)
- Should reuse existing code: `_classify_prompt()`, `is_shell_error()`, `categorize_tool()`, trouble_score formula
- Framing: "your AI sessions are more effective" not "you're prompting badly"

### Expected Output
A detailed research document covering:
1. Architecture diagram: data flow from session → detection → intervention → measurement
2. Feasibility assessment for each intervention point (pre/during/post)
3. Recommended delivery mechanism with justification
4. MVP scope: what's the smallest thing we can build that demonstrates quantifiable value?
5. Implementation plan for the MVP
