# Developer Behavior Analytics — Analysis Plan

## Context
We have trace data from AI coding assistants (Claude Code, Codex CLI, Cursor, Gemini CLI) across pratilipi-ai and pratilipi-platform orgs. Every session, message, tool call, tool result, and token usage is captured. The goal: extract actionable patterns about how developers use coding assistants, so we can help them be more effective.

---

## What value are we trying to deliver?

The ultimate thesis: **if we understand how a developer works, we can pre-load context and reduce the exploration overhead that LLMs spend on every new session.** Today, every new chat starts cold — the LLM has to rediscover the codebase. If we know:
- Which files this dev always touches together
- What kind of problems they typically solve
- Their prompting style and specificity level
- Which repos/areas they work in

...we can front-load that context and make every session faster and more accurate.

---

## Analysis Dimensions

### 1. File Edit Hotspots & Signals

**What we can extract:** From `tool_calls` where tool_name is Edit/Write/MultiEdit, the `tool_input` JSONB contains `file_path`. Cross-reference with session's `cwd` and `repo_name` to get relative paths.

**Signals to look for:**
- **Config churn** — files like `.env`, `config.ts`, `pyproject.toml` edited across many sessions = fragile setup
- **Core modules** — files edited in >30% of sessions for a repo = architectural hotspot
- **Test files** — ratio of test file edits to source file edits = testing discipline
- **Edit count per file per session** — high count = the LLM is struggling with that file (repeated fixes)

**Key question to answer:** Which files should be pre-loaded as context for a given repo+developer combination?

**Important prerequisite:** Before analyzing file edits, check which files the assistant *explored* (Read, Glob, Grep tool calls) vs actually *edited*. The exploration-to-edit ratio tells us how much discovery overhead exists.

---

### 2. File Co-occurrence / Dependency Graph

**What we can extract:** For each session, collect the set of files touched (read or edited). Pairs of files that appear together across multiple sessions are likely coupled.

**Signals to look for:**
- **Strong pairs** — files always edited together (e.g., `api.ts` + `types.ts`) = tight coupling
- **Clusters** — groups of 3-5 files that co-occur = a "feature area"
- **Cross-repo patterns** — do certain file types always co-occur? (e.g., migration + model + test)

**Value proposition:** If a developer opens a session and mentions file A, we can proactively suggest/preload files B, C, D that historically co-occur. This eliminates 2-3 rounds of codebase exploration.

---

### 3. Developer Work Style Classification

**What we can observe per session:**
- Message count (user turns)
- Session duration (first_seen to last_updated)
- Tool calls per session
- Ratio of user messages to assistant messages
- Number of unique files touched
- Whether session ends with a commit (Bash tool with "git commit")

**Developer archetypes to look for:**
- **One-shotters** — single long prompt, expects complete solution. Short sessions, 1-3 user messages, high tool count.
- **Iterators** — back-and-forth refinement. Many user messages, long sessions, lots of corrections.
- **Parallelizers** — many short sessions running concurrently (check overlapping timestamps across sessions for same user+device).
- **Explorers** — heavy Read/Grep/Glob usage, fewer edits. Using the assistant to understand code, not write it.

**Why this matters:** Different archetypes need different context strategies. One-shotters need maximum upfront context. Iterators need the assistant to remember corrections. Parallelizers need session isolation.

---

### 4. User Message Intent Analysis

**What we can extract:** All messages where `msg_type = 'user'`. Strip `<INSTRUCTION>` tags from codex content.

**Dimensions to analyze:**
- **Intent categories** (keyword/pattern based): fix/debug, create/add, refactor, explain, test, deploy, review
- **Specificity level**: Does the prompt mention file names? Function names? Line numbers? Error messages? Or is it vague ("fix the bug", "make it work")?
- **Message length distribution**: Short commands vs detailed specifications
- **Code inclusion**: Does the user paste code snippets, error traces, stack traces?
- **Multi-step vs atomic**: Does the user give a list of tasks or one task at a time?

**Key insight we're after:** What's the correlation between prompt specificity and session success? If vague prompts lead to longer sessions with more tool calls, that's a clear signal that coaching developers to be more specific pays off.

---

### 5. Session Resolution / Success Estimation

**This is inherently fuzzy, but we can use proxy signals:**

**Positive signals (likely successful):**
- Session contains a `git commit` in Bash tool calls
- Session ends with an assistant message (not an error)
- Low tool failure rate in the session
- Reasonable duration (not abandoned after 1 message, not stuck for hours)

**Negative signals (likely failed/abandoned):**
- Very short session with only 1-2 messages and no tool calls
- High tool failure rate
- Last messages contain error patterns
- Session has long gaps between messages (user walked away)
- Multiple attempts at the same file edit (repeated failures)

**Caveat the user raised:** Session dropoffs happen for many reasons (meeting, context switch, solved it mentally). So this metric is directional, not precise. Best used in aggregate ("user X has 70% estimated success rate") not per-session.

---

### 6. Quantitative Distributions

**Pure numbers to compute across all sessions:**

For **sessions:**
- Messages per session (by msg_type: user, assistant, tool_call, tool_result)
- Duration distribution
- Sessions per user per day/week
- Sessions per repo

For **messages:**
- Length distribution (chars/words) for user vs assistant messages
- Tool calls per session
- Distinct tools used per session
- Token usage: input, output, cached, thinking — by session, by user, by model

For **tools:**
- Most used tools overall and per user
- Tool failure rates by tool name
- Tool call sequences (what tools follow what tools)

**Strip `<INSTRUCTION>` tags** from codex messages before computing length stats.

---

### 7. Repetitive Developer Patterns

**What to look for:**
- **Opening moves** — what does each developer's first message typically look like? Some always start with context, others jump straight to "fix X"
- **Tool preferences** — does a dev favor Edit over Write? Do they use Grep or rely on the assistant to find things?
- **File affinity** — does a dev always work in the same part of the codebase?
- **Time patterns** — when do they use the assistant? Morning deep work vs evening debugging?
- **Repo patterns** — do they use the assistant differently for different repos?

**Most valuable output:** A per-developer profile that summarizes their typical workflow. This profile could seed future sessions with relevant context.

---

### 8. Prompt Complexity & Vagueness

**Heuristic scoring:**

**Complexity indicators** (more = more complex):
- Word count / sentence count
- Bullet points or numbered lists
- Multiple file references
- Code blocks included
- Explicit constraints ("must", "should not", "without breaking")

**Specificity indicators** (more = more specific):
- File paths mentioned
- Function/class names mentioned
- Error messages or stack traces included
- Line number references
- Concrete acceptance criteria

**Vagueness indicators:**
- "fix this", "make it work", "do this" without detail
- No file references
- No code context
- Very short (< 10 words)

**Correlation to explore:** vague prompts → more exploration tool calls → longer sessions → lower success rate? If this holds, it's the strongest actionable insight for users.

---

## Priority Order for Maximum Value

1. **Quantitative distributions (6)** — establishes baselines, fast to compute, reveals data quality issues early
2. **User message analysis (4)** + **Prompt complexity (8)** — these are tightly coupled and give the richest developer-facing insights
3. **File edit hotspots (1)** + **Co-occurrence (2)** — the dependency graph is the most technically valuable output for pre-loading context
4. **Developer work styles (3)** + **Repetitive patterns (7)** — user profiling, useful but needs the above as input
5. **Session resolution (5)** — hardest to get right, most useful as a validation metric for other analyses

---

---

## Data Reality Check (queried from DB)

| Fact | Value |
|------|-------|
| **Users** | 2 developers (ajaysingh@pratilipi-ai, zzjjaayy@pratilipi-platform) |
| **Sessions** | 189 total (148 Claude Code, 31 Codex CLI, 10 Gemini CLI) |
| **Messages** | ~32K total (pratilipi-platform has 4x more data) |
| **Date range** | Feb 10-11, 2026 (~2 days) |
| **Session size** | p25=21, p50=85, p75=238, p95=688, max=4302 messages |

**Top tools:** shell_command (2552), Read (2015), Edit (1082), Grep (768), Bash (608), Glob (481), exec_command (327), TodoWrite (246), Write (152), Task (46)

**Implications for analysis:**
- With only 2 users, "developer classification" becomes individual profiling — comparing 2 distinct workflows
- zzjjaayy (pratilipi-platform) is a much heavier user — 111 sessions, 32K+ messages, uses progress tracking, file_snapshot, summary features
- ajaysingh (pratilipi-ai) uses 3 different CLIs (Claude Code, Gemini, Codex) — cross-tool comparison possible
- `shell_command` (codex) and `Bash` (claude) are the same concept — need to normalize
- `exec_command` is likely Gemini's equivalent — same normalization needed
- Heavy Read+Edit pattern suggests significant codebase exploration before edits — quantifiable
- MCP tools present (figma) — interesting signal about workflow integration
- TodoWrite (246 calls) shows one user relies heavily on task planning

---

## Revised Analysis Plan (given 2 users, 2 days of data)

Given the small user count but rich per-user data, the analysis shifts from "population statistics" to **deep individual developer profiling and comparative analysis**.

### Notebook 1: Quantitative Baselines
- Session count, duration, message counts by user and CLI
- Message type distributions (user vs assistant vs tool_call vs tool_result)
- Token usage distributions (input/output/cached/thinking) by user
- Tool frequency per user — normalize shell_command/Bash/exec_command
- Sessions per CLI tool — ajaysingh uses 3 CLIs, compare behavior across them

### Notebook 2: User Message Deep Dive
- Extract all `msg_type='user'` messages, strip `<INSTRUCTION>` tags
- Message length distribution per user
- Intent classification (fix/create/refactor/explain/test) via keyword patterns
- Specificity scoring: file paths, function names, error traces mentioned?
- Vague prompt detection ("fix this", "do this", < 10 words)
- First message per session analysis — what's the opening prompt pattern?
- **Compare the 2 users**: who is more specific? Who gives more context?

### Notebook 3: File Edit Patterns
- Extract file_path from tool_input JSONB for Edit/Write/MultiEdit tool calls
- Normalize against session cwd to get relative paths
- Top edited files per user and per repo
- Edit count per file per session (repeated edits = struggle signal)
- **Exploration-to-edit ratio**: how many Read/Grep/Glob calls happen before the first Edit per session?
- File type breakdown (what extensions are being edited)

### Notebook 4: File Co-occurrence & Dependencies
- Per session, collect all files touched (read + edited)
- Build co-occurrence matrix — file pairs that appear in same session
- Cluster files into "feature areas"
- Per-user file affinity — does each dev have a home area?
- **Deliverable**: for each repo, a dependency map of "if you touch file A, you'll likely need files B, C"

### Notebook 5: Developer Workflow Comparison
- Compare the 2 users head-to-head:
  - Avg session length, messages per session, tool calls per session
  - Tool preference (Read-heavy vs Edit-heavy)
  - Planning behavior: TodoWrite usage, Task/TaskCreate usage, EnterPlanMode
  - Session structure: does the user front-load context or iterate?
  - Exploration patterns: Grep/Glob/Read before first edit
- **ajaysingh cross-CLI comparison**: same user on Claude Code vs Codex vs Gemini — how does behavior differ?

### Notebook 6: Session Resolution Estimation
- Proxy signals: git commit in Bash/shell_command, session ends with assistant (not error), tool failure rate
- Classify sessions as: completed / abandoned / errored
- Per-user success rate
- Correlation: prompt specificity (from notebook 2) vs session outcome
- Session duration vs outcome

### Notebook 7: Prompt Complexity Scoring
- Score each user message for complexity (length, structure, code blocks, constraints) and specificity (file refs, function names, error traces)
- 2D scatter: complexity vs specificity per user
- Correlate with: session length, tool call count, estimated success
- **Key question**: do more complex/specific prompts lead to shorter, more successful sessions?

### Notebook 8: Repetitive Patterns & Actionable Insights
- First-message clustering — do sessions start with similar prompts?
- Recurring file sets — are the same files always touched together?
- Tool sequences — common patterns (e.g., Grep → Read → Edit → Bash)
- **Synthesis**: for each user, build a "developer profile" summarizing their style, preferences, and common workflows
- **Recommendations**: what context should be pre-loaded for each user+repo combination?

---

## Execution Order
1. Notebook 1 (baselines) — establishes all the numbers
2. Notebook 2 (user messages) — understand intent and prompting style
3. Notebook 3 (file edits) — map the hot spots
4. Notebook 4 (co-occurrence) — build dependency graphs
5. Notebook 5 (workflow comparison) — compare the 2 devs
6. Notebook 7 (prompt complexity) — scoring framework
7. Notebook 6 (session resolution) — success estimation
8. Notebook 8 (synthesis) — actionable profiles and recommendations

## Verification
- Each notebook runs independently with `conn = init()` using `.prod.env`
- Spot-check results by opening specific sessions in the Streamlit session viewer (`notebooks/apps/session_viewer.py`)
- Cross-validate file edit counts against manually inspecting 2-3 sessions
- Validate intent classification by reading sample user messages

## Key Files to Reuse
- `notebooks/utils/connection.py` — init(), query_df(), scalar()
- `notebooks/utils/sessions.py` — by_org(), by_user()
- `notebooks/utils/messages.py` — by_session(), by_org(), search()
- `notebooks/utils/tokens.py` — summary_by_session(), summary_by_user()
- `notebooks/utils/tools.py` — calls_by_org(), call_frequency(), failure_rate(), distinct_tool_names()
- `qc_trace/db/schema.sql` — reference for JSONB field structure in tool_input
