# Feature Engineering for AI Coding Session Analysis

> **Data:** 671 sessions (541 Claude Code, 82 Codex, 37 Gemini, 11 Cursor), 151K messages, 4 developers.
>
> **Research scripts:** `analysis-14022026/session_view_scripts/research/`
>
> **Message shape:** `{id, type, timestamp, model, content, tool: {tool_name, tool_input}, result: {status, output}}`
>
> **Session index:** `{id, source, model, repo_name, cwd, start, end, total_messages, user_messages, assistant_messages, tool_calls, tool_results, interruptions}`

---

## Data Quirks To Handle (per source)

| Source | Quirk | How to handle |
|---|---|---|
| **Codex CLI** | First 1-3 `user` messages are system-injected XML: `<INSTRUCTIONS>`, `<environment_context>`, `# AGENTS.md` (25% of codex user msgs) | Skip these when analyzing "user prompts" — they are not human-written |
| **Gemini CLI** | User messages contain `--- Content from referenced files ---` appended with 100K+ chars of file content | Truncate at this signature — only text before it is human-written |
| **Claude Code** | `<command-name>/clear</command-name>` style messages are slash commands | Filter out of prompt analysis |
| **Cursor** | Very terse prompts (median 11 chars), often just `hi` — limited tool call metadata | Prompt depth analysis may not apply |
| **Compaction** | Messages starting with `This session is being continued from a previous conversation` | System-generated summary, not a user prompt |
| **tool_result.status** | 36,893 success vs only 14 failure — field is near-useless for error detection | Use `tool_result.output` content (check for error strings, exit codes) instead |

---

## A. SESSION-LEVEL FEATURES (Heuristic)

These can be computed purely from the session JSON files with no LLM calls.

---

### A1. Interruption Analysis

**Already have:** `interruption_count` per session. 507 events across 162 sessions.

**What the data tells us:**
- Interrupted tools: Bash (272), ExitPlanMode (79), Edit (50), Write (15)
- Position: early(<33%) = 134, mid(33-66%) = 143, late(>66%) = 230
- Post-interrupt: 87% user gave corrective guidance, 13% session ended

| Feature | Type | How to compute | Who cares |
|---|---|---|---|
| `interrupt_count` | int | Already computed | Both |
| `first_interrupt_position_pct` | float | `msg_index / total_msgs * 100` for the first interruption event | **Dev** — how quickly did trust break? |
| `interrupt_position_bucket` | str | Classify each interrupt as `early` (<33%), `mid`, `late` (>66%). Session-level: `mostly_early`, `mostly_late`, `spread` | **VP** — early = AI misunderstood the task from the start; late = AI was on track but went off on one specific action |
| `interrupted_tool_name` | str | Walk backwards from interrupt event to nearest `tool_call` | **Dev** — Bash interrupt = wrong/slow command; Edit interrupt = AI changing wrong code; ExitPlanMode = AI stuck in planning |
| `interrupted_tool_category` | str | Map: Bash/shell_command/exec_command → `shell`, Edit/Write → `edit`, Read/Grep/Glob → `explore`, TodoWrite/ExitPlanMode → `plan` | **VP** — aggregate view |
| `post_interrupt_has_guidance` | bool | Next user message after interrupt exists and is >20 chars | Both |
| `session_recovered_after_interrupt` | bool | >2 user messages exist after the last interruption | **VP** — did the session survive? |
| `interrupt_to_next_msg_seconds` | float | Timestamp diff between interrupt and next user message | **Dev** — how long did user think before correcting? |

**Detection (already working in `backfill_interruptions.py`):**
- Claude Code: `tool_result.output` contains `"The user doesn't want to proceed with this tool use"`
- Codex CLI: `user` message content starts with `<turn_aborted>`


**Critical: The corrective guidance is NOT always in a separate user message.** Claude Code has three distinct interrupt output patterns (see `research/06_interrupt_output_taxonomy.py`):

| Pattern | Count | What it looks like |
|---|---|---|
| **INLINE GUIDANCE** (34%) | 176 | `"...was rejected... To tell you how to proceed, the user said:\nwhy did you remove wrapper of runInTransaction in deleteSeries?"` — the user's correction is embedded directly inside `tool_result.output`, there is no separate user message |
| **REJECTED ONLY** (59%) | 302 | `"...was rejected (eg. if it was a file edit, the new_string was NOT written to the file)"` — no inline text. User gives guidance in the next `user` message (76%) or session ends (17%) or AI continues on its own (7%) |
| **STOP ONLY** (7%) | 37 | `"The user doesn't want to take this action right now. STOP what you are doing and wait for the user to tell you how to proceed."` — hard stop, usually paired with a REJECTED result on the same turn |

**Extraction logic for post-interrupt corrective action:**
1. **First** parse `tool_result.output` for the marker `"the user said:"` — everything after it is the user's inline correction (176 cases)
2. **If no inline guidance** find the next `msg_type='user'` message, skipping over consecutive interrupt `tool_result` messages (there are often 2-3 back-to-back: one "rejected" + one or two "stop") — that next user message is the correction (232 cases)
3. **If neither exists** the user abandoned the session after interrupting (51 cases)

Combined: **~79% of interrupts have extractable corrective guidance.**

| Feature | Type | How to compute | Who cares |
|---|---|---|---|
| `interrupt_guidance_source` | str | `inline` (from tool_result.output after "the user said:") / `next_msg` (separate user message) / `abandoned` (no guidance found) / `stop` (hard stop) | **Dev** — shows how the user communicated the correction |
| `interrupt_guidance_text` | str | The extracted corrective text from either source | Both — **this is the richest signal** for understanding WHY the user interrupted. Real examples: `"no we don't have to change anything here, this is the base code"`, `"change of plans, only fire for api success and not 400"`, `"I dont understand the need for change at all, addTracks is a suspend function already"` |

---

### A2. AI Autonomy

**What the data tells us:** Tool streaks (consecutive tool_calls between user msgs): median=4, mean=10.2, max=731, p90=26.

| Feature | Type | How to compute | Who cares |
|---|---|---|---|
| `tool_calls_per_user_msg` | float | `count(tool_call) / count(user)` in session | **VP** — high = AI works independently per prompt |
| `max_tool_streak` | int | Longest consecutive `tool_call` messages without a `user` message | **Dev** — shows the longest AI-autonomous work segment |
| `avg_tool_streak` | float | Mean streak length | Both |
| `autonomous_segments` | int | Number of tool streaks ≥ 5 | **VP** — meaningful chunks of independent work |
| `user_intervention_rate` | float | `user_messages / (tool_calls + assistant_messages)` — lower = more autonomous | **VP** |

---

### A3. Prompt Signals (Boolean)

Scan `content` of `msg_type='user'` messages. **Must strip source-specific noise first** (codex XML, gemini file refs, compaction text, slash commands).

**What the data tells us:** URLs in 273 user msgs, JSON-like content in 254, traceback in 60+, error strings in 150+.

| Feature | Type | Detection | Who cares |
|---|---|---|---|
| `has_error_paste` | bool | Content contains `Traceback (most recent call last)` or `Error:` at line boundary or `FAILED` or `ERR!` or `exit status 1` | **VP** — developer is debugging, not building |
| `has_log_paste` | bool | Content has ≥3 lines matching timestamp patterns like `2025/10/14 00:26:38` or ISO timestamps | **Dev** — pasting logs for diagnosis |
| `has_url` | bool | `https?://\S+` present in user-written portion | Both |
| `has_structured_data` | bool | Content has ≥3 `{` and ≥3 `}` and ≥3 `:` (pasted JSON/config/struct) | Both |
| `has_code_paste` | bool | Content ≥200 chars with high density of code-like tokens (`.`, `(`, `)`, `{`, `;`, `func `, `def `, `class `) and low natural language | **Dev** — pasting code for AI to work with |
| `has_terminal_output` | bool | Content contains `$` or `%` at line starts (terminal prompt) or multi-line output with consistent indentation | **Dev** — pasting terminal session |
| `prompt_is_terse` | bool | Cleaned user message is ≤15 chars (like `fix this`, `continue`, `hi`) | **VP** — low-effort prompting |
| `first_prompt_char_len` | int | Length of the first real user message (after noise stripping) | Both |

---

### A4. Session Outcome Signals

| Feature | Type | How to compute | Who cares |
|---|---|---|---|
| `has_edits` | bool | Any `tool_call` where `tool_name` in (Edit, Write, MultiEdit, MultiEditTool, replace) | **VP** — did the session produce code? |
| `edit_count` | int | Count of such tool calls | Both |
| `files_edited` | int | Count distinct file paths from `tool_input` of edit tools | **VP** — scope of changes |
| `has_git_commit` | bool | Any shell tool call with `git commit` in input | **VP** — work was shipped |
| `has_git_push` | bool | Any shell tool call with `git push` in input | **VP** — work reached remote |
| `has_test_run` | bool | Shell tool input contains `pytest`, `npm test`, `go test`, `jest`, `cargo test` | Both |
| `has_test_pass_after_fail` | bool | A test-related shell result with failure-like output followed later by success-like output | **Dev** — bug was fixed! |
| `session_ended_with_user_msg` | bool | Last message in session is `type='user'` (104 of 671 sessions) | **VP** — may indicate user abandoned or was unsatisfied |
| `git_operations` | dict | Count of `git commit`, `git push`, `git diff`, `git checkout`, `git reset`, `git stash` in shell tool inputs | Both |

**Shell tool names to check:** `Bash`, `shell_command`, `exec_command`, `run_shell_command`, `run_terminal_cmd`

---

### A5. Undo / Revert Detection

**What the data tells us:** ~46 genuine undo requests, 104 git checkout, 26 git reset, 443 AI self-corrections.

| Feature | Type | How to compute | Who cares |
|---|---|---|---|
| `user_requested_undo` | bool | User message (cleaned) contains: `undo`, `revert this/that/the`, `roll back`, `go back to`, `put it back` | **VP** — direct signal AI made a mistake |
| `undo_request_count` | int | Count of such messages | Both |
| `git_revert_in_session` | bool | Shell tool input contains `git checkout --`, `git reset`, `git restore`, `git stash` | Both |
| `ai_self_corrected` | bool | Assistant message contains `I apologize`, `my mistake`, `let me fix that`, `that was incorrect` | **Dev** — AI acknowledged its error |
| `ai_self_correction_count` | int | Count (443 total across all sessions) | **VP** |

**Important:** Must strip gemini file refs and compaction summaries before keyword matching, otherwise get false positives.

---

### A6. AI Clarification Behavior

**What the data tells us:** 660 assistant messages with question-like patterns, but many are sign-off pleasantries ("let me know if you have questions"). Real clarification questions are a subset.

| Feature | Type | How to compute | Who cares |
|---|---|---|---|
| `ai_asked_clarification` | bool | Assistant message contains: `would you like me to`, `do you want me to`, `should I `, `could you clarify`, `which one`, `before I proceed`, `would you prefer` — BUT NOT just sign-offs (`let me know`, `feel free to`) | **Dev** — AI was uncertain |
| `clarification_count` | int | Count of real clarifications | Both |
| `clarification_position` | str | `early` (first 20% of msgs), `mid`, `late` | **VP** — early = ambiguous initial prompt; late = task got complex |

---

### A7. Conversation Dynamics

| Feature | Type | How to compute | Who cares |
|---|---|---|---|
| `total_turns` | int | Count of user→AI exchanges (each `user` message starts a new turn) | Both |
| `avg_ai_msgs_per_turn` | float | `(assistant + tool_call) / user_messages` — how much work AI does per prompt | **VP** |
| `deepest_turn` | int | Maximum number of AI messages + tool calls between consecutive user messages | **Dev** |
| `user_to_ai_char_ratio` | float | `sum(len(user.content)) / sum(len(assistant.content))` — high = user is verbose, low = AI does most talking | Both |
| `session_duration_min` | float | `(end - start)` in minutes | Both |
| `avg_gap_between_user_msgs_sec` | float | Average timestamp diff between consecutive user messages | **Dev** — user's think time |
| `longest_gap_sec` | float | Maximum gap between any two consecutive messages | Both |

---

### A8. Session Linkage

| Feature | Type | How to compute | Who cares |
|---|---|---|---|
| `is_continuation` | bool | First user message is a compaction (starts with `This session is being continued`) | Both |
| `compaction_count` | int | Already in index | Both |
| `time_gap_to_prev_session_min` | float | Time between this session's start and the previous session's end (same user + same repo) | **Dev** |
| `is_sequential` | bool | `time_gap_to_prev_session_min < 120` and same repo/cwd | **VP** — multi-part task |
| `has_parallel_sessions` | bool | Another session by the same user overlaps in time (already detected in viewer with `detectOverlaps`) | **VP** — parallel workstreams |
| `parallel_same_repo` | bool | Overlapping sessions share the same repo | **Dev** |

---

### A9. Tool Usage Profile

| Feature | Type | How to compute | Who cares |
|---|---|---|---|
| `explore_count` | int | tool_calls where name in (Read, Grep, Glob, read_file, grep_search, codebase_search, list_dir, file_search) | Both |
| `edit_count` | int | Edit, Write, MultiEdit, MultiEditTool, replace | Both |
| `shell_count` | int | Bash, shell_command, exec_command, run_shell_command, run_terminal_cmd | Both |
| `plan_count` | int | TodoWrite, TodoRead, TaskCreate, TaskUpdate, ExitPlanMode | Both |
| `delegation_count` | int | Task, TaskOutput (sub-agent usage) | **VP** |
| `web_count` | int | WebSearch, WebFetch | Both |
| `mcp_count` | int | tool names starting with `mcp__` | **VP** |
| `explore_edit_ratio` | float | `explore_count / max(edit_count, 1)` — high = lots of reading before writing | **VP** — exploration vs execution |
| `unique_tools_used` | int | Count distinct tool names | **Dev** |
| `dominant_tool_category` | str | Which category has the most tool calls | **VP** |

---

### A10. Error Detection from Tool Output

Since `tool_result.status` is almost always `success` (36,893 vs 14 failure), we must detect errors from `tool_result.output` content.

| Feature | Type | How to compute | Who cares |
|---|---|---|---|
| `shell_error_count` | int | Shell tool results where output contains `exit code` > 0, or `Error`, `FAILED`, `error:`, `command not found`, `No such file` | Both |
| `consecutive_shell_errors` | int | Max streak of shell tool_results with error-like output | **Dev** — AI retrying and failing |
| `error_cascade_count` | int | Sequences of 3+ consecutive tool_results with errors | **VP** — systemic failure |
| `error_then_user_rescue` | bool | Error cascade interrupted by a user message (human had to step in) | **VP** |

---

## B. SESSION-LEVEL FEATURES (LLM-based)

These require an LLM call per session. **Use sparingly — batch weekly, cache results.**

---

### B1. Task/Problem Classification

**Why LLM:** Rule-based intent detection is brittle. User prompts like `"bro we are using clustered valkey"` don't match keywords for any category but an LLM instantly gets it's a debugging/architecture question.

| Feature | Type | LLM prompt strategy | Who cares |
|---|---|---|---|
| `task_type` | str | Feed first user prompt (cleaned) → classify as: `bug_fix`, `build_feature`, `refactor`, `understand_code`, `debug`, `test`, `config_infra`, `code_review`, `exploration` | **VP** — what types of work is AI being used for? |
| `task_complexity` | str | `simple` / `moderate` / `complex` based on prompt + number of files touched + session length | **VP** |
| `task_domain` | str | `backend`, `frontend`, `mobile`, `infra`, `data`, `mixed` — infer from file paths and prompt content | **VP** |

**LLM call strategy:** One call per session, extract all three fields. Use `gpt-4o-mini` with temperature=0. Cache by hash of first prompt.

**Heuristic pre-filter (skip LLM for obvious cases):**
- If first prompt ≤5 chars (`Warmup`, `hi`): `task_type=trivial`, skip LLM
- If first prompt is a compaction: `task_type=continuation`, skip LLM
- If first prompt is a codex system message only (no real user text): `task_type=system_init`, skip LLM

---

### B2. Session Narrative (already done for top 10)

Extend the approach from `06_add_learning_curves_error_cascades.py`.

| Feature | Type | LLM prompt strategy | Who cares |
|---|---|---|---|
| `session_narrative` | str | 2-sentence summary: what was attempted + was it productive | **VP** |
| `was_productive` | bool | LLM verdict: did this session produce meaningful output relative to its cost? | **VP** |

**LLM call strategy:** Only for sessions with cost > $X threshold or for weekly "top N most expensive" review. Already caching via SHA256 hash.

---

### B3. Prompt Quality Assessment

| Feature | Type | LLM prompt strategy | Who cares |
|---|---|---|---|
| `prompt_quality` | str | `clear`, `vague`, `missing_context` — based on first user prompt | **Dev** — coaching feedback |
| `prompt_improvement_suggestion` | str | One-line suggestion for how the prompt could have been better | **Dev** |

**LLM call strategy:** Batch weekly for sessions where `interrupt_count > 0` or `ai_asked_clarification = true` — these are the sessions where prompt quality was likely an issue.

---

### B4. Failure Root Cause

| Feature | Type | LLM prompt strategy | Who cares |
|---|---|---|---|
| `failure_root_cause` | str | For sessions with `error_cascade_count > 0` or `user_requested_undo`: feed the error context + user messages → classify as `wrong_approach`, `missing_context`, `tool_limitation`, `user_changed_mind`, `ai_hallucination` | **VP** — systemic patterns |

**LLM call strategy:** Only for sessions with failures/undos. ~50-100 sessions, affordable.

---

## C. CROSS-SESSION / GLOBAL FEATURES (Heuristic)

Aggregated across all sessions for a developer or across the whole team.

---

### C1. Developer Behavioral Profile

| Feature | Type | Aggregation | Who cares |
|---|---|---|---|
| `sessions_per_active_day` | float | Total sessions / days with ≥1 session | **VP** |
| `preferred_source` | str | Most frequently used CLI tool | Both |
| `avg_prompts_per_session` | float | Mean `user_messages` across sessions | Both |
| `avg_first_prompt_length` | float | Mean length of first real user message | **VP** — prompting thoroughness |
| `interrupt_rate` | float | `sessions_with_interrupts / total_sessions` | **VP** — how often does the dev lose trust |
| `undo_rate` | float | Sessions with undo requests / total sessions | **VP** |
| `edit_session_pct` | float | Sessions with edits / total sessions | **VP** — percentage that produce code |
| `commit_session_pct` | float | Sessions with git commit / total sessions | **VP** |
| `avg_autonomy_ratio` | float | Mean `tool_calls_per_user_msg` across sessions | **VP** — does this dev let AI work? |
| `exploration_tendency` | float | Mean `explore_edit_ratio` across sessions | **VP** |
| `avg_interrupt_position` | float | Mean of `first_interrupt_position_pct` across interrupted sessions | **VP** — does this dev's AI fail early or late? |

---

### C2. Team-Level Trends (weekly/monthly)

| Feature | Type | Aggregation | Who cares |
|---|---|---|---|
| `sessions_this_week` | int | Count | **VP** |
| `edit_sessions_pct_trend` | float[] | Weekly % of sessions producing edits | **VP** — is the team getting more productive? |
| `interrupt_rate_trend` | float[] | Weekly interrupt rate | **VP** — is AI getting better/worse? |
| `avg_session_cost_trend` | float[] | Weekly average cost per session | **VP** |
| `source_migration_trend` | dict[] | Weekly breakdown of source usage | **VP** — are devs adopting better tools? |
| `top_interrupted_tools_weekly` | str[] | Most interrupted tool each week | **VP** |

---

### C3. Session Linkage Graph

| Feature | Type | How | Who cares |
|---|---|---|---|
| `sequential_chain_length` | int | Walk consecutive sessions by same user+repo within 2h gaps | **VP** — tasks requiring multiple sessions |
| `parallel_session_pairs` | int | Count of time-overlapping session pairs per day | **VP** — multitasking behavior |
| `compaction_chain_depth` | int | How many compactions in a chain (session resumes session resumes session...) | **VP** — context exhaustion |

---

## C4. Cross-Session (LLM-based)

| Feature | Type | LLM prompt strategy | Who cares |
|---|---|---|---|
| `developer_llm_summary` | str | Already done in `06_add_learning_curves_error_cascades.py` | **VP** |
| `weekly_work_narrative` | str | Feed all session narratives for a dev for the week → 3-sentence summary of what they accomplished | **VP** |
| `task_type_distribution` | dict | Aggregate `task_type` across sessions per dev | **VP** — "60% of this dev's AI usage is exploration, not building" |

---

## Priority & Implementation Order

### Phase 1 — Pure heuristic, no LLM, high impact

| Feature group | Sessions affected | Implementation effort |
|---|---|---|
| **A1 Interruption depth** (position, tool, post-action) | 162 sessions with interrupts | Extend `backfill_interruptions.py` |
| **A3 Prompt signals** (error paste, structured data, terse) | All 671 sessions | New script, regex-based |
| **A2 Autonomy metrics** (streaks, tool_calls_per_turn) | 525 sessions with tools | New script |
| **A4 Outcome signals** (edits, commits, tests) | All sessions | New script |
| **A5 Undo detection** | ~46 sessions | Small addition |

### Phase 2 — Heuristic, moderate effort

| Feature group | Notes |
|---|---|
| **A6 AI clarification** | Need to separate real questions from sign-off pleasantries |
| **A8 Session linkage** | Build sequential chains and parallel detection |
| **A9 Tool profile** | Categorize tools, compute ratios |
| **A10 Error detection from output** | Parse `tool_result.output` for errors since status field is useless |
| **C1 Developer profile** | Roll up session features per developer |

### Phase 3 — LLM-based, weekly batch

| Feature group | Estimated calls/week | Model |
|---|---|---|
| **B1 Task classification** | ~100-150 new sessions/week | gpt-4o-mini |
| **B3 Prompt quality** | ~30-50 (only interrupted/clarification sessions) | gpt-4o-mini |
| **B4 Failure root cause** | ~10-20 (only failed sessions) | gpt-4o-mini |
| **C4 Weekly narrative** | ~4 (one per dev) | gpt-4o-mini |

---

## What Matters to Whom

### Developer Dashboard

The developer wants to see:
1. **"My AI worked autonomously for 26 tool calls before I had to step in"** → A2 autonomy
2. **"I interrupted 3 times, all during shell commands"** → A1 interruption detail
3. **"My first prompt was 43 chars — similar sessions with 200+ char prompts had 0 interrupts"** → A3 prompt signals
4. **"AI asked me to clarify 2 things before proceeding"** → A6 clarification
5. **"Session produced 5 edits, 1 commit, tests passed"** → A4 outcomes
6. **"I asked AI to undo changes twice"** → A5 undo

### VP Dashboard

The VP wants to see:
1. **"60% of sessions produce no code changes"** → A4 outcome aggregated
2. **"When AI fails, it fails early — 26% of interrupts happen in the first third of the session"** → A1 position bucket
3. **"Developer X interrupts AI 3x more than Developer Y, mostly during Edit operations"** → C1 interrupt rate + tool
4. **"Bash is the most interrupted tool (54% of all interrupts) — users don't trust AI's shell commands"** → A1 tool analysis
5. **"Sessions where users paste error logs are 2x more likely to result in commits"** → A3 × A4 correlation
6. **"This developer's AI autonomy is increasing week over week"** → C2 trends
7. **"Top failure mode: AI explores 40 files then tries to edit the wrong one"** → B4 failure root cause
8. **"70% of AI usage is for understanding code, only 20% for building features"** → B1 task classification

---

## Research Scripts Reference

All scripts in `analysis-14022026/session_view_scripts/research/`:

| Script | What it explores |
|---|---|
| `00_data_shape.py` | Session count, source distribution, message types, tool names, result status, user msg length distribution |
| `01_sample_user_prompts.py` | Raw vs cleaned first prompts per source, shows codex XML noise, gemini file refs, cursor terseness |
| `02_interruption_patterns.py` | Interruption position, tool interrupted, post-interrupt user messages, context around interrupts |
| `03_autonomy_and_streaks.py` | Tool streak lengths, codex system prefix detection, URL/JSON/traceback/error signals in user messages |
| `04_undo_revert_and_ai_behavior.py` | Undo/revert detection (with false positive handling), AI self-correction, AI clarification, git operations, session endings |
| `05_session_per_source_example.py` | Full message flow for one session per source + one interrupted session |
| `06_interrupt_output_taxonomy.py` | Three interrupt patterns: inline guidance (34%, correction in tool_result.output), rejected only (59%, correction in next user msg), stop only (7%). Shows extraction logic for corrective action from both sources |
