# Developer Behavior Analytics — Context Checkpoint

## What this project is

We have trace data from AI coding assistants (Claude Code, Codex CLI, Cursor, Gemini CLI) captured in a PostgreSQL database. Every session, message, tool call, tool result, and token usage is recorded. We're building analytics to understand how developers use coding assistants so we can help them be more effective.

**Ultimate thesis**: If we understand how a developer works, we can pre-load context and reduce the exploration overhead that LLMs spend on every new session.

---

## Database schema (quick reference)

- **sessions**: id, source (CLI name), model, user_email, org, repo_name, cwd, git_branch, first_seen, last_updated
- **messages**: id, session_id, msg_type (user/assistant/tool_call/tool_result), content, thinking, timestamp, model
- **token_usage**: message_id, input_tokens, output_tokens, cached_tokens, thinking_tokens
- **tool_calls**: message_id, tool_id, tool_name, tool_input (JSONB — stored as binary JSON in Postgres, auto-deserialized to Python `dict` by psycopg)
- **tool_results**: message_id, call_id, output (text), status (success/failure)

Schema file: `qc_trace/db/schema.sql`

## Data shape

| Fact | Value |
|------|-------|
| Users | 2 developers (ajaysingh@pratilipi-ai, zzjjaayy@pratilipi-platform) |
| Sessions | ~189 (148 Claude Code, 31 Codex CLI, 10 Gemini CLI) |
| Messages | ~32K total |
| Date range | Feb 10-11, 2026 (~2 days, will grow) |
| Top tools | shell_command (2552), Read (2015), Edit (1082), Grep (768), Bash (608), Glob (481) |

---

## Coding style & practices (MUST follow)

### Charts & visualization
- **Use seaborn only** for all charts. No raw matplotlib histograms or bar charts.
- Apply `sns.set_theme(style="whitegrid", palette="muted")` once at the top for a consistent look across all charts.
- Every histogram should show a **median line** (`ax.axvline`).

### Metrics must be PM-readable
- Never show raw ratios like "explore-to-edit ratio: 2.8x" without explanation.
- Always add plain English interpretation. Example:
  - Bad: `Explore-to-edit ratio: 2.8x`
  - Good: `For every file edited, the AI reads 2.8 files first — this is high, suggesting the AI spends a lot of time exploring before acting`
- Every metric should answer "so what?" for a non-technical reader.

### HTML reports
- Reports are self-contained HTML files with base64-embedded PNG charts and inline JS for interactive elements.
- Tables must have: dark header row, alternating row colors, hover effect, rounded corners. Long file paths must wrap (`overflow-wrap: break-word; max-width: 500px` on `td`). Wrap `<table>` in a `.table-wrap` div with `overflow-x: auto`.
- Interactive elements (graphs) go in `report["raw_html"]` blocks — raw HTML/JS embedded directly.
- Use `html.escape()` on any user-generated content (prompt samples etc.) before embedding in HTML.

### NLP on user messages
- **Strip ALL system prompt injection** before any text analysis. This includes:
  1. Codex `AGENTS.md` header: `# AGENTS.md instructions for [path]`
  2. `<INSTRUCTION>` or `<INSTRUCTIONS>` tags AND their content (the content is a system prompt, not user text)
  3. Non-ASCII characters
- Use this function:
  ```python
  def strip_system_prompts(text):
      if not text: return ""
      text = re.sub(r"^#\s*AGENTS\.md\s+instructions\s+for\s+.+?\n\n<INSTRUCTIONS?>[\s\S]*?</INSTRUCTIONS?>\s*",
                     "", text, flags=re.IGNORECASE)
      text = re.sub(r"<INSTRUCTIONS?>[\s\S]*?</INSTRUCTIONS?>", "", text, flags=re.IGNORECASE)
      text = text.encode("ascii", errors="ignore").decode("ascii")
      return text.strip()
  ```
- **Some Codex messages are entirely system prompt** (76 messages, ~0.4% of codex messages). After stripping, these become empty — filter them out (`user_text = user_text[user_text["clean"].str.len() > 0]`).
- Stopwords list must be extensive — include English stopwords AND code/agent jargon: `file`, `code`, `function`, `return`, `path`, `value`, `string`, `import`, `module`, `class`, `method`, `type`, `name`, `data`, `list`, `dict`, etc. These pollute word clouds.
- Generate word clouds from frequency dicts (`wc.generate_from_frequencies(freq_dict)`) not raw text, so stopword filtering is applied.

### Notebook format
- Jupytext percent-script format (`# %%` cell markers, `.py` files)
- Every notebook starts with:
  ```python
  import sys, os
  sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
  from utils.connection import init, query_df
  conn = init()
  ```
- Lint-check with `pyflakes` before shipping. No unused imports, no undefined names.

### General
- Don't install packages with pip; use `uv pip install --python .venv-notebooks/bin/python <package>`
- Filters (org, repo prefix, user) go at the top of the notebook as plain variables, not interactive widgets.

---

## Utility code (reuse these)

All in `notebooks/utils/`:

```python
from utils.connection import init, query_df, scalar
conn = init()  # loads .prod.env, returns psycopg connection

# Modules with pre-built queries:
from utils import sessions   # list_all(), by_user(), by_org(), by_repo(), search()
from utils import messages    # by_session(), by_user(), by_org(), count_by_type()
from utils import tokens      # summary_by_session(), summary_by_user(), summary_by_model()
from utils import tools       # call_frequency(), failure_rate(), calls_by_session()
```

## Tool name normalization (critical)

These are the same concept across CLIs — must merge in any analysis:
- **Shell**: `Bash` (Claude Code) = `shell_command` (Codex) = `exec_command` (Gemini)
- **Edit**: `Edit`, `Write`, `MultiEdit`, `MultiEditTool`
- **Explore**: `Read`, `Grep`, `Glob`, `View`, `ReadFile`, `read_file`, `grep_search`, `codebase_search`, `list_dir`, `file_search`

## tool_input JSONB keys per tool

The `tool_input` column is JSONB in Postgres. psycopg auto-deserializes it to a Python `dict`. Here are the keys per tool:

| Tool | Keys |
|------|------|
| **Read** | `file_path` |
| **Edit** | `file_path`, `old_string`, `new_string` |
| **Write** | `file_path`, `content` |
| **Bash** | `command`, `description` |
| **shell_command** | `command`, `workdir` |
| **exec_command** | `cmd` (note: not `command`) |
| **Grep** | `path`, `pattern`, `type`, `output_mode` |
| **Glob** | `path`, `pattern` |

### Extracting file paths

```python
def extract_path(tool_input):
    if isinstance(tool_input, dict):
        return tool_input.get("file_path") or tool_input.get("path") or tool_input.get("target_file")
    return None

def make_rel(fp, cwd):
    if fp and cwd and fp.startswith(cwd):
        return fp[len(cwd):].lstrip("/") or fp
    return fp
```

### Extracting shell commands

```python
def extract_command(tool_input):
    if isinstance(tool_input, dict):
        return tool_input.get("command") or tool_input.get("cmd") or ""
    return ""
```

### Lines added/removed from Edit calls

`old_string` = what was replaced (lines removed), `new_string` = what replaced it (lines added).
For `Write` calls, the entire `content` field is lines added (new file).

```python
old_lines = old.count("\n") + (1 if old else 0) if old else 0
new_lines = new.count("\n") + (1 if new else 0) if new else 0
```

## Commit detection (heuristic, lower bound)

No first-class commit event. Best signal: keyword match `git commit` or `git push` in shell tool_input. Misses manual commits outside the agent. Directional, not precise. Must use parameterized queries (psycopg interprets `%g` in `%git` as a placeholder):

```python
git_commits = query_df(conn, """
    SELECT DISTINCT m.session_id
    FROM tool_calls tc JOIN messages m ON m.id = tc.message_id
    WHERE tc.tool_name IN ('Bash', 'shell_command', 'exec_command')
    AND (tc.tool_input::text ILIKE %s OR tc.tool_input::text ILIKE %s)
""", ('%git commit%', '%git push%'))
```

---

## Metrics framework

### Session-level metrics

| Metric | How to compute | What it means (PM-readable) |
|--------|---------------|----------------------------|
| Session duration | `last_updated - first_seen` | How long the developer worked with the AI |
| User messages per session | COUNT where msg_type='user' | How much back-and-forth happened |
| Tool calls per session | COUNT where msg_type='tool_call' | How much work the AI did autonomously |
| Prompts before first edit | Count user messages before first Edit/Write tool call | How many rounds of conversation before the AI starts writing code |
| Discovery overhead | COUNT(explore tools) / COUNT(edit tools) | For every file edited, how many files were read first |
| Commit rate | Sessions with git commit / total sessions | What % of sessions produced a tangible commit |
| Tool success rate | success / total tool_results | How often the AI's tool calls work without errors |
| Files touched | COUNT(DISTINCT file_path) | How many files were involved in the session |

### Code change metrics

| Metric | How to compute |
|--------|---------------|
| Lines added per session | Sum of line counts in `new_string` (Edit) and `content` (Write) |
| Lines removed per session | Sum of line counts in `old_string` (Edit) |
| Net line delta | Added - removed |
| Repeated edits to same file | COUNT edits per file per session; >=5 = AI struggling with that file |

### File analysis

| Metric | How to compute |
|--------|---------------|
| Most-read files | Extract `file_path` from Read/ReadFile/View tool_input, count per file |
| Most-edited files | Extract `file_path` from Edit/Write tool_input, count per file |
| Most common shell commands | Parse `command`/`cmd` from shell tools, classify into categories (git status, npm install, etc.) |
| Files @-mentioned in prompts | Regex `@([\w./-]+\.\w{1,6})` in user messages |
| File co-occurrence | Per session, collect all files touched, find pairs that appear together across sessions |

### Prompt NLP metrics

| Metric | How to compute |
|--------|---------------|
| Word frequency / word cloud | Tokenize clean user messages, remove stopwords + non-ASCII |
| Bigram frequency | Common two-word phrases |
| Intent classification | Keyword regex: fixing bugs, building features, refactoring, understanding code, testing, deploying, configuration |

### Session continuity detection

- **Multi-day sessions**: `last_updated - first_seen > 24h`
- **Resumed sessions**: Max inter-message gap > 1 hour
- These matter because resumed sessions preserve context — a developer came back to the same conversation rather than starting fresh

### Developer personas (computable from traces)

| Persona | Signal | Actionable nudge |
|---------|--------|-----------------|
| **Delegator** | Few user msgs (<=3), many tool calls (>10). One big prompt, lets agent work | Front-load as much context as possible in the first message |
| **Collaborator** | Many user msgs (>8). Back-and-forth refinement | AI should remember corrections from earlier in the session |
| **Explorer** | Explore-to-edit ratio > 2.5. Using agent to understand code | Pre-load codebase map or file index to reduce discovery time |
| **Builder** | High edit ratio, commits frequently. Producing code | Pre-load frequently edited files to skip exploration |
| **Firefighter** | >30% of intents are fix/debug. Debug-heavy prompts | Pre-load error context, stack traces |

---

## Industry context (from research)

### How the industry measures coding assistants

- **GitHub Copilot**: Acceptance rate (~30%), persistence rate (code unchanged after 24h), PR throughput (+8.69%), time to first PR
- **DORA 2025**: AI writes 29% of code but org productivity only +3.6%. Individual task completion +21%, PR volume +98%. "Mirror and multiplier" — AI amplifies existing team dynamics
- **Cursor study**: 281% initial velocity boost → collapses to baseline by month 3. 30% more static analysis warnings persist. Speed vs quality tradeoff is real
- **SWE-bench**: Resolve rate (% of GitHub issues fixed). Task-stratified: docs 82% vs features 66%
- **Key insight**: Individual metrics != team metrics. Need multi-dimensional measurement

### Reinforcement signals used by assistants

1. **Acceptance/rejection** of suggestions (primary signal)
2. **Edit-after-suggestion** — user accepts then modifies (partial value)
3. **Persistence rate** — code unchanged after 24-72h (lasting value)
4. **Time to accept/reject** — fast accept = high confidence
5. **Retry rate** — repeated tool calls = system struggling

### What we CAN'T measure without git integration

- True persistence rate (code unchanged after 24h)
- Code churn (lines deleted within 14 days)
- PR cycle time, review rework
- Static analysis warnings delta
- Test coverage delta

---

## What exists

### Notebooks

| Path | Purpose | Status |
|------|---------|--------|
| `notebooks/000-connections/000-view-tables.py` | DB inspection — row counts, columns, sample data | Done |
| `notebooks/002-developer-analytics/001-008` | First-pass analysis (8 notebooks: baselines, user messages, file edits, co-occurrence, workflow comparison, session resolution, prompt complexity, synthesis) | Done but rough |
| `notebooks/003-focused-analytics/001-dev-analytics.py` | Filterable scoreboard with org/user/repo filters, per-dev metrics, week-on-week trends | Done |
| `notebooks/003-focused-analytics/002-dev-report.py` | **Main report generator** — produces self-contained HTML per dev | Done, actively iterated |
| `notebooks/apps/session_viewer.py` | Streamlit app for browsing individual sessions | Done |

### The main report (`002-dev-report.py`)

This is the primary deliverable. It generates a self-contained HTML report per developer with:

**Metric sections** (PM-readable key-value blocks):
- Overview (sessions, CLIs, repos, duration, commit rate)
- How the AI spends its time (explore/edit/shell breakdown, discovery overhead, tool reliability)
- Code changes (lines added/removed/net, per session medians)
- Warmup time (prompts before first edit)
- Session continuity (multi-day sessions, resumed sessions)

**Charts** (seaborn, base64-embedded PNGs):
- Session duration histogram
- User prompts per session histogram
- Tool calls per session histogram
- Prompts before first code edit histogram
- Lines added/removed per session histograms
- Weekly line churn (added vs removed bar chart)
- Most used AI tools (horizontal bar)
- Longest pause within sessions histogram
- Sessions per week, commit rate by week, tool usage by week
- Most-read files (horizontal bar)
- Most common shell commands (horizontal bar)
- Files @-mentioned in prompts (horizontal bar)
- Word frequency, word cloud, bigram frequency
- Intent distribution (fixing bugs, building features, etc.)
- Prompt length histogram, first-message length histogram

**Interactive elements** (inline JS):
- File dependency graph — force-directed canvas graph showing file co-occurrence. Nodes show filename only, hover reveals full path. Edges = sessions together (thicker = more). Draggable.

**Tables** (styled HTML):
- Week-on-week trends
- Most frequently edited files
- Struggle signals (files edited 5+ times in one session)
- Most frequently read files
- Most common shell commands
- Files @-mentioned in prompts

**Developer persona** classification with actionable tips.

**Sample opening prompts** (first message per session, longest first).

### Filters

Set at top of `002-dev-report.py`:
```python
ORG_FILTER  = None            # e.g. "pratilipi-platform"
REPO_PREFIX = "Pratilipi/"    # only repos with this prefix
USER_FILTER = None            # None = all users, or specific email
OUTPUT_DIR  = "reports"
```

---

## Repo layout

```
trace/
├── .prod.env                          # DB connection string (QC_TRACE_DSN)
├── qc_trace/db/schema.sql             # Full DB schema
├── notebooks/
│   ├── utils/
│   │   ├── __init__.py
│   │   ├── connection.py              # init(), query_df(), scalar()
│   │   ├── sessions.py                # list_all(), by_user(), by_org(), search()
│   │   ├── messages.py                # by_session(), by_user(), count_by_type()
│   │   ├── tokens.py                  # summary_by_session(), summary_by_user()
│   │   └── tools.py                   # call_frequency(), failure_rate()
│   ├── 000-connections/
│   │   └── 000-view-tables.py         # DB inspection notebook
│   ├── 002-developer-analytics/       # First-pass analysis (8 notebooks)
│   ├── 003-focused-analytics/
│   │   ├── 001-dev-analytics.py       # Filterable scoreboard + trends
│   │   ├── 002-dev-report.py          # Main HTML report generator
│   │   ├── reports/                   # Generated HTML reports go here
│   │   └── CONTEXT.md                 # THIS FILE
│   └── apps/
│       └── session_viewer.py          # Streamlit session browser
```

---

## Future directions (not yet built)

- Embedding-based prompt clustering (group similar prompts across sessions)
- Cross-developer comparison report (side-by-side)
- Git integration for persistence rate and code churn
- Session success prediction (correlate prompt specificity with outcome)
- Pre-loading recommendation engine (given user+repo, suggest files to front-load)
