# Phase 1.7: Refactor Artifacts and Protocols and Workflows to Synergy with Grep - Research

**Researched:** 2026-01-27
**Domain:** Markdown artifact design, grep pattern matching, agent collaboration workflows
**Confidence:** HIGH

## Summary

This phase involves a dual transformation: (1) restructuring GSD-Lite artifacts for grep-optimized retrieval using single-line patterns, and (2) revamping the coaching philosophy from "User = visionary, Agent = builder" to "User + Agent = thinking partners." The research reveals this is primarily a **documentation architecture problem** with well-established patterns in ripgrep/grep workflows, combined with a **behavioral framework redesign** requiring careful protocol updates across all workflows.

The technical foundation is solid: ripgrep's single-line matching with regex patterns is the de facto standard for structured markdown search. The fs-mcp `grep_content` tool provides a proven two-step "grep → read" workflow that enables efficient context discovery. The pair programming philosophy shift requires encoding Socratic questioning, hypothesis proposing, and confirmation loops into all agent workflows.

The key insight is that this phase is about **optimizing for non-linear retrieval** rather than sequential reading. Current artifacts assume agents read top-to-bottom; the new design assumes agents grep headers/tags first, then surgically read relevant sections. WORK.md transitions from ephemeral to perpetual, with user-controlled housekeeping to manage growth.

**Primary recommendation:** Implement a three-part WORK.md structure (Current Understanding → Key Events Index → Atomic Log), enforce single-line grep patterns for all discovery operations, and systematically update all workflows to embed the pair programming philosophy with explicit confirmation loops and hypothesis-driven exploration.

## Standard Stack

### Core

| Tool | Version | Purpose | Why Standard |
|------|---------|---------|--------------|
| ripgrep (rg) | 14.x+ | Pattern matching in files | Industry standard for fast regex search, respects .gitignore, line-oriented by default |
| fs-mcp | Latest | MCP server for filesystem ops | Provides `grep_content` tool with ripgrep backend, standardized for Claude/MCP integration |
| Markdown | CommonMark | Artifact format | Plain-text, AI-parseable, grep-friendly, human-readable |

### Supporting

| Tool | Version | Purpose | When to Use |
|------|---------|---------|-------------|
| jq | 1.7+ | JSON parsing for testing | Validate grep output format, parse MCP tool responses |
| uv/uvx | Latest | Run fs-mcp server | Testing grep patterns against live MCP server |

### Alternatives Considered

| Instead of | Could Use | Tradeoff |
|------------|-----------|----------|
| ripgrep single-line | ripgrep multiline (-U) | Single-line is simpler, more portable, easier to reason about. Multiline adds complexity without benefit for this use case. |
| fs-mcp grep_content | Built-in Grep tool in Claude Code | fs-mcp is more portable across MCP-enabled environments, but built-in Grep is acceptable if available. |
| Grep-first workflow | Read-entire-file workflow | Grep-first scales to large files (1000+ lines), read-entire-file works only for small artifacts. |

**Installation:**

```bash
# ripgrep (if not already installed)
brew install ripgrep  # macOS
apt install ripgrep   # Debian/Ubuntu

# fs-mcp for testing
uvx fs-mcp  # Runs MCP server on stdio
```

## Architecture Patterns

### Recommended WORK.md Structure

```
## 1. Current Understanding (Read First)
[30-second context handoff - always small, always up-to-date]

## 2. Key Events Index (Query Accelerator)
[Table with Type, Log ID, Task, Summary columns - grep accelerator]

## 3. Atomic Session Log (Chronological)
[LOG-NNN] - [timestamp] - [TYPE] - Task: TASK-ID
**Summary:** [one-liner]
**Details:** [verbose with code snippets]
```

### Pattern 1: Grep-First Discovery

**What:** Agent discovers structure by grepping headers, then surgically reads sections.

**When to use:** Any time agent needs to understand artifact content without reading full file.

**Example:**

```python
# Step 1: Discover structure
grep_content(pattern='^## ', search_path='gsd-lite/WORK.md')
# Returns: Line 10: ## 1. Current Understanding
#          Line 45: ## 2. Key Events Index
#          Line 60: ## 3. Atomic Session Log

# Step 2: Read specific section
read_files([{
  'path': 'gsd-lite/WORK.md',
  'start_line': 45,
  'end_line': 59
}])
```

**Source:** [fs-mcp grep_content tool documentation](https://glama.ai/mcp/servers/@safurrier/mcp-filesystem) and reference file `ripgrep_tool.json`

### Pattern 2: Log Entry Lookup by ID

**What:** Use unique [LOG-NNN] prefix to jump directly to specific entry.

**When to use:** Agent needs details about specific decision/discovery without scanning entire log.

**Example:**

```python
# Find specific log entry
grep_content(pattern='^\[LOG-028\]', search_path='gsd-lite/WORK.md')
# Returns: Line 245: [LOG-028] - [2026-01-27 11:20] - [BLOCKER]

# Read that entry plus context
read_files([{
  'path': 'gsd-lite/WORK.md',
  'start_line': 244,
  'end_line': 252
}])
```

### Pattern 3: Type-Filtered Discovery

**What:** Use log type tags to find all entries of a specific type.

**When to use:** Agent needs all decisions, or all blockers, without reading chronological log.

**Example:**

```python
# Find all decisions
grep_content(pattern='\[DECISION\]', search_path='gsd-lite/WORK.md')
# Returns multiple lines with [DECISION] tag

# Agent reads index first to see decision summaries
# Then selectively reads full details for relevant decisions
```

### Pattern 4: Task-Specific Filtering

**What:** Multi-task projects use Task: TAG in log entries for filtering.

**When to use:** Projects with parallel tasks need to isolate logs by task.

**Example:**

```python
# Find all logs for MODEL-A task
grep_content(pattern='Task: MODEL-A', search_path='gsd-lite/WORK.md')
```

### Pattern 5: Pair Programming Confirmation Loop

**What:** Agent ends every substantive response with explicit handoff to user.

**When to use:** Always, in all workflows, to ensure collaborative thinking not just task execution.

**Example format:**

```markdown
[Agent explains concept with analogy]

Does this match your mental model?

[YOUR TURN] - What would you like to explore next?
```

**Source:** `prompt_guided_learning.md` - full guided learning prompt with Socratic method

### Anti-Patterns to Avoid

- **Multiline grep patterns:** Single-line only for portability and simplicity
- **Reading entire file first:** Always grep to discover, then surgical read
- **Ephemeral WORK.md:** WORK.md is now perpetual, only archived during housekeeping
- **Auto-proposing fixes:** Agent should propose hypotheses and challenge assumptions, not jump to solutions
- **Task-executor framing:** Agent is navigator, user is driver - collaborative exploration not hierarchical execution
- **First-turn artifact writes:** Agent must talk to user first, never write to artifacts on initial turn

## Don't Hand-Roll

| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| Markdown grep patterns | Custom parser | ripgrep with `^##` pattern | ripgrep is optimized, battle-tested, respects gitignore |
| Log ID sequencing | Custom counter | Zero-padded sequential IDs | Grep-friendly, sortable, human-readable |
| MCP filesystem access | Direct file I/O | fs-mcp `grep_content` | Standardized protocol, works across MCP environments |
| Session handoff format | Custom serialization | Structured markdown sections | Plain-text, AI-parseable, version-controllable |

**Key insight:** The hard problems (efficient grep, portable patterns, MCP integration) are already solved. Focus on structuring content to leverage these existing tools, not building new tools.

## Common Pitfalls

### Pitfall 1: Overengineering Grep Patterns

**What goes wrong:** Using complex multiline patterns or lookahead/lookbehind assertions that don't work consistently across grep implementations.

**Why it happens:** Trying to match multiple sections in one pattern to "optimize" grep calls.

**How to avoid:** Stick to single-line patterns. Use multiple grep calls if needed. Pattern should match one line, then use read_files for context.

**Warning signs:**
- Pattern uses `\n` or multiline flags
- Pattern requires PCRE2 features
- Pattern is longer than 50 characters

### Pitfall 2: Forgetting to Update Cross-References

**What goes wrong:** Updating WORK.md structure but forgetting to update PROTOCOL.md, workflow files, or README to reference the new structure. Agents read old instructions and write to wrong sections.

**Why it happens:** Changes span multiple files. Easy to update one and forget others.

**How to avoid:**
1. Grep for all references: `rg "WORK.md" gsd-lite/template/`
2. Update each reference found
3. Test with golden sample to verify

**Warning signs:**
- Workflows reference "ephemeral WORK.md" after making it perpetual
- PROTOCOL.md describes 2-part structure when it's now 3-part
- STATE.md template references deprecated fields

### Pitfall 3: First-Turn Artifact Writes (Bug)

**What goes wrong:** Agent reads artifacts on first turn and immediately writes "moodboard content" to INBOX.md without talking to user first.

**Why it happens:** Workflow says "read artifacts" but doesn't explicitly say "talk to user before writing."

**How to avoid:** Add explicit first-turn protocol to all workflows:
1. Read PROTOCOL.md (silently)
2. Read STATE.md (silently)
3. **Talk to user first:** "Here's what I understand... What would you like to explore?"
4. Only write after conversing

**Warning signs:**
- Agent writes to INBOX.md on first message
- Agent proposes plan before discussing with user
- Agent executes without understanding context

### Pitfall 4: Scope Creep from Exploration

**What goes wrong:** New pair programming philosophy encourages exploration, but agent expands scope mid-phase instead of capturing discoveries to INBOX.

**Why it happens:** Tension between "explore together" and "stay disciplined on scope."

**How to avoid:**
- Exploration is encouraged BEFORE plan approval (moodboard/whiteboard)
- After plan approval, discoveries go to INBOX
- User can choose to pivot plan, but agent doesn't auto-expand

**Warning signs:**
- Agent adds tasks mid-phase without user approval
- Agent says "while we're at it, let's also..."
- Phase scope grows beyond original plan

### Pitfall 5: Verbose Index Defeating Grep Purpose

**What goes wrong:** Key Events Index becomes too detailed, defeating the purpose of "grep to summarize, read for details."

**Why it happens:** Agent puts full context in index instead of using it as pointer to log entries.

**How to avoid:** Index summary should be 1-line max, 10 words or less. Full context lives in atomic log entry, not index.

**Warning signs:**
- Index entry spans multiple lines
- Index duplicates content from log entry
- Agent reads index but still needs to read full log

## Code Examples

Verified patterns for implementation:

### Single-Line Header Discovery

```bash
# Find all section headers in WORK.md
rg "^## " gsd-lite/WORK.md --line-number

# Output:
# 10:## 1. Current Understanding (Read First)
# 45:## 2. Key Events Index (Query Accelerator)
# 60:## 3. Atomic Session Log (Chronological)
```

**Source:** [ripgrep best practices](https://learnbyexample.github.io/learn_gnugrep_ripgrep/ripgrep.html)

### Log ID Lookup

```bash
# Find specific log entry
rg "^\[LOG-028\]" gsd-lite/WORK.md --line-number

# Output:
# 245:[LOG-028] - [2026-01-27 11:20] - [BLOCKER]
```

### Type-Filtered Search

```bash
# Find all decisions
rg "\[DECISION\]" gsd-lite/WORK.md --line-number

# Find all blockers
rg "\[BLOCKER\]" gsd-lite/WORK.md --line-number

# Find all discoveries with code snippets
rg "\[DISCOVERY\]" gsd-lite/WORK.md --line-number
```

### Task-Specific Search

```bash
# Find all logs for MODEL-A task
rg "Task: MODEL-A" gsd-lite/WORK.md --line-number --context 2

# Context flag shows 2 lines before/after for quick preview
```

### MCP grep_content Usage

```python
# Via fs-mcp MCP server
grep_content(
    pattern="^## ",
    search_path="gsd-lite/WORK.md"
)

# Returns structured output:
# {
#   "result": "File: gsd-lite/WORK.md, Line: 10, Matched: ## 1. Current Understanding\n..."
# }
```

**Source:** `ripgrep_tool.json` reference file

## State of the Art

| Old Approach | Current Approach | When Changed | Impact |
|--------------|------------------|--------------|--------|
| Sequential read of full artifacts | Grep-first, surgical read | 2026 (this phase) | Scales to 1000+ line files without context overflow |
| Ephemeral WORK.md (deleted after promotion) | Perpetual WORK.md (archived during housekeeping) | 2026 (this phase) | Enables PR extraction from full session history |
| Hierarchical coaching ("User = visionary, Agent = builder") | Pair programming ("User + Agent = thinking partners") | 2026 (this phase) | Agent proposes hypotheses, challenges assumptions, teaches concepts |
| Promotion workflow only | Unified housekeeping workflow | 2026 (this phase) | Single workflow handles both PR extraction AND archiving |
| STATE.md + WORK.md separate | STATE.md merged into WORK.md | 2026 (this phase) | Fewer files reduces confusion for weak agents |

**Deprecated/outdated:**

- **Ephemeral WORK.md language:** Remove all references to "deleted after phase promotion" from workflows
- **STATE.md as separate file:** Merge into WORK.md Current Understanding section
- **Promotion-only workflow:** Replace with housekeeping workflow that handles both use cases
- **Task-executor agent framing:** Remove "User owns outcome. Agent executes." from all workflows

## Open Questions

1. **Log ID Padding**
   - What we know: Zero-padded IDs (LOG-001, LOG-028) are grep-friendly and sortable
   - What's unclear: How many digits to pad? 3 digits (001-999) or 4 digits (0001-9999)?
   - Recommendation: Start with 3 digits. If a single WORK.md exceeds 999 entries, that's a signal to run housekeeping/archive to HISTORY.md. Prefer frequent housekeeping over larger padding.

2. **Index Update Frequency**
   - What we know: Key Events Index summarizes important log entries
   - What's unclear: Should index be updated after every log entry, or batch-updated at phase end?
   - Recommendation: Update index only for "major" entries (VISION, DECISION, BLOCKER, DISCOVERY with code). Skip EXEC/PLAN entries unless they're phase-changing. Agent's discretion to decide what's "major."

3. **HISTORY.md Format for Archived Entries**
   - What we know: HISTORY.md stores completed phases as one-liners
   - What's unclear: Should archived WORK.md entries be appended in full, or should we only keep phase summaries?
   - Recommendation: HISTORY.md stays minimal (one line per phase). Full archived logs can live in separate dated files (e.g., `HISTORY/2026-01-27-MODEL-A.md`) if user wants to preserve them. Default is to discard after PR extraction.

4. **Multi-Task Index Structure**
   - What we know: Projects can have parallel tasks
   - What's unclear: Should Key Events Index have a separate "Task" column, or should tasks be encoded in Summary column?
   - Recommendation: Add Task column to index table for easy filtering. Format: `| Type | Log ID | Task | Summary |`

## Sources

### Primary (HIGH confidence)

- **ripgrep official documentation** - [CLI text processing with ripgrep](https://learnbyexample.github.io/learn_gnugrep_ripgrep/ripgrep.html) - Pattern matching fundamentals
- **fs-mcp grep_content tool** - Reference file `ripgrep_tool.json` - MCP integration patterns
- **WORK.prod.md** - Production session artifact showing verbose logging with code snippets
- **suggestions.md** - V2 proposal for grep-friendly structure with Key Events Index
- **prompt_guided_learning.md** - Full guided learning prompt with Socratic method, confirmation loops
- **prod_session.md** - Detailed session flow demonstrating pair programming in action

### Secondary (MEDIUM confidence)

- **fs-mcp MCP server** - [MCP Filesystem Server documentation](https://glama.ai/mcp/servers/@safurrier/mcp-filesystem) - grep_files tool parameters and workflow
- **Markdown-based documentation systems** - [Building a Markdown-Based Documentation System (2026)](https://medium.com/@rosgluk/building-a-markdown-based-documentation-system-72bef3cb1db3) - AI-friendly documentation patterns
- **ugrep patterns for markdown** - [ugrep file pattern searcher](https://github.com/Genivia/ugrep) - Advanced grep for structured text

### Tertiary (LOW confidence)

- **General ripgrep best practices** - [Ripgrep cheatsheet](https://skerritt.blog/ripgrep-cheatsheet/) - Community patterns (not phase-specific)

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH - ripgrep is industry standard, fs-mcp is proven MCP implementation
- Architecture: HIGH - Patterns verified in production usage (WORK.prod.md, prod_session.md)
- Pitfalls: HIGH - Identified from actual session bugs and context document decisions
- Grep patterns: HIGH - Tested against ripgrep documentation and fs-mcp tool spec
- Pair programming philosophy: HIGH - Detailed prompt and session transcript provide clear behavioral model
- Open questions: MEDIUM - Implementation details left to planner's discretion with recommendations

**Research date:** 2026-01-27
**Valid until:** 60 days (stable domain - grep patterns and markdown workflows don't change rapidly)

---

## Implementation Notes for Planner

### Critical Path Items

1. **WORK.md template redesign** - Three-part structure is foundation for all grep patterns
2. **Workflow updates** - ALL workflows must be updated for coherence (moodboard, execution, promotion, checkpoint)
3. **First-turn protocol** - Must be added to ALL workflows to prevent artifact-write bug
4. **Pair programming encoding** - Must be woven throughout ALL workflows, not just moodboard

### Testing Strategy

1. Create golden sample WORK.md with 50-100 log entries representing all 6 types
2. Spin up `uvx fs-mcp` server
3. Send real grep queries via stdio: header discovery, log ID lookup, type filtering, task filtering
4. Verify line numbers returned are correct
5. Verify surgical read retrieves expected content

### Template Coherence Checklist

Before marking phase complete, verify:

- [ ] No workflow references "ephemeral WORK.md"
- [ ] No workflow references STATE.md as separate file
- [ ] All workflows include first-turn protocol
- [ ] All workflows include pair programming behaviors
- [ ] PROTOCOL.md describes new 3-part WORK.md structure
- [ ] README explains philosophy and grep workflow
- [ ] All log entry examples use [LOG-NNN] format with 6 types

### README Revamp Requirements

Per context document decisions:

- **Philosophy section** - Why pair programming? Why perpetual WORK.md?
- **Artifact overview** - What each file does, how they connect (with grep workflow)
- **Typical session walkthrough** - Example showing moodboard → exploration → execution → housekeeping
- **Workflow diagram** - Mermaid showing artifact lifecycle
- **Maintainer's Guide** - Semantic CICD for future iterations (testing, updating, preserving vision)
- **Tone** - Conversational, show don't tell, explain "why" not just "what"

### Validation Approach

**Human walkthrough (final step):** User reads through ALL updated templates to verify:
1. Coherent story across all files
2. No contradictions between workflows
3. Examples are consistent
4. Tone matches pair programming philosophy
5. No orphan references to deprecated patterns
