# Prime Directive: Development Excellence Standards

**Last Updated:** December 1, 2025  
**Purpose:** Ensure high-quality, maintainable code through proven practices, systematic verification, and continuous learning.

**Target Audience:** All developers, AI agents, code reviewers, and contributors to edaflow

---

## 📋 Quick Start Checklist

**Before writing ANY code:**
- [ ] Read relevant sections of this document
- [ ] Verify existing code patterns using search tools
- [ ] Establish test baseline (current pass rate + warning count)
- [ ] Understand the complete requirement

**During development:**
- [ ] Test incrementally after each logical change
- [ ] Handle None/edge cases defensively
- [ ] Maintain 100% pass rate + zero warnings
- [ ] Follow established patterns in codebase

**Before committing:**
- [ ] All tests passing (verify with full test run)
- [ ] Zero warnings (investigate and fix all)
- [ ] Code reviewed (self or peer)
- [ ] Commit message includes test count and warning status

---

## 🎯 Core Principles

### 0. **Quality Gate: 100% Pass + Zero Warnings**

**All tests must pass with zero warnings before AND after ANY code changes. No exceptions.**

#### The Verification Protocol

```
1. 📊 BASELINE: Run full test suite BEFORE changes
   └─> Document: "X tests passed, 0 warnings"
   
2. ✏️  CHANGE: Make ONE logical change at a time
   └─> Keep scope small and focused
   
3. 🧪 VERIFY: Run full test suite AFTER change
   └─> Must achieve: Same X tests passed, 0 warnings
   
4. 🚨 FAILURE: If tests fail OR warnings appear
   └─> Fix immediately OR revert the change
   └─> Do NOT proceed until clean
   
5. ✅ COMMIT: Only when all tests pass with zero warnings
   └─> Message format: "Feature X (393 passed, 0 warnings)"
```

#### Why Warnings Matter

| Warning Type | Why Critical | Example Impact |
|-------------|--------------|----------------|
| **Deprecations** | Becomes breaking error in next version | SQLAlchemy 2.0 migration |
| **Type Mismatches** | Runtime errors in production | None comparison crashes |
| **Anti-patterns** | Performance/security issues | Inefficient queries |
| **Import Issues** | Future compatibility breaks | Module reorganization |

**Technical Debt Equation:**
```
1 ignored warning today = 2 hours debugging tomorrow
```

#### Absolute Rules

**❌ NEVER:**
- Skip baseline verification
- Dismiss warnings as "not important"
- Make multiple unrelated changes at once
- Commit with failing tests OR warnings
- Defer fixing test failures or warnings
- Use `# type: ignore` without understanding why

**✅ ALWAYS:**
- Run full test suite before starting work
- Investigate every warning immediately
- Fix warnings before proceeding with new features
- Document test count AND warning count in commits
- Preserve git history when removing code
- Ask for help if warning is unclear

### 1. **Research First, Code Second**

**Never assume how existing code works. Always verify through systematic research.**

#### The Research Process

```
For New Features:
1. Find method signatures    → grep_search for definitions
2. Verify enum values        → read_file to see actual values
3. Check base classes        → list_code_usages for inheritance
4. Find usage examples       → grep_search with regex patterns
5. Document findings         → Add reference comments in code

For Integration Tests:
1. List all components       → Write comprehensive component list
2. Find existing tests       → grep_search for test_*.py files
3. Read ALL existing tests   → Note every API detail
4. Document API reference    → Create comment block with verified APIs
5. Verify ambiguous cases    → Read source only when needed
```

#### Common Research Mistakes

**❌ WRONG - Assumption-Based:**
```python
# Assumed without checking
timeframe = TimeFrame.DAILY        # Does DAILY exist?
def on_bar(self, symbol, bar):     # Is this the signature?
position = tracker.status          # What's the attribute name?
```

**✅ RIGHT - Verification-Based:**
```python
# Step 1: Research via grep_search("class TimeFrame")
# Found: DAY_1, HOUR_1, MINUTE_1, MINUTE_5, MINUTE_15

# Step 2: Research via grep_search("def on_bar.*Strategy")
# Found: def on_bar(self, market_data: MarketData) -> None

# Step 3: Use verified APIs
timeframe = TimeFrame.DAY_1
def on_bar(self, market_data):
    position = self.get_position(symbol) or 0  # Verified Optional[int]
```

#### Time Investment vs. Debugging Cost

| Approach | Research | Writing | Debugging | Total |
|----------|----------|---------|-----------|-------|
| **Assume first** | 0 min | 60 min | 120 min | **180 min** |
| **Verify first** | 30 min | 60 min | 10 min | **100 min** |
| **Savings** | | | | **44%** |

**Key Insight:** 30 minutes of research saves 2 hours of debugging.

### 2. **Defensive Programming is Mandatory**

**Assume inputs can fail. Handle None, validate bounds, check types.**

#### The Defense Layers

```
Layer 1: Input Validation
  ├─> Check for None/empty
  ├─> Validate type matches expected
  ├─> Verify bounds (positive, in range)
  └─> Fail fast with clear error

Layer 2: Safe Defaults
  ├─> Use `or` operator for None: `value or default`
  ├─> Provide sensible fallbacks
  └─> Document default behavior

Layer 3: Error Handling
  ├─> Try-except for risky operations
  ├─> Catch specific exceptions
  ├─> Log errors with context
  └─> Return safe value or raise clearly
```

#### Common Defensive Patterns

**❌ DANGEROUS - No None Handling:**
```python
position = self.get_position(symbol)
if position <= 0:  # 💥 TypeError if None!
    self.buy(symbol, 100)
```

**✅ SAFE - None Handling:**
```python
# Option 1: Default with 'or'
position = self.get_position(symbol) or 0
if position <= 0:
    self.buy(symbol, 100)

# Option 2: Explicit check
position = self.get_position(symbol)
if position is None:
    position = 0
if position <= 0:
    self.buy(symbol, 100)

# Option 3: Early return
position = self.get_position(symbol)
if position is None:
    logger.warning(f"No position data for {symbol}")
    return
```

**❌ DANGEROUS - No Bounds Checking:**
```python
def calculate_sharpe(returns: List[float]) -> float:
    return np.mean(returns) / np.std(returns)  # 💥 Division by zero!
```

**✅ SAFE - Bounds Checked:**
```python
def calculate_sharpe(returns: List[float]) -> float:
    if not returns or len(returns) < 2:
        return 0.0
    
    std = np.std(returns)
    if std == 0:
        return 0.0
    
    return np.mean(returns) / std
```

#### Type Hints + Runtime Checks

```python
from typing import Optional, List

def process_trades(
    trades: List[Trade],
    min_size: Optional[int] = None
) -> List[Trade]:
    """
    Filter trades by minimum size.
    
    Args:
        trades: List of Trade objects (can be empty)
        min_size: Minimum trade size (None = no filter)
    
    Returns:
        Filtered list (empty if no matches)
    """
    # Defensive: handle empty input
    if not trades:
        return []
    
    # Defensive: handle None min_size
    if min_size is None:
        return trades
    
    # Defensive: validate min_size
    if min_size <= 0:
        raise ValueError(f"min_size must be positive, got {min_size}")
    
    return [t for t in trades if t.size >= min_size]
```

### 3. **Test Incrementally, Not in Batches**

**Build and verify in small steps. Never write 300 lines before first test.**

#### The Incremental Build Pattern

```
Traditional (Risky):
  Write 300 lines → Test → 50 errors → Debug 2 hours → Done
  
Incremental (Safe):
  Write 50 lines → Test → 2 errors → Fix 10 min →
  Write 50 lines → Test → 1 error → Fix 5 min →
  Write 50 lines → Test → 0 errors → Done
  
Result: Same LOC, 75% less debugging time
```

#### Component-First Development

**❌ WRONG - Big Bang Integration:**
```python
# Write all 300 lines first, then test
class ComplexStrategy(Strategy):
    def __init__(self, config):
        # 50 lines of initialization
        
    def on_bar(self, market_data):
        # 100 lines of logic
        
    def calculate_signals(self, data):
        # 80 lines of calculations
        
    def execute_trades(self, signals):
        # 70 lines of execution

# Run and discover 20 errors across all methods 😱
```

**✅ RIGHT - Component-First Testing:**
```python
# Step 1: Test initialization only
class ComplexStrategy(Strategy):
    def __init__(self, config):
        self.config = config
        self.indicators = {}

# Quick test
strategy = ComplexStrategy(config)
print(f"✓ Initialized with {len(strategy.indicators)} indicators")

# Step 2: Add and test one method
def calculate_signals(self, data):
    # Implementation
    return signals

# Test it
signals = strategy.calculate_signals(test_data)
assert len(signals) > 0, "Should generate signals"
print(f"✓ Generated {len(signals)} signals")

# Step 3: Add next method and test
# ... continue incrementally

# Step 4: Integration test (components already verified!)
result = engine.run()  # High confidence this will work
```

#### The 50-Line Rule

**Guideline:** Test after every ~50 lines of new code

```python
# ✅ Good rhythm:
Write component A (50 lines) → Test → ✓
Write component B (50 lines) → Test → ✓  
Write component C (50 lines) → Test → ✓
Integrate A+B+C → Test → ✓ (high confidence)

# ❌ Bad rhythm:
Write A+B+C (300 lines) → Test → 💥 (debugging nightmare)
```

---

---

## 🔍 Pre-Implementation Research Protocol

**This section is mandatory reading before writing ANY code that integrates existing components.**

### Research Decision Tree

```
Are you writing integration tests?
├─ YES → Use Integration Test Research Protocol (below)
│         Time: 20-40 min | Success Rate: 80%+ first run
│
└─ NO → Are you using existing classes/methods?
    ├─ YES → Use Feature Research Protocol (below)
    │         Time: 10-20 min | Prevents: API mismatches
    │
    └─ NO → Skip research, proceed with TDD
              (You're implementing from scratch)
```

### ☑️ Feature Research Protocol

**Use this when implementing features that call existing code.**

#### Step-by-Step Process

1. **Identify Dependencies**
   ```python
   # List all external classes/methods you'll use
   dependencies = [
       "TimeFrame enum",
       "Strategy.on_bar() method",
       "PositionManager.get_position() method",
       "OrderType enum"
   ]
   ```

2. **Research Each Dependency**
   ```python
   # For enums
   grep_search(query="class TimeFrame", includePattern="*.py")
   read_file("path/to/file.py", offset=line, limit=30)
   # Result: DAY_1, HOUR_1, MINUTE_1 (not DAILY!)
   
   # For methods
   grep_search(query="def on_bar", includePattern="strategy.py")
   read_file("strategy.py", offset=line, limit=20)
   # Result: def on_bar(self, market_data: MarketData) -> None
   
   # For return types
   grep_search(query="def get_position", includePattern="*.py")
   # Result: -> Optional[int]  (can be None!)
   ```

3. **Find Usage Examples**
   ```python
   # See how others use it
   grep_search(query="TimeFrame\\.DAY_1", isRegexp=True)
   grep_search(query="on_bar.*market_data", isRegexp=True)
   # Learn from existing working code
   ```

4. **Document Your Findings**
   ```python
   """
   API Reference (verified 2025-12-01):
   
   TimeFrame (data_models.py):
   - Values: DAY_1, HOUR_1, MINUTE_1, MINUTE_5, MINUTE_15
   - Usage: TimeFrame.DAY_1 (not DAILY)
   
   Strategy.on_bar (strategy_base.py):
   - Signature: def on_bar(self, market_data: MarketData) -> None
   - market_data.symbols is List[str] (plural!)
   - market_data.get_bar(symbol) returns Optional[Bar]
   
   PositionManager.get_position (position_manager.py):
   - Signature: def get_position(self, symbol: str) -> Optional[int]
   - Returns None if no position (must handle!)
   - Use: position = self.get_position(symbol) or 0
   """
   ```

#### Quick Reference Checklist

- [ ] Method signatures verified (params, types, order)
- [ ] Return types checked (Optional? List? Can be None?)
- [ ] Enum values confirmed (actual names, not assumed)
- [ ] Required imports identified
- [ ] Usage examples found
- [ ] API reference comment written

#### For Integration Tests of Existing Code (NEW):
**⚠️ CRITICAL: Research APIs BEFORE writing integration tests**

1. **List All Components to Integrate**
   ```python
   # Write down every class/module you'll test together
   # Example: StrategyLifecycle, PaperMetricsTracker, StrategyOrchestrator, RiskManager
   ```

2. **Find Existing Tests for Each Component**
   ```python
   # These tests show the CORRECT API usage
   grep_search(query="test.*ComponentName", includePattern="test_*.py")
   # Result: tests/test_component.py
   ```

3. **Read Existing Tests (API Documentation)**
   ```python
   # Read the test file completely
   read_file("tests/test_component.py")
   # Note ALL constructor signatures, method calls, return values
   # Pay special attention to:
   # - Parameter order (db_path first? strategy_name second?)
   # - Parameter count (5 params not 3?)
   # - Method names (get_report not validate?)
   # - Attribute names (overall_passed not passed?)
   # - Return types (dict not object? list not single item?)
   ```

4. **Verify Ambiguous Cases from Implementation**
   ```python
   # If existing tests don't clarify something, check source
   grep_search(query="class ComponentName", includePattern="*.py")
   read_file("path/to/component.py", offset=line, limit=50)
   ```

5. **Document API Reference in Comments**
   ```python
   # Create comprehensive API reference block at top of test file
   """
   API Reference (verified from existing tests):
   
   ComponentA (from test_component_a.py):
   - Constructor: ComponentA(param1: type, param2: type)
   - method1(arg1, arg2) -> return_type
   - attribute_name (not attribute_name_variant!)
   
   ComponentB (from test_component_b.py):
   - Constructor: ComponentB(db_path, name, initial_val) - db_path FIRST
   - method2(p1, p2, p3, p4, p5) - 5 params required!
   - Returns dict with keys: 'key1', 'key2' (not object)
   """
   ```

6. **Verify Integration Points**
   ```python
   # Check how components interact in existing code
   grep_search(query="ComponentA.*ComponentB", isRegexp=True)
   # Look for: data passing patterns, callback signatures, event handling
   ```

**Integration Test Research Checklist:**
- [ ] All components listed
- [ ] Existing tests found for each (test_*.py files)
- [ ] Existing tests read and APIs documented
- [ ] Constructor signatures verified (param order, types, required vs optional)
- [ ] Method signatures verified (names, param counts, return types)
- [ ] Attribute names verified (singular/plural, exact spelling)
- [ ] Enum values verified (actual values, not assumed)
- [ ] Integration patterns verified (how components interact)
- [ ] API reference comment block created
- [ ] Ambiguous cases checked in implementation
- [ ] Expected 80%+ first-run pass rate

**Time Investment:**
- API Research: 20-40 minutes (depends on component count)
- Test Writing: 40-60 minutes (faster with verified APIs)
- Debugging: 5-15 minutes (minimal with correct APIs)
- **Total: 65-115 minutes** (vs 180+ minutes without research)

**When to Skip Research:**
- Unit tests for code you just implemented (you know the API)
- Tests written alongside new implementation (TDD)
- You implemented the component recently (within same sprint)

### ☑️ Before Calling Any Method

- [ ] Checked the method signature (parameters, types, order)
- [ ] Verified return type (can it be None? Optional?)
- [ ] Checked for required imports
- [ ] Looked for existing usage examples
- [ ] Understood parameter types (enum vs string, int vs Optional[int])

### ☑️ Before Using Enums/Constants

- [ ] Verified actual enum value names (not assumed)
- [ ] Checked if it's a string enum or value enum
- [ ] Imported the enum properly

---

## 🚨 Common Pitfalls & Solutions

### Pitfall 1: Wrong Enum Values
**Problem:** Assuming enum names without checking
```python
TimeFrame.DAILY  # ❌ Doesn't exist
```
**Solution:** Always verify enum definitions
```python
TimeFrame.DAY_1  # ✅ Actual value
```
**Prevention:** Read enum source code first, add reference comment:
```python
# TimeFrame: DAY_1, HOUR_1, MINUTE_1, MINUTE_5, etc.
```

### Pitfall 2: Method Signature Mismatch
**Problem:** Implementing interface without checking base class
```python
def on_bar(self, symbol, bar):  # ❌ Wrong signature
```
**Solution:** Check abstract method in base class
```python
def on_bar(self, market_data):  # ✅ Matches Strategy base class
```
**Prevention:** Use `grep_search` to find base class method signature

### Pitfall 3: Attribute Name Assumptions
**Problem:** Guessing attribute names
```python
market_data.symbol  # ❌ It's plural!
```
**Solution:** Check class definition
```python
market_data.symbols  # ✅ Returns list
market_data.get_bar(symbol)  # ✅ Gets bar for symbol
```
**Prevention:** Read class docstrings and inspect attributes

### Pitfall 4: None Type Errors
**Problem:** Not handling None return values
```python
position = get_position(symbol)
if position > 0:  # ❌ Crashes if None
```
**Solution:** Always handle None explicitly
```python
position = get_position(symbol) or 0
if position > 0:  # ✅ Safe
```
**Prevention:** Check return type hints: `Optional[T]` means None is possible

### Pitfall 5: Parameter Type Confusion
**Problem:** Passing string where enum expected
```python
self.buy(symbol, quantity, "market order")  # ❌ String not OrderType
```
**Solution:** Use proper enum type
```python
self.buy(symbol, quantity, OrderType.MARKET)  # ✅ Enum
```
**Prevention:** Check method signature for parameter types

---

## 🛠️ Mandatory Tools Usage

### Before Implementation

| Task | Tool | Example |
|------|------|---------|
| Find method definition | `grep_search` | `grep_search(query="def on_bar", includePattern="strategy.py")` |
| Read enum values | `read_file` | `read_file("data_models.py", offset=17, limit=30)` |
| Check class usage | `list_code_usages` | `list_code_usages(symbolName="Strategy")` |
| Find patterns | `grep_search` (regex) | `grep_search(query="TimeFrame\\.", isRegexp=True)` |
| Verify structure | `read_file` | `read_file("file.py", offset=50, limit=100)` |

### During Development

| Task | Tool | Example |
|------|------|---------|
| Run tests | `run_in_terminal` | `python test_module.py` |
| Check errors | `get_errors` | `get_errors(filePaths=["path/to/file.py"])` |
| Verify changes | `read_file` | Read modified sections |

---

## 📋 Development Workflow (The Right Way™)

### Phase 1: Research (15-30% of time)
```
For New Feature Implementation:
1. Understand the requirement
2. Find relevant existing code
3. Check base classes and interfaces
4. Verify data structures and enums
5. Look for usage examples
6. Document findings in comments

For Integration Tests (ADDITIONAL - MANDATORY):
1. List ALL components to integrate
2. Find existing tests for EACH component (test_*.py)
3. Read existing tests COMPLETELY (note every API detail)
4. Verify constructor signatures (param order, types, counts)
5. Verify method signatures (names, params, return types)
6. Verify attribute names (exact spelling, singular/plural)
7. Verify integration patterns (how components interact)
8. Document comprehensive API reference in comments
9. Expected outcome: 80%+ first-run pass rate
```

**Integration Test Research is NON-NEGOTIABLE:**
- Skipping this step leads to 37+ API mismatches (Sprint 4 Task 1 experience)
- 30 minutes research saves 2 hours debugging
- Existing tests are your API documentation
- Write the API reference comment block BEFORE writing tests

### Phase 2: Design (10-15% of time)
```
1. Sketch out the implementation
2. Identify dependencies
3. Plan for error handling
4. Consider edge cases
5. Keep it simple initially
```

### Phase 3: Implementation (40-50% of time)
```
1. Start with minimal working code
2. Match signatures exactly
3. Add defensive checks (None, bounds, types)
4. Use proper types (enums, not strings)
5. Add logging/print statements for debugging
6. Keep functions small and focused
```

### Phase 4: Testing (20-30% of time)
```
1. Test each component in isolation
2. Add unit tests as you go
3. Run tests frequently (after each component)
4. Fix errors immediately - don't accumulate
5. Verify integration works
6. Run full test suite before committing
```

---

## ✅ Code Quality Standards

### Type Safety
```python
# ✅ GOOD: Use type hints
def calculate_position_size(
    self, 
    symbol: str, 
    price: float,
    fraction: Optional[float] = None
) -> int:
    ...

# ✅ GOOD: Handle Optional types
position: Optional[int] = self.get_position(symbol)
if position is None:
    position = 0
```

### Error Handling
```python
# ✅ GOOD: Validate inputs
if not symbol or symbol not in self.config.symbols:
    return

# ✅ GOOD: Check preconditions
if len(prices) < self.lookback_period:
    return  # Not enough data

# ✅ GOOD: Handle exceptions gracefully
try:
    result = risky_operation()
except SpecificError as e:
    log.warning(f"Operation failed: {e}")
    return default_value
```

### Documentation
```python
# ✅ GOOD: Document assumptions and constraints
def on_bar(self, market_data):
    """
    Process market data bar
    
    Note: MarketData.symbols is a list (plural)
    Note: get_position() returns Optional[int] - check for None!
    """
    for symbol in market_data.symbols:  # Note: plural!
        position = self.get_position(symbol) or 0  # Handle None
        ...
```

---

## 🎓 Learning from Errors

### Week 4 Day 4 Lessons Learned (November 22, 2025)

#### Lesson 1: Delete Systematically with Test Verification
**Context:** Cleaning up duplicate backtesting modules and legacy scripts
- **What we did right:**
  1. Verified baseline: 138/138 tests passing
  2. Made one logical change (rename backtesting → backtesting_old)
  3. Verified tests still pass: 138/138
  4. Committed with clear message
  5. Made second change (delete backtesting_old/)
  6. Verified tests still pass: 138/138
  7. Committed with clear message
  8. Made third change (delete legacy scripts)
  9. Verified tests still pass: 138/138
  10. Ready to commit

- **Why this matters:** 
  - Each checkpoint provides safety net
  - Can pinpoint exact change if something breaks
  - Git history preserves deleted code
  - 100% confidence in each step

- **Key insight:** "Delete with confidence, verify with discipline"

#### Lesson 2: Legacy Code Cleanup Strategy
**Context:** Found 5 legacy scripts importing deleted module
- **The right way:**
  1. Search for all imports: `grep_search(query="from backtesting|import backtesting", isRegexp=True)`
  2. Analyze each file (is it part of core system?)
  3. Verify none are in current test suite
  4. Delete all at once (they're related)
  5. Verify test suite still passes
  
- **Files deleted:**
  - `optimize_strategy.py` - used deleted optimizer
  - `run_backtest.py` - used deleted visualizer
  - `tests/test_risk_manager.py` - tested deleted component
  - `tests/test_performance_analytics.py` - tested deleted component
  - `tests/test_backtesting.py` - tested deleted module
  
- **Why delete instead of rewrite:** 
  - Week 4 backtest/ module is superior and complete
  - These were superseded, not complementary
  - Maintaining two implementations creates confusion
  - Can recreate if needed (git history preserved)

#### Lesson 3: Module Consolidation
**Context:** Had two backtesting directories causing import confusion
- **Problem indicators:**
  - Developers confused about which to import
  - Duplicate functionality
  - Import errors in new code
  
- **Solution:**
  - Keep the superior, complete implementation (backtest/)
  - Delete the legacy, incomplete implementation (backtesting/)
  - Update all imports (but in this case, old imports were in dead code)
  
- **Lesson:** "One authoritative implementation per concept"

#### Lesson 4: Git History as Safety Net
**Context:** Deleting 13 files totaling 3,000+ lines
- **Why we could delete confidently:**
  - All code preserved in git history (commits da9a714, 8206109)
  - Can retrieve if needed: `git show da9a714:backtesting_old/optimizer.py`
  - Clear commit messages document what was deleted and why
  
- **Best practices:**
  - Commit before major deletions
  - Write detailed commit messages
  - Reference commit hashes in documentation
  - Never force-push deleted code (preserve history)

### Sprint 4 Lessons Learned (November 25, 2025)

#### Lesson 1: Research APIs Before Writing Integration Tests
**Context:** Sprint 4 Task 1 - Writing 30 integration tests for existing components
- **What went wrong:**
  - Wrote 916 lines of integration tests based on assumed APIs
  - 37+ API mismatches discovered during test execution
  - Required 5 correction rounds to reach 100% pass rate
  - Issues: wrong imports, incorrect signatures, missing parameters, wrong attributes
  
- **Why it happened:**
  - Treated integration tests like unit tests (write, then implement)
  - Assumed APIs instead of researching actual implementations
  - No pre-test verification phase for existing code
  
- **The right way - API Research Protocol:**
  ```python
  # BEFORE writing integration tests for existing components:
  
  # 1. List all components you'll integrate
  components = [
      "strategy.lifecycle.StrategyLifecycle",
      "strategy.metrics_tracker.PaperMetricsTracker",
      "strategies.strategy_orchestrator.StrategyOrchestrator",
      # ... etc
  ]
  
  # 2. For EACH component, research its API:
  
  # Step 2a: Find existing tests (they show correct usage)
  grep_search(query="test.*StrategyLifecycle", includePattern="test_*.py")
  # Result: tests/test_strategy_lifecycle.py
  
  # Step 2b: Read the test file to see actual API usage
  read_file("tests/test_strategy_lifecycle.py")
  # Note: StrategyLifecycle(db_path, strategy_name)
  # Note: transition_to(state) returns bool
  # Note: get_current_state() returns LifecycleState
  
  # Step 2c: Find the implementation for ambiguous cases
  grep_search(query="class PaperMetricsTracker", includePattern="*.py")
  read_file("strategy/metrics_tracker.py", offset=line, limit=50)
  # Note: __init__(self, db_path: str, strategy_name: str, initial_capital: float)
  # Note: record_daily_snapshot(date, portfolio_value, cash, positions_value, daily_pnl)
  
  # Step 2d: Document findings in code comments
  # PaperMetricsTracker API:
  # - Constructor: (db_path, strategy_name, initial_capital)
  # - record_daily_snapshot(date, portfolio_value, cash, positions_value, daily_pnl) - 5 params
  # - record_trade(symbol, side, quantity, entry_price, exit_price, entry_time, exit_time)
  # - get_metrics_snapshot() -> MetricsSnapshot
  
  # 3. Create API reference comment block at top of test file
  # 4. NOW write integration tests using verified APIs
  # 5. Tests should pass on first full run (or have minimal fixes)
  ```

- **Time comparison:**
  - Our approach: 0 min research + 60 min writing + 120 min debugging = 180 min
  - Right approach: 30 min research + 60 min writing + 10 min fixes = 100 min
  - **Time saved: 44%** (plus less frustration)

- **How to avoid in future:**
  1. **Add "API Research Phase" before integration tests**
     - Mandatory for tests integrating 3+ existing components
     - Research each component's API from existing tests
     - Document findings in comment block
     - Verify ambiguous cases by reading implementation
  
  2. **Use existing tests as API documentation**
     - Passing tests show correct usage
     - More reliable than reading implementation directly
     - Shows actual parameter values and return types
  
  3. **Create API reference at top of integration test file**
     ```python
     """
     Integration Test Suite: End-to-End Workflows
     
     API Reference (verified from existing tests):
     
     StrategyLifecycle (from test_strategy_lifecycle.py):
     - Constructor: StrategyLifecycle(db_path: str, strategy_name: str)
     - transition_to(state: LifecycleState) -> bool
     - get_current_state() -> LifecycleState
     - get_history() -> List[StateTransition]
     
     PaperMetricsTracker (from test_metrics_tracker.py):
     - Constructor: PaperMetricsTracker(db_path, strategy_name, initial_capital)
     - record_daily_snapshot(date, pv, cash, pos_val, daily_pnl) - 5 params!
     - record_trade(sym, side, qty, entry_px, exit_px, entry_t, exit_t) - needs times!
     - get_metrics_snapshot() -> MetricsSnapshot (not get_summary!)
     
     ... (continue for all integrated components)
     """
     ```
  
  4. **Integration test writing checklist:**
     - [ ] Listed all components to integrate
     - [ ] Found existing tests for each component
     - [ ] Read existing tests to verify APIs
     - [ ] Documented API reference in comments
     - [ ] Verified ambiguous cases from implementation
     - [ ] Created fixtures matching verified signatures
     - [ ] Wrote tests using verified APIs only
     - [ ] Expected high first-run pass rate (80%+)

- **When this applies:**
  - Writing integration tests for existing components
  - Writing E2E tests across multiple modules
  - Adding tests to legacy code
  - Creating system-level test suites
  
- **When research is optional:**
  - Unit testing new code you just wrote (TDD)
  - Integration tests written alongside implementation
  - Tests for code you personally implemented recently

- **Key insight:** "For integration tests of existing code, research APIs first. 30 minutes of research saves 2 hours of debugging."

**Sprint 4 Task 1 Outcome:**
Despite the inefficient approach, we achieved:
- 30/30 integration tests passing (100%)
- 562/562 full test suite passing (100%)
- Comprehensive coverage of all Sprint 1-3 integration points
- Clean commit (11da7f4)

**Lesson learned:** API research phase is mandatory before writing integration tests. Add it to the Prime Directive.

---

### Sprint 3 Lessons Learned (November 24-25, 2025)

#### Lesson 1: Dataclasses + Enums = Clean Architecture
**Context:** Implementing attribution system and health monitoring
- **What worked exceptionally well:**
  - Using dataclasses for data structures (AttributionBreakdown, HealthMetrics, HealthAlert)
  - Using Enums for type safety (AttributionMetric, AttributionPeriod, HealthStatus, AlertLevel)
  - Zero boilerplate, maximum clarity
  - Type hints everywhere = fewer bugs
  
- **Pattern that emerged:**
  ```python
  @dataclass
  class HealthMetrics:
      win_rate: float
      sharpe_ratio: float
      max_drawdown: float
      profit_factor: float
      
      def calculate_health_score(self) -> float:
          # Business logic in the dataclass
          return (self.win_rate * 0.25 + ...)
  ```
  
- **Benefits realized:**
  - Tests are easier to write (clear data creation)
  - Serialization built-in (to_dict/from_dict)
  - Type checking catches errors at development time
  - Self-documenting code structure
  
- **Key insight:** "Dataclasses + Enums create self-documenting, type-safe architectures"

#### Lesson 2: Statistical Analysis Requires Domain Knowledge
**Context:** Implementing degradation detection with linear regression
- **Challenge:** Detecting performance degradation in real-time
- **Solution approach:**
  1. Use Python's `statistics` module (built-in, tested, reliable)
  2. Simple linear regression for trend analysis
  3. Sliding window approach (deque with maxlen)
  4. Configurable thresholds (lookback_window=20, threshold=20%)
  
- **Why simple approach won:**
  - Linear regression is interpretable (operators understand slope)
  - Built-in statistics module is battle-tested
  - Configurable parameters allow tuning per strategy
  - No external ML dependencies needed
  
- **What to avoid:**
  - Overly complex ML models for simple trend detection
  - Black-box algorithms operators can't understand
  - External dependencies when built-ins suffice
  
- **Key insight:** "Use simplest statistical method that solves the problem - interpretability matters"

#### Lesson 3: TDD Accelerates Complex Feature Development
**Context:** Each Sprint 3 task started with comprehensive test file first
- **Pattern that worked:**
  1. Write comprehensive test file (~500-600 lines, 30-35 tests)
  2. Run tests (they all fail, as expected)
  3. Implement features to make tests pass
  4. All tests pass on first full run!
  
- **Sprint 3 Results:**
  - Task 1 (Config Hot-Reload): 35/35 tests passing immediately
  - Task 2 (Multi-Strategy Orchestration): 35/35 tests passing immediately
  - Task 3 (Strategy Comparison): 34/34 tests passing immediately
  - Task 4 (Performance Attribution): 35/35 tests passing immediately
  - Task 5 (Health Monitoring): 35/35 tests passing immediately
  
- **Why TDD worked so well:**
  - Tests define exact requirements
  - No ambiguity about "done"
  - Implementation is guided by tests
  - Refactoring is safe (tests catch breaks)
  - Zero debugging time wasted on "what's wrong?"
  
- **Time savings:**
  - Without TDD: Implement → Debug → Fix → Test → Debug → Fix (iterative, slow)
  - With TDD: Test → Implement → Done (linear, fast)
  - Sprint 3: 174 tests in 3 tasks, minimal debugging
  
- **Key insight:** "TDD isn't slower - it eliminates debugging time and makes 'done' unambiguous"

#### Lesson 4: Comprehensive Fixtures Enable Fast Testing
**Context:** Creating realistic test data for attribution and comparison
- **Pattern discovered:**
  ```python
  @pytest.fixture
  def sample_trades():
      """Create realistic trade data once, use everywhere"""
      return [
          TradeAttribution(symbol="AAPL", pnl=150, entry=100, exit=115, ...),
          TradeAttribution(symbol="MSFT", pnl=-50, entry=200, exit=195, ...),
          # 10+ realistic trades covering various scenarios
      ]
  ```
  
- **Benefits:**
  - Tests are concise (just use fixture)
  - Realistic data = realistic test scenarios
  - Consistent data across tests = reproducible
  - Easy to add edge cases (add to fixture)
  
- **What we avoided:**
  - Duplicating test data in every test
  - Unrealistic data (all trades winning, perfect patterns)
  - Inconsistent data across tests
  
- **Key insight:** "Comprehensive fixtures with realistic data make tests faster to write and more reliable"

#### Lesson 5: Format Reports for Human Consumption
**Context:** Building comparison dashboard and attribution reports
- **What made reports valuable:**
  1. **Visual elements:** Sparklines (▁▂▃▄▅▆▇█) show trends instantly
  2. **Color coding:** Green/yellow/red = instant status assessment
  3. **Rankings:** 🥇🥈🥉 make comparisons clear
  4. **Bar charts:** ASCII bars (████░░░░) show proportions visually
  5. **Formatted tables:** Aligned columns with proper borders
  
- **Example that worked:**
  ```
  ╔══════════════════════════════════════════╗
  ║ Performance Attribution Report           ║
  ║ Momentum_Agg   $3,200  42.7%  ████████  ║
  ║ MA_Cross_Cons  $2,500  33.3%  ██████    ║
  ╚══════════════════════════════════════════╝
  ```
  
- **Why formatting matters:**
  - Operators scan reports quickly
  - Visual patterns recognized faster than numbers
  - Well-formatted reports get read, ugly ones get ignored
  - Terminal-based works everywhere (no UI needed)
  
- **Key insight:** "Reports are for humans - format them for human consumption with visual elements"

#### Lesson 6: Edge Case Testing Prevents Production Bugs
**Context:** Testing with empty data, single items, boundary conditions
- **Edge cases we tested every time:**
  1. Empty collections (no trades, no metrics)
  2. Single item collections (1 trade, 1 strategy)
  3. Boundary values (exactly 0, exactly at threshold)
  4. None/missing data (optional fields)
  5. Invalid inputs (negative values, bad types)
  
- **Bugs caught by edge case testing:**
  - Division by zero when no trades
  - Index errors with single strategy
  - Health score calculation with missing metrics
  - Comparison sorting with equal values
  
- **Time to find these in production:** Hours or days  
  **Time to find in tests:** Seconds
  
- **Pattern that worked:**
  ```python
  # For every feature test, add edge cases:
  def test_feature_with_empty_data():
      result = feature([])
      assert result == expected_for_empty
      
  def test_feature_with_single_item():
      result = feature([single_item])
      assert result == expected_for_one
  ```
  
- **Key insight:** "Edge case tests catch production bugs before they happen - write them every time"

#### Lesson 7: Integration Tests Validate Complete Workflows
**Context:** Testing full workflow from data → orchestrator → attribution → monitoring
- **Integration test pattern:**
  ```python
  def test_complete_workflow():
      # 1. Set up multi-strategy orchestrator
      orchestrator = StrategyOrchestrator(...)
      orchestrator.register_strategy(strategy1, allocation=0.5)
      orchestrator.register_strategy(strategy2, allocation=0.5)
      
      # 2. Feed market data
      orchestrator.on_market_data(market_data)
      
      # 3. Generate signals
      signals = orchestrator.collect_signals()
      
      # 4. Execute trades
      orchestrator.execute_signals(signals)
      
      # 5. Calculate attribution
      attribution = orchestrator.get_attribution()
      
      # 6. Check health
      health = orchestrator.get_health_status()
      
      # 7. Verify complete chain worked
      assert all components worked correctly
  ```
  
- **What integration tests caught:**
  - Signal conflicts between strategies
  - Attribution tracking gaps
  - Health metrics not updating
  - Dashboard rendering errors
  
- **Coverage strategy:**
  - Unit tests: Individual components (80% of tests)
  - Integration tests: Complete workflows (20% of tests)
  - System tests: End-to-end scenarios (manual initially)
  
- **Key insight:** "Integration tests catch interaction bugs that unit tests miss - include them in every sprint"

#### Lesson 8: Commit Frequently with Descriptive Messages
**Context:** Sprint 3 had 5 commits (one per task)
- **Commit pattern that worked:**
  ```
  Task N: Feature Name (X tests passing)
  
  Implementation: Key classes and features
  Tests: Coverage breakdown
  Results: Test counts and status
  ```
  
- **Benefits:**
  - Clear project history
  - Easy to find when features were added
  - Test counts show progress
  - Can bisect bugs if needed
  - Documentation in git log
  
- **What we avoided:**
  - Huge commits with multiple features
  - Vague messages ("updates", "fixes")
  - Committing broken code
  
- **Sprint 3 commit history:**
  - Task 1: c9b5cbb (35 tests)
  - Task 2: c9b5cbb (35 tests, same commit)
  - Task 3: 2e009ab (34 tests)
  - Task 4: b24c48f (35 tests)
  - Task 5: 0f45bc7 (35 tests)
  
- **Key insight:** "Commit after each complete task with descriptive message and test count"

#### Lesson 9: Maintain 100% Pass Rate Through Every Change
**Context:** 532 tests maintained throughout Sprint 3
- **Verification discipline:**
  - Before Task 1: 393/393 tests passing
  - After Task 1: 428/428 tests passing ✓
  - After Task 2: 463/463 tests passing ✓
  - After Task 3: 497/497 tests passing ✓
  - After Task 4: 532/532 tests passing ✓
  - After Task 5: 532/532 tests passing ✓
  
- **What this prevented:**
  - Regressions in existing features
  - Integration breakage
  - Accumulated technical debt
  - "Fix it later" mentality
  
- **Time cost:** 30-60 seconds per verification  
  **Time saved:** Hours of debugging regressions
  
- **Key insight:** "Full test suite verification after every change is the cheapest insurance against regressions"

#### Lesson 10: Deque for Efficient Sliding Windows
**Context:** Health monitoring needs sliding window of recent metrics
- **Problem:** Need last N metrics, update frequently
- **Bad solution:** List with append + slice
  ```python
  self.metrics.append(new_metric)
  self.metrics = self.metrics[-window_size:]  # Creates new list every time!
  ```
  
- **Good solution:** Deque with maxlen
  ```python
  from collections import deque
  self.metrics = deque(maxlen=window_size)  # Auto-evicts oldest
  self.metrics.append(new_metric)  # O(1), no slicing needed
  ```
  
- **Performance difference:**
  - List approach: O(n) for every append (creates new list)
  - Deque approach: O(1) for every append (just updates pointers)
  - With 1000 appends of 100-item window: 100x faster with deque
  
- **Other deque benefits:**
  - Thread-safe operations
  - Memory efficient (no temporary lists)
  - Cleaner code (maxlen handles eviction)
  
- **Key insight:** "Use deque with maxlen for sliding windows - it's built for this use case"

### Sprint 2 Lessons Learned (November 22-24, 2025)

#### Lesson 1: Multi-Dimensional Validation is Essential
**Context:** Implementing paper trading validation criteria
- **What we learned:**
  - Single metrics (Sharpe ratio alone) are insufficient
  - Need validation across 7+ dimensions: time, volume, risk-adjusted returns, drawdown, win rate, profit factor, consecutive losses
  - A strategy with high Sharpe but only 5 trades is not validated
  - Time-based validation (30+ days) is non-negotiable
  
- **Why this matters:**
  - Multiple criteria provide confidence in strategy robustness
  - Prevents overfitting and false positives
  - Ensures statistical significance
  
- **Key insight:** "Validate strategies across all dimensions, not just one metric"

#### Lesson 2: Two-Layer Risk Management
**Context:** Real-time risk monitoring integration
- **The right way:**
  1. **Pre-trade checks (blocking):** Validate before order submission
     - Check position size limits
     - Check portfolio heat
     - Check concentration risk
     - REJECT order if any limit would be exceeded
  
  2. **Post-trade monitoring (continuous):** Recalculate after fills
     - Update portfolio metrics after every fill
     - Detect accumulated risk from multiple fills
     - Alert or pause if thresholds crossed
     
- **Why both layers matter:**
  - Pre-trade stops problems before they start
  - Post-trade catches issues from multiple fills accumulating
  - Two-layer defense prevents risk limit violations
  
- **Key insight:** "Both layers are necessary for robust risk management"

#### Lesson 3: Database Persistence for Validation
**Context:** Tracking metrics for 30+ day validation period
- **Why database over in-memory:**
  - In-memory tracking loses data on restart
  - Validation requires persistent time-series data
  - Historical performance matters for approval decisions
  - Audit trail must survive system restarts
  
- **Schema design that worked:**
  ```sql
  -- Separate tables for different data types
  strategy_metrics        -- Current aggregated metrics
  strategy_snapshots      -- Daily time-series data
  strategy_trades         -- Individual trade records
  approval_history        -- Complete audit trail
  ```
  
- **Why this design:**
  - Fast queries for current status
  - Historical analysis from snapshots
  - Trade-by-trade review when needed
  - Complete audit trail for accountability
  
- **Key insight:** "Critical validation data must persist in database"

#### Lesson 4: Visual Feedback Increases Operator Confidence
**Context:** Building validation dashboard with sparklines and progress bars
- **What worked well:**
  - Progress bars for percentage completion (clear visual of status)
  - Sparklines for time-series trends (instant pattern recognition)
  - Color coding green/yellow/red (quick status assessment)
  - Composite health scores 0-100 (complex metrics simplified)
  - Action-required notifications (next steps obvious)
  
- **User feedback:**
  - "I can see exactly where the strategy stands"
  - "Progress bars make it clear what's needed"
  - "Sparklines give instant visual feedback"
  
- **Why it matters:**
  - Operators need confidence before approving live trading
  - Visual feedback reduces cognitive load
  - Clear next-action guidance prevents mistakes
  
- **Key insight:** "Visual dashboard features increase operator confidence and reduce errors"

#### Lesson 5: Multi-Gate Approval Prevents "Skip to Live"
**Context:** Implementing strategy promotion workflow
- **Human nature problem:** "This strategy looks good, let's go live now!"
- **Solution: Multi-gate approval**
  ```
  PAPER → (auto validation) → VALIDATED → (manual approval) → LIVE_APPROVED → (final confirm) → LIVE_ACTIVE
  Gate 1: Automated          Gate 2: Manual            Gate 3: Final
  ```
  
- **Why multiple gates:**
  - Prevents emotional decision-making
  - Forces thorough review of all criteria
  - Creates audit trail for accountability
  - Provides multiple "think about it" moments
  - No direct path from PAPER to LIVE possible
  
- **Checklist enforcement:**
  - Strategy code reviewed
  - Risk parameters verified
  - Position sizing confirmed
  - Emergency procedures tested
  - Monitoring alerts configured
  - All items must be checked before approval
  
- **Key insight:** "Never allow direct path from paper to live - require human approval with checklist"

#### Lesson 6: Incremental Testing Catches Issues Early
**Context:** Building complex metrics calculations (Sharpe ratio, drawdown tracking)
- **Sprint 2 approach that worked:**
  ```
  Task 2: Metrics Tracker
  1. Wrote database schema → tested persistence ✓
  2. Wrote trade recording → tested calculation ✓
  3. Wrote Sharpe ratio → tested with known data ✓
  4. Wrote drawdown tracking → tested edge cases ✓
  5. Integrated all → tests already passing! ✓
  ```
  
- **Why this approach worked:**
  - Early bug detection = easy fixes
  - Confident integration when components tested
  - Faster overall development (less debugging time)
  - Each test provides safety net for next component
  
- **What to avoid:**
  - Writing 500 lines before testing = debugging nightmare
  - "I'll test it all at the end" = technical debt accumulation
  
- **Key insight:** "Test components in isolation first, integration should 'just work'"

#### Lesson 7: Test Quality > Test Quantity
**Context:** 393 tests with 37% overall coverage
- **What we learned:**
  - High test count doesn't guarantee quality
  - Coverage should match criticality of code
  - Risk-critical code needs near-100% coverage
  - Integration code can rely on manual testing initially
  
- **Sprint 2 coverage where it matters:**
  - strategy/lifecycle.py: 97% ✅
  - strategy/metrics_tracker.py: 99% ✅
  - strategy/validation.py: 96% ✅
  - execution/risk_monitor.py: 95% ✅
  
- **Intentionally lower coverage:**
  - tws_client.py: 0% (TWS integration, manual testing initially)
  - Legacy scripts: 0% (will be replaced)
  
- **Key insight:** "Focus on edge cases and error conditions in critical code, not just happy path coverage"

#### Lesson 8: Velocity Compounds with Strong Foundation
**Context:** Sprint 2 velocity 64% faster than Sprint 1
- **Why velocity increased:**
  - Foundation from Sprint 1 enabled faster Sprint 2 work
  - Established patterns reduced decision time
  - Clear requirements from Sprint plan
  - Test-first approach prevented rework
  - Team learning curve effect
  
- **Metrics:**
  - Sprint 1: 33 tests/day
  - Sprint 2: 54 tests/day
  - Increase: 64%
  
- **Key insight:** "Invest in strong foundation early, velocity compounds over time"

#### Lesson 9: Zero Tolerance for Warnings (November 24, 2025)
**Context:** SQLAlchemy deprecation warning discovered in test output
- **The Warning:**
  ```
  MovedIn20Warning: The declarative_base() function is now available as
  sqlalchemy.orm.declarative_base(). (deprecated since: 2.0)
  ```

- **Why it matters:**
  - Deprecation warnings become breaking errors in future versions
  - "Just warnings" create technical debt that compounds
  - Warnings mask real issues in test output noise
  - Future-proofing requires addressing deprecations immediately
  
- **The Fix:**
  - Changed: `from sqlalchemy.ext.declarative import declarative_base`
  - To: `from sqlalchemy.orm import declarative_base, relationship`
  - Result: Zero warnings, SQLAlchemy 2.0+ compliant
  
- **Policy Established:**
  - All warnings must be investigated immediately when they appear
  - Zero warnings required before committing (not just zero failures)
  - Warnings are treated with same urgency as test failures
  - Document warning count in commit messages (e.g., "393 passed, 0 warnings")
  
- **Why zero tolerance:**
  - Clean test output makes real issues immediately visible
  - Prevents "warning fatigue" where important warnings get ignored
  - Keeps codebase modern and maintainable
  - Eliminates future breaking changes proactively
  
- **Key insight:** "Warnings are errors waiting to happen - fix them immediately, don't defer to later"

### Week 4 Day 3 Lessons Learned

#### Error 1: TimeFrame.DAILY
- **What happened:** Used `TimeFrame.DAILY` without checking enum
- **Why:** Assumed naming convention
- **Fix:** Check enum source, use `TimeFrame.DAY_1`
- **Lesson:** Never assume enum values - always verify

#### Error 2: on_bar(symbol, bar)
- **What happened:** Wrong method signature for Strategy.on_bar()
- **Why:** Didn't check base class abstract method
- **Fix:** Use correct signature `on_bar(market_data)`
- **Lesson:** Always check base class/interface signatures

#### Error 3: market_data.symbol
- **What happened:** Accessed `.symbol` (singular) instead of `.symbols` (plural)
- **Why:** Assumed single symbol, didn't inspect class
- **Fix:** Use `.symbols` list and iterate
- **Lesson:** Inspect data structure before accessing attributes

#### Error 4: buy(symbol, qty, "reason")
- **What happened:** Passed string as third parameter instead of OrderType enum
- **Why:** Misread method signature
- **Fix:** Use `OrderType.MARKET` enum
- **Lesson:** Check parameter types, not just parameter names

#### Error 5: position <= 0
- **What happened:** Compared None to int, causing TypeError
- **Why:** Didn't handle None return value
- **Fix:** Use `position or 0` to default None to 0
- **Lesson:** Check if methods return Optional types

---

## 📝 Quick Reference Template

Add this comment block at the top of new files:

```python
"""
Quick Reference for this module:

Key Classes:
- ClassName: .attribute1 (type), .method1(params) -> return_type

Important Enums:
- EnumName: VALUE_1, VALUE_2, VALUE_3

Common Patterns:
- Pattern 1: description
- Pattern 2: description

Gotchas:
- Thing that returns None - handle it!
- Attribute is plural (.symbols not .symbol)
- Use EnumType not string
"""
```

---

## 🚀 Integration Best Practices

### When Integrating New Code

1. **Start Simple**
   - Minimal working example first
   - One feature at a time
   - Verify each step

2. **Incremental Integration**
   ```python
   # ✅ Add one component at a time
   # Step 1: Add data loading only
   # Step 2: Add strategy initialization
   # Step 3: Add execution logic
   # Step 4: Add analytics
   ```

3. **Verify At Each Step**
   - Run tests after each addition
   - Check for warnings (they matter!)
   - Fix issues before proceeding

4. **Don't Skip Testing**
   - "I'll test it later" = technical debt
   - Test as you go
   - Automated tests > manual testing

---

## 🎯 Success Metrics

### Before Considering Code "Done"

- [ ] All unit tests passing (100%) - baseline verified with 0 warnings
- [ ] All unit tests passing (100%) - after changes verified with 0 warnings
- [ ] Zero compiler/linter warnings (all investigated and fixed)
- [ ] Zero test warnings (deprecations, type issues, etc. all resolved)
- [ ] No None-type errors
- [ ] All edge cases handled
- [ ] Code reviewed (by peer or self)
- [ ] Documentation complete
- [ ] Integration tested
- [ ] Performance acceptable
- [ ] Git commit with clear message including warning count

### Definition of "Done"

Code is only done when:
1. ✅ Baseline tests verified (before changes): X passed, 0 warnings
2. ✅ Changes implemented
3. ✅ Tests pass after changes (verify again): X passed, 0 warnings
4. ✅ Zero warnings (all warnings investigated and resolved)
5. ✅ Error handling complete
6. ✅ Integrated and verified
7. ✅ Documented
8. ✅ Committed with test count AND warning count (e.g., "393 passed, 0 warnings")

### Deletion Protocol (Additional Requirements)

When deleting code:
1. ✅ Verify baseline tests pass
2. ✅ Search for all usages: `grep_search(query="module_name", isRegexp=True)`
3. ✅ Analyze impact (is it in test suite? imported elsewhere?)
4. ✅ Delete in logical groups (related files together)
5. ✅ Verify tests still pass after deletion
6. ✅ Commit with detailed message explaining what and why
7. ✅ Reference commit hash in documentation if significant
8. ✅ Never force-push (preserve git history)

---

## 🤝 Team Expectations

### For All Team Members (Human & AI)

1. **Read this document before starting work**
2. **Follow the checklist - every time**
3. **Ask questions if unsure - don't guess**
4. **Test incrementally - don't batch**
5. **Fix errors immediately - don't defer**
6. **Document learnings - update this file**

### For AI Agents Specifically

1. **Always use verification tools before implementing**
   - `grep_search`, `read_file`, `list_code_usages`
2. **Never assume - always verify**
3. **Test each component before moving on**
4. **Read error messages completely and act on them**
5. **Keep implementations simple initially**
6. **Ask for clarification if requirements are ambiguous**

### For Code Reviewers

1. **Check that verification was done**
   - Were existing patterns checked?
   - Were types validated?
2. **Look for None handling**
3. **Verify test coverage**
4. **Ensure error handling present**
5. **Confirm documentation exists**

---

## 📚 Resources

### Internal Documentation
- `README.md` - Project overview
- `backtest/README.md` - Backtesting framework docs
- `risk/README.md` - Risk management docs
- Test files - Living examples of correct usage

### When in Doubt
1. Check existing tests - they show correct usage
2. Use `grep_search` to find patterns
3. Read the source code - it's the truth
4. Ask the team - collaboration over guessing

---

## 🔄 Document Maintenance

### When to Update This Document

- After encountering a new type of error
- When establishing a new pattern
- After team retrospectives
- When tooling changes
- Quarterly review minimum

### How to Update

1. Add specific examples
2. Keep it practical, not theoretical
3. Include code snippets
4. Update "Lessons Learned" section
5. Date the changes

---

## 💡 Remember

> **"100% test pass rate with zero warnings is not a goal - it's a requirement."**

> **"Warnings are errors waiting to happen - fix them immediately."**

> **"The best code is code that works correctly the first time because you took the time to verify before implementing."**

> **"Tests are not overhead - they're proof your code works."**

> **"Defensive programming isn't paranoia - it's professionalism."**

> **"Delete with confidence when you verify with discipline."**

> **"One authoritative implementation per concept - no duplicates."**

> **"Zero tolerance for warnings = zero technical debt from ignored issues."**

---

### Project Metrics (Sprint 4 Task 1 Complete)

### Current Test Suite Status
- **Total Tests:** 562
- **Pass Rate:** 100%
- **Warning Count:** 0
- **Last Verified:** 2025-11-25
- **Test Coverage:** 45% overall, 95%+ in risk-critical modules
- **Recent Additions:**
  - Sprint 4 Task 1: 30 tests added (Nov 25) - E2E integration tests, full workflow validation
  - Sprint 3: 174 tests added (Nov 24-25) - Config, orchestration, comparison, attribution, health monitoring
  - Sprint 2: 162 tests added (Nov 22-24) - Risk, validation, metrics, promotion, dashboard
  - Sprint 1: 132 tests added (Nov 20-22) - Lifecycle, paper trading, data pipeline, monitoring, integration
  - Week 4: 64 baseline tests - Backtest engine, data, profiles, strategy templates
  
### Test Files by Sprint
**Sprint 4 Task 1 (Day 11):**
  - test_integration_e2e.py (30 tests) - Complete E2E integration validation

**Sprint 3 (Days 8-10):**
  - test_config_hot_reload.py (35 tests)
  - test_multi_strategy_orchestration.py (35 tests)
  - test_strategy_comparison.py (34 tests)
  - test_performance_attribution.py (35 tests)
  - test_health_monitor.py (35 tests)

**Sprint 2 (Days 5-7):**
  - test_risk_monitor.py (28 tests)
  - test_metrics_tracker.py (34 tests)
  - test_validation.py (35 tests)
  - test_promotion.py (27 tests)
  - test_validation_monitor.py (38 tests)

**Sprint 1 (Days 1-4):**
  - test_strategy_lifecycle.py (29 tests)
  - test_paper_adapter.py (32 tests)
  - test_realtime_pipeline.py (17 tests)
  - test_paper_monitor.py (28 tests)
  - test_paper_trading_integration.py (26 tests)

**Week 4 Baseline:**
  - test_backtest_engine.py (18 tests)
  - test_backtest_data.py (18 tests)
  - test_profiles.py (36 tests)
  - test_profile_comparison.py (20 tests)
  - test_strategy_templates.py (46 tests)

### Sprint Completion History
- **Sprint 4 Task 1 (2025-11-25):** E2E Integration Testing
  - 30 tests added (comprehensive integration validation)
  - ~916 lines of test code
  - 100% test pass rate achieved (532→562)
  - Zero warnings maintained
  - Key lesson: API research before integration tests (documented in Lesson 1)
  - Test progression: 0%→50%→70%→90%→97%→100% (5 correction rounds)
  - 37+ API mismatches corrected systematically
  - Commit: 11da7f4
  - Key achievements: Complete lifecycle validation, multi-component coordination, error handling, data pipeline integrity

- **Sprint 3 (2025-11-24 to 2025-11-25):** Strategy Development Pipeline
  - 174 tests added (35+35+34+35+35)
  - ~4,200 lines of code (2,600 production + 1,600 tests)
  - 100% test pass rate maintained (393→428→463→497→532)
  - Zero warnings maintained throughout
  - Velocity: 87 tests/day (61% increase over Sprint 2)
  - TDD approach: All tests passing immediately after implementation
  - All 5 tasks completed on schedule
  - Key achievements: Config hot-reload, multi-strategy orchestration, comparison dashboard, attribution system, health monitoring

- **Sprint 2 (2025-11-22 to 2025-11-24):** Risk & Validation Framework
  - 162 tests added (28+34+35+27+38)
  - 3,667 lines of code (2,310 production + 1,357 tests)
  - 100% test pass rate maintained (231→293→328→357→393)
  - Zero warnings maintained throughout (fixed SQLAlchemy deprecation on Day 7)
  - Velocity: 54 tests/day (64% increase over Sprint 1)
  - All 5 tasks completed on schedule
  
- **Sprint 1 (2025-11-20 to 2025-11-22):** Paper Trading Foundation
  - 132 tests added (29+32+17+28+26)
  - 100% test pass rate maintained (138→231)
  - Velocity: 33 tests/day
  - All 5 tasks completed on schedule

### Cleanup History
- **2025-11-22:** Deleted legacy backtesting module (8 files, 3,070 lines)
  - Commit da9a714: Archive step (backtesting → backtesting_old)
  - Commit 8206109: Deletion step (removed backtesting_old/)
  - Tests maintained: 138/138 (100%) throughout all deletions

### Code Quality Standards Achieved
✅ Single authoritative backtest module (backtest/)  
✅ No duplicate implementations  
✅ 100% test pass rate maintained (532/532)  
✅ Zero warnings maintained (all warnings investigated and resolved)  
✅ Clear git history with detailed commit messages  
✅ Zero breaking changes to production code  
✅ High coverage in risk-critical modules (95%+)  
✅ Comprehensive validation framework operational  
✅ Multi-gate approval workflow enforced  
✅ Complete audit trail for strategy promotion  
✅ Multi-strategy orchestration system with attribution tracking  
✅ Real-time health monitoring with statistical degradation detection  
✅ Dynamic configuration hot-reload without restarts  
✅ TDD approach with comprehensive test-first development

---

**Revision History:**
- 2025-12-01 (v5): **Major Restructure for Clarity** - Added Quick Start Checklist at top, Enhanced Core Principles with better formatting and visual tables, Added Research Decision Tree for determining appropriate protocol, Improved Integration Test Research Protocol with detailed time breakdowns, Added Quick Reference Cards section with ASCII art for quick scanning, Better organized defensive programming patterns with specific examples, Improved overall document navigation and readability
- 2025-11-25 (v4): **Sprint 4 Task 1 Lesson** - Added critical "Research APIs Before Integration Tests" lesson from Sprint 4 Task 1 experience (37+ API mismatches), Enhanced Pre-Implementation Checklist with Integration Test Research Protocol (mandatory 6-step process), Updated Development Workflow Phase 1 to 15-30% with integration test research requirements, Documented time savings (44% reduction) from proper API research, Current metrics: 562 tests (30 new integration tests)
- 2025-11-24 (v3): **Sprint 3 Complete** - Added 10 Sprint 3 lessons (dataclasses+enums, statistical analysis, TDD acceleration, comprehensive fixtures, human-readable reports, edge case testing, integration workflows, commit discipline, 100% pass rate, deque for sliding windows), Updated metrics to 532 tests, Documented 87 tests/day velocity (61% increase)
- 2025-11-24 (v2): **Zero Warnings Policy** - Updated Principle 0 to require zero warnings (not just zero failures), Added warning investigation requirement to all checklists and protocols, Fixed SQLAlchemy deprecation warning (declarative_base import), Documented warning resolution in Sprint 2 history
- 2025-11-24 (v1): Added Sprint 2 lessons (multi-dimensional validation, two-layer risk management, database persistence, visual feedback, multi-gate approval, incremental testing, test quality focus, velocity compounding), Updated project metrics to 393 tests
- 2025-11-22: Added Prime Directive Principle 0 (100% Test Pass Rate), Week 4 Day 4 lessons, Deletion Protocol, Project Metrics
- 2025-11-21: Initial creation based on Week 4 Day 3 lessons learned

**Next Review:** After Sprint 5 completion (estimated 2025-12-08)

---

## 🎓 Key Takeaways

### The Three Pillars of Code Quality

```
1. VERIFY BEFORE CODE
   \u251c\u2500 Research existing APIs thoroughly
   \u251c\u2500 Document findings before implementing
   \u2514\u2500 30 min research saves 2 hours debugging

2. TEST INCREMENTALLY
   \u251c\u2500 Test after every ~50 lines
   \u251c\u2500 Component tests before integration
   \u2514\u2500 75% less debugging time

3. ZERO TOLERANCE QUALITY GATE
   \u251c\u2500 100% test pass rate (before & after)
   \u251c\u2500 Zero warnings (investigate all)
   \u2514\u2500 Only commit when both criteria met
```

### Most Important Rules (Top 5)

1. **Run tests before AND after changes** - Establish baseline, verify changes
2. **Research APIs before integration tests** - Read existing tests, document APIs
3. **Handle None defensively** - Use `or default`, check explicitly, fail fast
4. **Test incrementally** - 50 lines → test → fix, never batch
5. **Fix warnings immediately** - Zero tolerance, warnings become errors

### Success Metrics

**Code is production-ready when:**
- ✅ All tests passing (100%)
- ✅ Zero warnings
- ✅ Defensive None handling present
- ✅ API research documented
- ✅ Incremental testing verified
- ✅ Code reviewed
- ✅ Commit message includes test count + warning status

---

## 📞 Getting Help

### When You're Stuck

**Before asking for help:**
1. Read relevant section of this document
2. Search for similar code patterns in codebase
3. Check existing tests for usage examples
4. Review recent commit messages for context

**When asking for help:**
- Share what you've already tried
- Include error messages (complete, not summarized)
- Reference this document section you're following
- Provide minimal reproducible example

**Resources:**
- This document: Core principles and protocols
- Existing tests: API usage examples
- Git history: Context on why decisions were made
- Code reviews: Learning opportunities

---

*This is a living document. Update it as we learn. Share it with the team. Follow it every time.*

**Document Maintainer:** Development Team  
**Last Major Review:** December 1, 2025  
**Review Frequency:** After each sprint or major lesson learned
