# Story 1.30: Hot Path Performance Profiling

**Status:** Draft
**Priority:** High
**Depends on:** None

---

## Context / Background

Fapilog is currently ~27x slower than stdlib logging for fast sinks (3,295 vs 90,393 logs/sec) with per-call latency of 279 μs vs 24 μs. This is acceptable for slow sink scenarios where fapilog's async architecture provides value, but limits adoption for general use cases.

Initial analysis suggests `build_envelope()` accounts for ~250 μs (90%) of the hot path, but the component breakdown only accounts for ~13 μs of measurable operations:
- Timestamp generation: ~10 μs
- Context extraction: ~2 μs
- Dict construction: ~1 μs

**The remaining ~237 μs is unexplained.** Before implementing optimizations like lazy envelope building (Story 1.31), we need to understand where the overhead actually comes from.

Relevant code paths:
- `src/fapilog/core/logger.py:704-828` - `_enqueue()` entry point
- `src/fapilog/core/logger.py:260-356` - `_prepare_payload()`
- `src/fapilog/core/envelope.py:23-145` - `build_envelope()`
- `src/fapilog/core/errors.py:517-572` - `serialize_exception()`

---

## Scope (In / Out)

### In Scope

- Profile the complete hot path from `logger.info()` to queue insertion
- Identify where the ~250 μs overhead actually originates
- Measure the theoretical floor for minimal capture and enqueue
- Measure dict copy costs for defensive copying scenarios
- Measure exception serialization costs separately
- Document findings to inform optimization decisions
- Create a reproducible profiling script

### Out of Scope

- Implementing any optimizations (deferred to Story 1.31+)
- Profiling worker/sink performance (separate concern)
- Profiling enrichers/filters/redactors (already known to be conditional)
- Memory profiling (covered by existing benchmarks)

---

## Acceptance Criteria

### AC1: Hot Path Flame Graph

**Description:** Generate a flame graph or equivalent visualization showing where time is spent in the hot path.

**Validation:**
```bash
# Should produce a profile showing function-level breakdown
python scripts/profile_hot_path.py --output profile_results.json
# Visualization available (flamegraph, snakeviz, or similar)
```

### AC2: Component Timing Breakdown

**Description:** Document precise timing for each component of the hot path with statistical confidence.

**Validation:**
```
Expected output format:
┌─────────────────────────────────────┬──────────┬─────────┐
│ Component                           │ Avg (μs) │ % Total │
├─────────────────────────────────────┼──────────┼─────────┤
│ _enqueue() entry + level gating     │     X.XX │    X.X% │
│ _prepare_payload()                  │     X.XX │    X.X% │
│   ├─ ContextVar lookups             │     X.XX │    X.X% │
│   ├─ UUID generation                │     X.XX │    X.X% │
│   └─ Dict operations                │     X.XX │    X.X% │
│ build_envelope()                    │     X.XX │    X.X% │
│   ├─ Timestamp generation           │     X.XX │    X.X% │
│   ├─ Context field extraction       │     X.XX │    X.X% │
│   ├─ Diagnostics dict               │     X.XX │    X.X% │
│   └─ Final dict construction        │     X.XX │    X.X% │
│ Queue enqueue                       │     X.XX │    X.X% │
│ Thread coordination                 │     X.XX │    X.X% │
├─────────────────────────────────────┼──────────┼─────────┤
│ TOTAL                               │   279.XX │  100.0% │
│ UNEXPLAINED                         │     X.XX │    X.X% │
└─────────────────────────────────────┴──────────┴─────────┘
```

### AC3: Minimal Capture Floor Measurement

**Description:** Measure the theoretical minimum latency for capturing essential data and enqueueing without building a full envelope.

**Validation:**
```python
def minimal_capture(level, message, metadata):
    """Theoretical floor for lazy envelope approach."""
    raw = (
        level,
        message,
        metadata,
        time.time(),
        bound_context_var.get(),
    )
    queue.append(raw)

# Benchmark shows: X.XX μs average
# This establishes the floor for lazy envelope optimization
```

### AC4: Dict Copy Cost Analysis

**Description:** Measure the cost of shallow vs deep copying for typical metadata payloads.

**Validation:**
```python
# Typical payload (~256 bytes)
payload = {"user": "u123", "action": "test", "nested": {"a": 1, "b": 2}}

# Results:
# - No copy (reference): X.XX μs
# - Shallow copy:        X.XX μs
# - Deep copy:           X.XX μs
# - dict() constructor:  X.XX μs
```

### AC5: Exception Serialization Isolation

**Description:** Measure exception serialization cost separately to understand impact on error logging paths.

**Validation:**
```python
# With varying stack depths (10, 25, 50 frames):
# - traceback.format_exception(): X.XX μs
# - traceback.extract_tb():       X.XX μs
# - Full serialize_exception():   X.XX μs
```

### AC6: Profiling Script Added

**Description:** A reusable profiling script is added to `scripts/` for future performance analysis.

**Validation:**
```bash
python scripts/profile_hot_path.py --help
# Shows usage for running profiling with various options
```

### AC7: Findings Documented

**Description:** Results are documented with clear conclusions about optimization opportunities.

**Validation:**
- `docs/architecture/hot-path-profile-results.md` exists
- Documents where the ~250 μs actually comes from
- Provides recommendations for Story 1.31 (lazy envelope)
- Includes confidence levels for projected improvements

---

## Implementation Notes

### Profiling Approaches to Use

1. **cProfile** - Function-level timing
   ```bash
   python -m cProfile -s cumtime scripts/benchmarking.py
   ```

2. **line_profiler** - Line-by-line timing for hot functions
   ```python
   @profile
   def build_envelope(...):
       ...
   ```

3. **perf_counter instrumentation** - Manual timing points
   ```python
   t0 = time.perf_counter()
   # operation
   t1 = time.perf_counter()
   timings['operation'] = (t1 - t0) * 1e6
   ```

4. **py-spy** - Sampling profiler for flame graphs
   ```bash
   py-spy record -o profile.svg -- python scripts/benchmarking.py
   ```

### File Changes

```
scripts/profile_hot_path.py (NEW - profiling script)
docs/architecture/hot-path-profile-results.md (NEW - findings)
```

---

## Tasks

### Phase 1: Instrumentation

- [ ] Create `scripts/profile_hot_path.py` with multiple profiling modes
- [ ] Add perf_counter instrumentation points to key functions
- [ ] Set up line_profiler for build_envelope and _prepare_payload

### Phase 2: Measurement

- [ ] Run cProfile on benchmarking script, capture results
- [ ] Run line_profiler on hot path functions
- [ ] Generate flame graph with py-spy
- [ ] Measure minimal capture floor (AC3)
- [ ] Measure dict copy costs (AC4)
- [ ] Measure exception serialization in isolation (AC5)

### Phase 3: Analysis & Documentation

- [ ] Compile component timing breakdown table (AC2)
- [ ] Identify unexplained overhead sources
- [ ] Write findings document with recommendations
- [ ] Update Story 1.31 with profiling-informed estimates

---

## Tests

### Unit Tests

None required - this is an analysis story.

### Validation

- [ ] Profiling script runs without errors
- [ ] Results are reproducible across multiple runs (±10% variance)
- [ ] Timing breakdown accounts for >90% of total measured time

---

## Definition of Done

### Code Complete

- [ ] Profiling script added and working
- [ ] All acceptance criteria measurements completed

### Quality Assurance

- [ ] Script follows project code style
- [ ] `ruff check scripts/profile_hot_path.py` passes
- [ ] Results are reproducible

### Documentation

- [ ] Findings documented in `docs/architecture/`
- [ ] Story 1.31 updated with informed estimates
- [ ] CHANGELOG updated

---

## Risks / Rollback

### Risks

1. **Risk:** Profiling overhead skews measurements
   - **Mitigation:** Use sampling profilers (py-spy) for overall picture, targeted instrumentation for specific components

2. **Risk:** Results are environment-dependent
   - **Mitigation:** Document test environment, run on CI for baseline, note variance

3. **Risk:** Findings show overhead is inherent to Python (not optimizable)
   - **Mitigation:** This is a valid finding - document and adjust Story 1.31 expectations

### Rollback Plan

Analysis story - no rollback needed. If results are inconclusive, document what was learned and refine approach.

---

## Related Stories

- **Enables:** Story 1.31 - Lazy Envelope Building (provides data for go/no-go decision)
- **Related:** Story 1.25 - Eliminate Settings Hot Path Calls (prior optimization)

---

## Change Log

| Date | Change | Author |
|------|--------|--------|
| 2026-01-19 | Initial draft | Claude |
