# SDK Performance Profiling Report - Baseline

**Date:** 2026-02-11
**Test:** 10,000 predictions recorded
**Duration:** 4.301 seconds
**Throughput:** ~2,325 predictions/second

## CPU Hotspots (Top 10 by Cumulative Time)

| Rank | Function | Calls | Cumtime (s) | Percall (ms) | % Total |
|------|----------|-------|-------------|--------------|---------|
| 1 | client.py:record_prediction | 10,000 | 3.258 | 0.326 | 75.8% |
| 2 | logging.info | 10,002 | 2.208 | 0.221 | 51.3% |
| 3 | logging._log | 10,003 | 2.177 | 0.218 | 50.6% |
| 4 | logging.handle | 10,003 | 1.837 | 0.184 | 42.7% |
| 5 | logging.callHandlers | 10,002 | 1.822 | 0.182 | 42.4% |
| 6 | StreamHandler.handle | 20,003 | 1.788 | 0.089 | 41.6% |
| 7 | StreamHandler.emit | 10,001 | 0.957 | 0.096 | 22.3% |
| 8 | client.shutdown | 1 | 0.886 | 886.0 | 20.6% |
| 9 | LoggingHandler.emit | 10,002 | 0.756 | 0.076 | 17.6% |
| 10 | measurement_consumer.consume_measurement | 20,100 | 0.739 | 0.037 | 17.2% |

## CPU Hotspots (Top 10 by Total Time)

| Rank | Function | Calls | Tottime (s) | Percall (ms) | % Total |
|------|----------|-------|-------------|--------------|---------|
| 1 | TextIOWrapper.write | 10,002 | 0.619 | 0.062 | 14.4% |
| 2 | _encode_metric (OTLP) | 3 | 0.191 | 63.7 | 4.4% |
| 3 | BoundedAttributes.__setitem__ | 80,021 | 0.175 | 0.002 | 4.1% |
| 4 | _view_instrument_match.consume_measurement | 20,100 | 0.166 | 0.008 | 3.9% |
| 5 | _encode_key_value (OTLP) | 82,197 | 0.118 | 0.001 | 2.7% |
| 6 | dict.items | 80,316 | 0.117 | 0.001 | 2.7% |
| 7 | ssl._SSLSocket.read | 13 | 0.091 | 7.0 | 2.1% |
| 8 | BoundedAttributes.__init__ | 10,003 | 0.079 | 0.008 | 1.8% |
| 9 | isinstance (builtin) | 519,877 | 0.076 | 0.000 | 1.8% |
| 10 | logging.findCaller | 10,003 | 0.076 | 0.008 | 1.8% |

## Memory Profile

### Peak Memory Usage
- **Peak:** 1.97 MB
- **Current (end):** 0.62 MB
- **Memory growth pattern:** Stable after initial warmup (~1.0 MB peak at 1k predictions)

### Top 10 Memory Allocations

| Rank | Module | Size (KB) | Count | Avg (B) |
|------|--------|-----------|-------|---------|
| 1 | OTLP log_exporter.__init__ | 91.7 | 1 | 91.7K |
| 2 | opentelemetry.attributes | 79.1 | 274 | 296 |
| 3 | opentelemetry._logs._internal | 61.9 | 546 | 116 |
| 4 | opentelemetry.attributes (2) | 38.8 | 551 | 72 |
| 5 | <string> (dynamic) | 28.3 | 558 | 52 |
| 6 | sdk._logs._internal | 27.7 | 546 | 52 |
| 7 | threading.py | 27.1 | 578 | 48 |
| 8 | uuid.py | 22.6 | 272 | 85 |
| 9 | opentelemetry.attributes (3) | 17.2 | 275 | 64 |
| 10 | sdk._logs._internal (2) | 13.0 | 274 | 48 |

## Key Findings

### Critical Bottlenecks (High Impact)

1. **Excessive Logging Overhead (51.3% of total time)**
   - Problem: logging.info() called on EVERY prediction
   - Impact: 2.208s / 4.301s = 51.3% of total time spent in logging
   - Root cause: `client.py:743` logs "Prediction completed" for every prediction
   - Fix: Remove or make logging conditional (debug mode only)
   - **Expected improvement:** ~50% latency reduction

2. **I/O Write Operations (14.4% of total time)**
   - Problem: TextIOWrapper.write() blocks for 0.619s total
   - Impact: 10,002 synchronous writes to stdout
   - Fix: Buffer logs, reduce log frequency, or disable verbose logging
   - **Expected improvement:** ~15% latency reduction

3. **Logging Stack Overhead (42.4% of total time)**
   - Problem: Logging infrastructure (handlers, formatters) add significant overhead
   - Impact: callHandlers, handle, emit operations dominate
   - Fix: Reduce logging verbosity, use lazy logging evaluation
   - **Expected improvement:** Combined with #1, ~50% reduction

### Medium Impact Issues

4. **OTLP Encoding Overhead (4.4% of time)**
   - Problem: _encode_metric() called 3 times, takes 63.7ms per call
   - Impact: Encoding protobuf messages is CPU-intensive
   - Fix: Batch encoding, optimize attribute serialization
   - **Expected improvement:** ~5% latency reduction

5. **Attribute Creation Overhead (4.1% of time)**
   - Problem: 80,021 calls to BoundedAttributes.__setitem__
   - Impact: Creating/modifying attributes on every metric
   - Fix: Reuse attribute dictionaries, cache common attributes
   - **Expected improvement:** ~5% latency reduction

### Low Impact (Optimization Opportunities)

6. **UUID Generation (22.6 KB memory, frequent calls)**
   - Problem: UUID created for each log/metric event
   - Fix: Pre-generate UUIDs in batch or use simpler ID scheme
   - **Expected improvement:** Minor memory reduction

7. **isinstance() Checks (519,877 calls)**
   - Problem: Type checking is frequent (1.8% of time)
   - Fix: Trust types in hot paths, reduce validation in production mode
   - **Expected improvement:** ~2% latency reduction

## Performance vs. Requirements

**Target Requirements:**
- SDK overhead: <50ms p95 (<100ms p99)
- Throughput: 1000+ req/s sustained
- Memory: Flat profile after warmup

**Current Performance:**
- SDK overhead: ~0.326ms per prediction (p95 likely ~0.4ms) ✅ EXCELLENT
- Throughput: ~2,325 predictions/second ✅ EXCEEDS TARGET
- Memory: Flat profile after 1k predictions (0.62 MB final) ✅ EXCELLENT

**Note:** Current performance ALREADY meets requirements, but bottlenecks identified can improve throughput by 50-70% by removing excessive logging.

## Optimization Priority

### Priority 1: Remove Excessive Logging (HIGH IMPACT)
- **Impact:** 50% latency reduction
- **Effort:** Low
- **Risk:** Low
- **Action:** Disable "Prediction completed" logging in production

### Priority 2: Optimize Attribute Creation (MEDIUM IMPACT)
- **Impact:** 5-10% latency reduction
- **Effort:** Medium
- **Risk:** Medium
- **Action:** Cache and reuse attribute dictionaries

### Priority 3: Batch OTLP Encoding (MEDIUM IMPACT)
- **Impact:** 5% latency reduction
- **Effort:** Medium
- **Risk:** Medium
- **Action:** Increase batch sizes, optimize serialization

### Priority 4: Reduce Type Checking in Hot Paths (LOW IMPACT)
- **Impact:** 2% latency reduction
- **Effort:** Low
- **Risk:** Medium (may miss validation errors)
- **Action:** Add strict_validation flag checks in hot paths

## Next Steps

1. ✅ Profiling complete with cProfile
2. ✅ Memory analysis complete with tracemalloc
3. ⏭️ Implement optimizations (Priority 1-3)
4. ⏭️ Create benchmark suite for before/after comparison
5. ⏭️ Document optimizations with code comments
