# Exemplar Support Implementation Plan

Exemplars link metrics to traces by recording sample trace/span IDs with metric data points.

## Background

From the otel-python vs otelmini comparison, otel-python produces exemplars like:
```json
"exemplars": [{
  "timeUnixNano": "1770405086044531000",
  "spanId": "SmkDrHIfK/8=",
  "traceId": "TGjKf0mdx8+yKZQ+L0QKww==",
  "asInt": "219"
}]
```

otelmini currently does not support exemplars.

## Data Structures

### New Exemplar Class (`point.py`)

```python
@dataclass(frozen=True)
class Exemplar:
    filtered_attributes: Attributes  # Attributes not already in the data point
    value: Union[int, float]
    time_unix_nano: int
    span_id: Optional[bytes] = None   # 8 bytes
    trace_id: Optional[bytes] = None  # 16 bytes
```

### Modified Data Point Classes (`point.py`)

Add `exemplars` field to:

```python
@dataclass(frozen=True)
class NumberDataPoint:
    attributes: Attributes
    start_time_unix_nano: int
    time_unix_nano: int
    value: Union[int, float]
    exemplars: Sequence[Exemplar] = field(default_factory=tuple)  # ADD

@dataclass(frozen=True)
class HistogramDataPoint:
    attributes: Attributes
    start_time_unix_nano: int
    time_unix_nano: int
    count: int
    sum: Union[int, float]
    bucket_counts: Sequence[int]
    explicit_bounds: Sequence[float]
    min: float
    max: float
    exemplars: Sequence[Exemplar] = field(default_factory=tuple)  # ADD
```

## Exemplar Filter

Decides which measurements qualify as exemplar candidates.

### Filter Types

| Filter | Behavior | Use Case |
|--------|----------|----------|
| `AlwaysOffExemplarFilter` | No exemplars | Default, minimal overhead |
| `AlwaysOnExemplarFilter` | All measurements qualify | Testing/debugging |
| `TraceBasedExemplarFilter` | Only sampled spans | Production use |

### Implementation (`exemplar.py` or similar)

```python
class ExemplarFilter(ABC):
    @abstractmethod
    def should_sample(self, value, time_unix_nano, attributes, context) -> bool:
        pass

class AlwaysOffExemplarFilter(ExemplarFilter):
    def should_sample(self, value, time_unix_nano, attributes, context) -> bool:
        return False

class AlwaysOnExemplarFilter(ExemplarFilter):
    def should_sample(self, value, time_unix_nano, attributes, context) -> bool:
        return True

class TraceBasedExemplarFilter(ExemplarFilter):
    def should_sample(self, value, time_unix_nano, attributes, context) -> bool:
        span = trace.get_current_span(context)
        if span == INVALID_SPAN:
            return False
        return span.get_span_context().trace_flags.sampled
```

### Environment Variable

```bash
OTEL_METRICS_EXEMPLAR_FILTER=always_on|always_off|trace_based
```

## Exemplar Reservoir

Stores candidate exemplars using reservoir sampling.

### Reservoir Types

| Reservoir | Behavior | Use Case |
|-----------|----------|----------|
| `SimpleFixedSizeExemplarReservoir` | Uniform sampling, configurable size | Counters, Gauges |
| `AlignedHistogramBucketExemplarReservoir` | One exemplar per bucket | Histograms |

### Implementation

```python
class ExemplarReservoir(ABC):
    @abstractmethod
    def offer(self, value, time_unix_nano, attributes, context) -> None:
        """Called when a measurement passes the filter."""
        pass

    @abstractmethod
    def collect(self, point_attributes: Attributes) -> Sequence[Exemplar]:
        """Called during metric production. Returns and clears stored exemplars."""
        pass

class SimpleFixedSizeExemplarReservoir(ExemplarReservoir):
    def __init__(self, size: int = 1):
        self._size = size
        self._exemplars = []
        self._count = 0

    def offer(self, value, time_unix_nano, attributes, context):
        # Reservoir sampling algorithm
        self._count += 1
        if len(self._exemplars) < self._size:
            self._exemplars.append(self._create_exemplar(value, time_unix_nano, attributes, context))
        else:
            # Probabilistically replace
            idx = random.randint(0, self._count - 1)
            if idx < self._size:
                self._exemplars[idx] = self._create_exemplar(value, time_unix_nano, attributes, context)

    def _create_exemplar(self, value, time_unix_nano, attributes, context):
        span = trace.get_current_span(context)
        span_id = trace_id = None
        if span != INVALID_SPAN:
            ctx = span.get_span_context()
            span_id = ctx.span_id
            trace_id = ctx.trace_id
        return Exemplar(
            filtered_attributes=attributes,  # May need filtering
            value=value,
            time_unix_nano=time_unix_nano,
            span_id=span_id,
            trace_id=trace_id,
        )

    def collect(self, point_attributes):
        result = tuple(self._exemplars)
        self._exemplars.clear()
        self._count = 0
        return result
```

## Metric Recording Integration (`metric.py`)

### Instrument Changes

Each instrument needs a reservoir per attribute combination:

```python
class Counter:
    def __init__(self, name, unit, description, exemplar_filter, reservoir_factory):
        self._name = name
        self._values = {}
        self._reservoirs = {}  # ADD: attr_key -> ExemplarReservoir
        self._exemplar_filter = exemplar_filter
        self._reservoir_factory = reservoir_factory

    def add(self, amount, attributes=None, context=None):
        attr_key = _attributes_to_key(attributes)

        # Existing value accumulation
        self._values[attr_key] = self._values.get(attr_key, 0) + amount

        # Exemplar handling
        if self._exemplar_filter.should_sample(amount, time.time_ns(), attributes, context):
            if attr_key not in self._reservoirs:
                self._reservoirs[attr_key] = self._reservoir_factory()
            self._reservoirs[attr_key].offer(amount, time.time_ns(), attributes, context)
```

### MetricProducer Changes

```python
def _produce_sum_metric(self, instrument, time_unix_nano, *, is_monotonic):
    data_points = []
    for attr_key, value in instrument.get_values().items():
        attributes = _key_to_attributes(attr_key)

        # Collect exemplars
        exemplars = ()
        if attr_key in instrument._reservoirs:
            exemplars = instrument._reservoirs[attr_key].collect(attributes)

        dp = NumberDataPoint(
            attributes=attributes,
            start_time_unix_nano=self._start_time_unix_nano,
            time_unix_nano=time_unix_nano,
            value=value,
            exemplars=exemplars,  # ADD
        )
        data_points.append(dp)
    return data_points
```

## Encoding (`encode.py`)

### New Functions

```python
def _encode_exemplar(exemplar: Exemplar) -> dict:
    result = {
        "timeUnixNano": str(exemplar.time_unix_nano),
    }

    if exemplar.filtered_attributes:
        result["filteredAttributes"] = _encode_attributes(exemplar.filtered_attributes)

    if exemplar.trace_id is not None:
        result["traceId"] = exemplar.trace_id.hex()
    if exemplar.span_id is not None:
        result["spanId"] = exemplar.span_id.hex()

    if isinstance(exemplar.value, float):
        result["asDouble"] = exemplar.value
    else:
        result["asInt"] = str(exemplar.value)

    return result

def _encode_exemplars(exemplars: Sequence[Exemplar]) -> list:
    if not exemplars:
        return []
    return [_encode_exemplar(e) for e in exemplars]
```

### Modified Data Point Encoding

```python
def _encode_number_data_point(point: NumberDataPoint) -> dict:
    result = {
        "attributes": _encode_attributes(point.attributes),
        "startTimeUnixNano": str(point.start_time_unix_nano),
        "timeUnixNano": str(point.time_unix_nano),
    }
    # ... existing value encoding ...

    if point.exemplars:  # ADD
        result["exemplars"] = _encode_exemplars(point.exemplars)

    return result
```

## Implementation Phases

### Phase 1: Data Structures Only (Minimal)
- Add `Exemplar` dataclass to `point.py`
- Add `exemplars` field to `NumberDataPoint`, `HistogramDataPoint`
- Add encoding in `encode.py`
- Exemplars always empty - just ensures compatibility

### Phase 2: Basic Exemplar Collection (Medium)
- Add `TraceBasedExemplarFilter`
- Add `SimpleFixedSizeExemplarReservoir`
- Integrate into `Counter` and `Histogram`
- Update `MetricProducer`

### Phase 3: Full Implementation
- Add `AlwaysOnExemplarFilter`, `AlwaysOffExemplarFilter`
- Add `AlignedHistogramBucketExemplarReservoir`
- Support `OTEL_METRICS_EXEMPLAR_FILTER` env var
- Add configuration to `MeterProvider`

## Testing

```python
def test_exemplar_with_active_span():
    """Verify exemplar captures trace context."""
    provider = MeterProvider(exemplar_filter=AlwaysOnExemplarFilter())
    meter = provider.get_meter("test")
    counter = meter.create_counter("test_counter")

    tracer = trace.get_tracer("test")
    with tracer.start_as_current_span("test_span") as span:
        counter.add(42, {"key": "value"})

    metrics = provider.produce()
    exemplars = metrics.resource_metrics[0].scope_metrics[0].metrics[0].data.data_points[0].exemplars

    assert len(exemplars) == 1
    assert exemplars[0].value == 42
    assert exemplars[0].trace_id == span.get_span_context().trace_id
    assert exemplars[0].span_id == span.get_span_context().span_id
```

## Memory Considerations

| Component | Memory per attribute combination |
|-----------|----------------------------------|
| SimpleFixedSizeReservoir (size=1) | ~100 bytes |
| AlignedHistogramBucketReservoir (16 buckets) | ~1.6 KB |
| Each Exemplar | ~50-100 bytes + attributes |

## References

- [OTLP Metrics Spec](https://opentelemetry.io/docs/specs/otlp/#otlpmetrics)
- [OpenTelemetry Python SDK exemplar implementation](https://github.com/open-telemetry/opentelemetry-python/tree/main/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal)
