# oteltest Baseline Tests Brief

**Purpose:** Briefing document for creating comprehensive baseline tests in the oteltest repository

**Date:** January 2026

**Context:** Starting work on OpenTelemetry Python OTLP exporter improvements (JSON support + architectural refactoring)

---

## Background

### The Problem

I'm planning to work on OpenTelemetry Python OTLP exporters to:
1. Add JSON encoding support (currently only binary protobuf)
2. Refactor architecture to use composition instead of mixins (cleaner design)
3. Create incubating packages with SOLID principles and immutability

**However:** OTLP exporters are critical infrastructure used by companies worldwide in production. Any regression would have global impact.

### Current Testing Situation

OpenTelemetry Python has unit tests for exporters, but they:
- Mock network calls (don't test actual OTLP protocol)
- Don't comprehensively test error conditions
- Don't test concurrency/threading thoroughly
- Don't test edge cases (large payloads, empty spans, etc.)
- Don't provide regression detection for refactoring

**Risk:** Making architectural changes without comprehensive integration tests = high probability of shipping production bugs globally.

---

## The Solution: Comprehensive Baseline Tests

### What We Need

A comprehensive suite of **integration tests** that:
1. Test actual OTLP protocol (not mocked)
2. Cover happy path, errors, concurrency, edge cases
3. Capture current behavior as "golden files"
4. Detect regressions when refactoring
5. Validate new implementations match old behavior

### Why oteltest?

oteltest is **perfect** for this because:
- Black-box testing: Runs real scripts, captures real telemetry
- Integration-focused: Tests actual OTLP receiver communication
- Flexible: Can test any Python script/configuration
- Golden files: Saves telemetry JSON for regression comparison
- Multi-scenario: Handles client-server, async, threading, etc.

### Why NOT in OpenTelemetry Python repo?

**Better to keep tests in oteltest because:**
1. **Separation of concerns** - Testing framework separate from implementation
2. **Political neutrality** - Not implying existing OTel tests are insufficient
3. **Multi-purpose** - Can test upstream OTel, Splunk distro, different versions
4. **Full control** - No PR reviews needed to iterate on tests
5. **Reusable** - Others can use these tests for their exporters

---

## Goals

### Primary Goal: Enable Safe Refactoring

**Before making ANY changes to OTLP exporters:**
1. Write comprehensive baseline tests
2. Run against current exporters
3. Capture telemetry as golden files
4. Then refactor/add features
5. Run same tests, compare telemetry
6. Ship only if no regressions

### Secondary Goal: Demonstrate oteltest Value

These baseline tests will serve as:
- Real-world oteltest examples
- Demonstration of comprehensive testing
- Reference implementation for others
- Proof that oteltest enables safe refactoring

---

## Scope

### What to Test

**Categories:**

1. **Happy Path** (basic functionality works)
   - Traces: protobuf over HTTP, protobuf over gRPC
   - Metrics: protobuf over HTTP, protobuf over gRPC
   - Logs: protobuf over HTTP, protobuf over gRPC

2. **Configuration** (env vars, parameters work correctly)
   - Endpoint configuration (env vars, constructor)
   - Compression (gzip, deflate, none)
   - Headers (custom headers, auth)
   - Timeout configuration
   - TLS/certificates

3. **Error Conditions** (failures handled gracefully)
   - Network unreachable
   - Invalid endpoint
   - Timeout
   - Server errors (500, 503)
   - Connection refused
   - DNS failure

4. **Concurrency** (thread-safe, no race conditions)
   - Multiple threads exporting simultaneously
   - Async/await patterns
   - Fork safety (if applicable)

5. **Metrics-Specific** (temporality, aggregation)
   - DELTA vs CUMULATIVE temporality
   - Explicit vs Exponential histogram aggregation
   - Gauge, Counter, Histogram, UpDownCounter

6. **Edge Cases** (unusual but valid scenarios)
   - Empty spans (no attributes, no events)
   - Large payloads (1000+ spans)
   - Unicode in attributes
   - Deeply nested spans
   - Span limits (max attributes, max events)
   - Zero/negative metric values

### What Each Test Should Do

```python
# Pattern for baseline tests

class MyBaselineTest:
    def requirements(self):
        # Install current stable version of OTel
        return [
            "opentelemetry-api==1.27.0",
            "opentelemetry-sdk==1.27.0",
            "opentelemetry-exporter-otlp-proto-http==1.27.0",
        ]

    def environment_variables(self):
        return {"OTEL_SERVICE_NAME": "baseline-test"}

    def wrapper_command(self):
        return None  # Direct SDK usage

    def on_start(self):
        return 10  # Timeout in seconds

    def on_stop(self, tel, stdout, stderr, returncode):
        # ASSERT current behavior
        # These assertions document what SHOULD happen
        assert returncode == 0, "Script should not crash"
        assert tel.traces is not None, "Telemetry should be received"
        # ... detailed assertions ...

        print("✓ Baseline behavior verified")

    def is_http(self):
        return True  # Or False for gRPC
```

---

## Structure

### Proposed Directory Organization

```
oteltest/
  baseline_tests/
    README.md                           # Overview, how to run
    opentelemetry_python/
      v1_27/                            # Version-specific tests
        traces/
          http_protobuf.py              # Basic trace export via HTTP
          grpc_protobuf.py              # Basic trace export via gRPC
          compression_gzip.py           # Test gzip compression
          nested_spans.py               # Test span parent/child
        metrics/
          http_protobuf.py
          temporality_delta.py
          temporality_cumulative.py
          aggregation_explicit.py
          aggregation_exponential.py
        logs/
          http_protobuf.py
        errors/
          network_unreachable.py
          invalid_endpoint.py
          timeout.py
          server_error_500.py
        concurrency/
          multithreaded.py
          async_export.py
        edge_cases/
          empty_spans.py
          large_payload.py
          unicode_attributes.py
      v1_28/                            # Future version
        ...
    splunk_distro/                      # Can also test Splunk distro
      v1_0/
        ...
```

### Golden Files

When tests run, oteltest saves telemetry to JSON:
```
baseline_tests/opentelemetry_python/v1_27/traces/http_protobuf.1.json
baseline_tests/opentelemetry_python/v1_27/traces/http_protobuf.2.json
...
```

These become the **regression baselines**.

---

## How They'll Be Used

### Phase 1: Establish Baseline (Weeks 1-3)

```bash
cd oteltest

# Run all baseline tests against current OTel exporters
oteltest baseline_tests/opentelemetry_python/v1_27/

# Produces golden *.1.json files
# These represent "correct" behavior
```

### Phase 2: Development (Weeks 4-12)

```bash
# Build incubating exporters in OpenTelemetry Python fork
cd ~/github/opentelemetry-python-fork
# ... write code ...

# Modify baseline tests to use incubating exporters
# Run same tests
cd ~/github/oteltest
oteltest baseline_tests/opentelemetry_python/v1_27/

# Produces new *.2.json files
```

### Phase 3: Regression Detection

```bash
# Compare telemetry
python scripts/compare_telemetry.py \
  baseline_tests/opentelemetry_python/v1_27/traces/http_protobuf.1.json \
  baseline_tests/opentelemetry_python/v1_27/traces/http_protobuf.2.json

# Output:
# ✓ No regressions detected
# OR
# ✗ REGRESSION: Span count differs (expected 10, got 9)
```

### Phase 4: Reference in PRs

When submitting PRs to OpenTelemetry Python:

```markdown
## Testing

- ✅ All existing unit tests pass
- ✅ Comprehensive baseline testing via oteltest
  - 30+ integration tests covering happy path, errors, concurrency, edge cases
  - Results: https://github.com/pmcollins/oteltest/runs/...
  - Golden file comparison: No regressions detected

To reproduce:
```bash
pip install oteltest
git clone https://github.com/pmcollins/oteltest
oteltest oteltest/baseline_tests/opentelemetry_python/v1_27/
```
```

---

## Success Criteria

### Minimum Viable Baseline (MVP)

**15-20 tests covering:**
- Happy path: traces, metrics, logs (HTTP + gRPC protobuf)
- Key errors: network failure, timeout, invalid endpoint
- Basic concurrency: multithreaded export
- Key edge cases: empty spans, large payloads

**Timeline:** 2 weeks

### Comprehensive Baseline

**30+ tests covering:**
- All categories listed in Scope
- Multiple configurations
- All error conditions
- All edge cases

**Timeline:** 3-4 weeks

---

## Technical Details

### Dependencies

Tests will use:
- `opentelemetry-api` (stable version, e.g., 1.27.0)
- `opentelemetry-sdk` (stable version)
- `opentelemetry-exporter-otlp-proto-http` (stable version)
- `opentelemetry-exporter-otlp-proto-grpc` (stable version)

### Test Execution

```bash
# Run all baseline tests
oteltest baseline_tests/opentelemetry_python/v1_27/

# Run specific category
oteltest baseline_tests/opentelemetry_python/v1_27/errors/

# Run single test
oteltest baseline_tests/opentelemetry_python/v1_27/traces/http_protobuf.py
```

### Assertions

Each test should assert:
1. **Script behavior** - returncode, stdout/stderr
2. **Telemetry structure** - correct protobuf structure
3. **Telemetry content** - spans/metrics/logs have expected values
4. **Protocol correctness** - follows OTLP spec

Example:
```python
def on_stop(self, tel, stdout, stderr, returncode):
    # Script succeeded
    assert returncode == 0

    # Telemetry received
    assert tel.traces is not None
    assert len(tel.traces.resource_spans) > 0

    # Structure correct
    spans = tel.traces.resource_spans[0].scope_spans[0].spans
    assert len(spans) == 1

    # Content correct
    span = spans[0]
    assert span.name == "expected-span-name"
    assert len(span.attributes) > 0

    print("✓ All assertions passed")
```

---

## Benefits

### For Me (Developer)

- ✅ **Confidence** - Can refactor without fear of breaking things
- ✅ **Fast feedback** - Catch regressions locally before shipping
- ✅ **Documentation** - Tests document expected behavior
- ✅ **Credibility** - Demonstrate thoroughness in PRs

### For OpenTelemetry Community

- ✅ **Quality** - Higher confidence in refactoring/changes
- ✅ **Examples** - Real-world oteltest usage examples
- ✅ **Reusability** - Others can use these tests
- ✅ **Trust** - Shows rigorous testing approach

### For oteltest Project

- ✅ **Real-world usage** - Demonstrates value
- ✅ **Examples** - Comprehensive test suite others can learn from
- ✅ **Validation** - Proves oteltest works for complex scenarios
- ✅ **Adoption** - Increases visibility/usage

---

## Risks & Mitigations

### Risk: Tests become stale

**Mitigation:** Version tests by OTel version (v1_27/, v1_28/, etc.)

### Risk: Tests too slow

**Mitigation:**
- Keep tests focused (each test ~5-10 seconds)
- Parallelize where possible
- Can run subsets during development

### Risk: Flaky tests

**Mitigation:**
- Use deterministic scenarios
- Adequate timeouts
- Retry logic in oteltest

### Risk: Maintaining tests

**Mitigation:**
- DRY principles (shared utilities)
- Clear documentation
- Version isolation

---

## Next Steps

### Immediate (This Conversation)

1. **Set up directory structure** in oteltest repo
   ```bash
   mkdir -p baseline_tests/opentelemetry_python/v1_27/{traces,metrics,logs,errors,concurrency,edge_cases}
   ```

2. **Create README** explaining structure and usage

3. **Write first 5-10 tests** (MVP):
   - traces/http_protobuf.py
   - traces/grpc_protobuf.py
   - metrics/http_protobuf.py
   - errors/network_unreachable.py
   - concurrency/multithreaded.py

4. **Run and validate** tests produce expected results

5. **Document** how to run, how to compare results

### Short-term (Next 2-3 Weeks)

6. **Expand to 30+ tests** covering all categories

7. **Create comparison tooling** to diff golden files

8. **Document workflow** for using in development

### Long-term (Ongoing)

9. **Add new versions** as OTel releases (v1_28/, v1_29/, etc.)

10. **Optional:** Propose CI integration with OpenTelemetry Python

11. **Share** with community as example of comprehensive testing

---

## Questions to Address

As we build the baseline tests:

1. **Which OTel version to start with?**
   - Recommend: Latest stable (currently 1.27.x)

2. **How detailed should assertions be?**
   - Balance: Catch real regressions without being brittle
   - Focus on observable behavior, not internal implementation

3. **How to handle acceptable differences?**
   - Timestamps will differ - that's okay
   - Some internal IDs might differ - that's okay
   - Structure and values should match

4. **Should we test deprecated features?**
   - Yes, if still supported in current version
   - Document deprecation in test

5. **How to organize utilities?**
   - Create `baseline_tests/utils/` for shared code
   - Assertion helpers
   - Telemetry comparison functions

---

## Example: First Test to Write

```python
# baseline_tests/opentelemetry_python/v1_27/traces/http_protobuf.py

"""
Baseline test: Basic trace export via HTTP with protobuf encoding

This test establishes the baseline behavior for the most common use case:
exporting traces over HTTP using protobuf encoding.

Expected behavior:
- Script completes successfully (returncode 0)
- Telemetry is received by OTLP receiver
- Span structure matches OTLP specification
- Attributes and events are preserved
"""

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

SERVICE_NAME = "baseline-trace-http-protobuf"

if __name__ == "__main__":
    # Set up tracing
    provider = TracerProvider()
    exporter = OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
    processor = BatchSpanProcessor(exporter)
    provider.add_span_processor(processor)
    trace.set_tracer_provider(provider)

    # Create test span
    tracer = trace.get_tracer(__name__)
    with tracer.start_as_current_span("test-span") as span:
        span.set_attribute("test.key", "test.value")
        span.set_attribute("test.number", 42)
        span.add_event("test-event", {"event.key": "event.value"})

    # Ensure export
    provider.force_flush()
    provider.shutdown()

    print("✓ Test complete")


class BaselineTraceHttpProtobuf:
    """oteltest test class"""

    def requirements(self):
        return [
            "opentelemetry-api==1.27.0",
            "opentelemetry-sdk==1.27.0",
            "opentelemetry-exporter-otlp-proto-http==1.27.0",
        ]

    def environment_variables(self):
        return {"OTEL_SERVICE_NAME": SERVICE_NAME}

    def wrapper_command(self):
        return None  # Direct SDK usage

    def on_start(self):
        return 10  # Wait up to 10 seconds

    def on_stop(self, tel, stdout: str, stderr: str, returncode: int):
        """Validate baseline behavior"""

        # Basic success checks
        assert returncode == 0, f"Script failed with code {returncode}"
        assert "✓" in stdout, "Script didn't complete successfully"
        assert tel.traces is not None, "No traces received"

        # Structural validation
        assert len(tel.traces.resource_spans) > 0, "No resource spans"
        resource_span = tel.traces.resource_spans[0]

        assert len(resource_span.scope_spans) > 0, "No scope spans"
        scope_span = resource_span.scope_spans[0]

        assert len(scope_span.spans) == 1, f"Expected 1 span, got {len(scope_span.spans)}"

        # Span content validation
        span = scope_span.spans[0]
        assert span.name == "test-span", f"Wrong span name: {span.name}"

        # Attributes
        attrs = {attr.key: attr.value for attr in span.attributes}
        assert "test.key" in attrs, "Missing test.key attribute"
        assert attrs["test.key"].string_value == "test.value"
        assert "test.number" in attrs, "Missing test.number attribute"
        assert attrs["test.number"].int_value == 42

        # Events
        assert len(span.events) == 1, f"Expected 1 event, got {len(span.events)}"
        event = span.events[0]
        assert event.name == "test-event", f"Wrong event name: {event.name}"

        event_attrs = {attr.key: attr.value for attr in event.attributes}
        assert "event.key" in event_attrs
        assert event_attrs["event.key"].string_value == "event.value"

        print("✓ All baseline assertions passed")

    def is_http(self):
        return True  # Use HTTP receiver (port 4318)
```

---

## Summary

**What:** Comprehensive baseline test suite for OpenTelemetry Python OTLP exporters

**Where:** oteltest repository (`baseline_tests/opentelemetry_python/`)

**Why:** Enable safe refactoring and new feature development without production regressions

**How:** Black-box integration tests using oteltest, capturing golden files for regression detection

**When:** Before making any changes to OTLP exporters

**Timeline:** 2-3 weeks to build comprehensive baseline

**Next:** Set up directory structure and write first 5-10 tests

---

## Ready to Start

This document should provide complete context for starting a new conversation in the oteltest repo to build the baseline test suite.

Key points to remember:
1. Tests go in oteltest (not OpenTelemetry Python)
2. Focus on integration/black-box testing
3. Capture current behavior, then compare during refactoring
4. ~30 tests covering happy path, errors, concurrency, edge cases
5. Version-specific (v1_27/, v1_28/, etc.)
