# OtelTest API Guide: How to Create Tests

This guide documents the complete API for creating tests in oteltest, based on source code exploration.

## Test Class Structure

### Basic Pattern

All test classes must either:
1. Inherit from `OtelTest` (from `oteltest import OtelTest`)
2. Or have "OtelTest" in the class name

```python
# Option 1: Inherit from OtelTest
from oteltest import OtelTest

class MyTest(OtelTest):
    # implement required methods
    pass

# Option 2: Class name contains "OtelTest"
class MyOtelTest:
    # implement required methods
    pass
```

## Required Methods

Every test class must implement these 5 methods:

### 1. `environment_variables() -> Mapping[str, str]`

Returns environment variables for the script subprocess.

**Example:**
```python
def environment_variables(self):
    return {
        "OTEL_SERVICE_NAME": "my-service",
        "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
        "OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:4318",
    }
```

### 2. `requirements() -> Sequence[str]`

Returns pip packages to install in the test's isolated virtual environment.

**Example:**
```python
def requirements(self):
    return [
        "opentelemetry-api==1.27.0",
        "opentelemetry-sdk==1.27.0",
        "opentelemetry-exporter-otlp-proto-http==1.27.0",
    ]
```

### 3. `wrapper_command() -> str | None`

Returns command to wrap script execution.

**Options:**
- `""` or `None` - Direct Python execution (direct SDK usage)
- `"opentelemetry-instrument"` - Auto-instrumentation wrapper

**Example:**
```python
def wrapper_command(self):
    return "opentelemetry-instrument"  # For auto-instrumentation

def wrapper_command(self):
    return None  # For direct SDK usage
```

### 4. `is_http() -> bool`

Controls which OTel receiver protocol to use.

**Options:**
- `True` - HTTP receiver on port 4318
- `False` - gRPC receiver on port 4317

**Example:**
```python
def is_http(self):
    return True  # HTTP
```

### 5. `on_start() -> float | None`

Called immediately after script starts. Runs in oteltest's environment (not the subprocess).

**Use cases:**
- Make HTTP requests to trigger server behavior
- Wait for service liveness
- Trigger script actions

**Return value:**
- `None` - Let script run until it exits naturally
- `float` - Seconds to wait before force-terminating subprocess

**Example:**
```python
def on_start(self):
    import time
    import http.client

    time.sleep(5)  # Wait for server startup

    # Trigger telemetry by making request
    conn = http.client.HTTPConnection("localhost", 8080)
    conn.request("GET", "/")
    conn.getresponse()
    conn.close()

    return 30  # Wait up to 30 seconds for script to finish
```

### 6. `on_stop(tel: Telemetry, stdout: str, stderr: str, returncode: int) -> None`

Called after script completes. Use for validation and assertions.

**Parameters:**
- `tel: Telemetry` - Captured telemetry data (traces, metrics, logs)
- `stdout: str` - Script's stdout output
- `stderr: str` - Script's stderr output
- `returncode: int` - Script's exit code

**Example:**
```python
def on_stop(self, tel, stdout: str, stderr: str, returncode: int):
    from oteltest.telemetry import count_spans, get_span_names

    # Basic assertions
    assert returncode == 0, f"Script failed with code {returncode}"
    assert tel.traces is not None, "No traces received"

    # Validate telemetry
    assert count_spans(tel) == 2, "Expected 2 spans"
    assert "my-span-name" in get_span_names(tel)

    print("✓ All assertions passed")
```

## Telemetry Object

The `Telemetry` object passed to `on_stop()` contains:

```python
class Telemetry:
    trace_requests: list[Request]   # All trace requests
    metric_requests: list[Request]  # All metric requests
    log_requests: list[Request]     # All log requests
```

### Accessing Protobuf Data

Access protobuf structures directly:

```python
def on_stop(self, tel, stdout, stderr, returncode):
    # Access traces
    if tel.traces is not None:
        for resource_span in tel.traces.resource_spans:
            for scope_span in resource_span.scope_spans:
                for span in scope_span.spans:
                    print(f"Span: {span.name}")

    # Access metrics
    if tel.metrics is not None:
        for resource_metric in tel.metrics.resource_metrics:
            for scope_metric in resource_metric.scope_metrics:
                for metric in scope_metric.metrics:
                    print(f"Metric: {metric.name}")

    # Access logs
    if tel.logs is not None:
        for resource_log in tel.logs.resource_logs:
            for scope_log in resource_log.scope_logs:
                for log_record in scope_log.log_records:
                    print(f"Log: {log_record.body}")
```

### Helper Functions

Use these helper functions from `oteltest.telemetry`:

```python
from oteltest.telemetry import (
    count_spans,              # int - total spans received
    count_metrics,            # int - total metrics received
    count_logs,               # int - total logs received
    get_spans,                # list - all span objects
    get_metrics,              # list - all metric objects
    get_logs,                 # list - all log objects
    get_span_names,           # set - unique span names
    get_metric_names,         # set - unique metric names
    first_span,               # Span - first span in telemetry
    span_attribute_by_name,   # get span attribute value
    get_attribute,            # get attribute from list
    has_log_attribute,        # bool - check if log has attribute
)
```

**Example usage:**
```python
def on_stop(self, tel, stdout, stderr, returncode):
    from oteltest.telemetry import count_spans, get_span_names, first_span

    # Count spans
    num_spans = count_spans(tel)
    assert num_spans > 0, "No spans received"

    # Check span names
    span_names = get_span_names(tel)
    assert "my-operation" in span_names

    # Get first span
    span = first_span(tel)
    assert span.name == "my-operation"
```

## Test Execution Lifecycle

```
1. oteltest discovers test class in .py file
2. Creates isolated virtualenv
3. Installs packages from requirements()
4. Starts OTel sink (gRPC port 4317 or HTTP port 4318)
5. Starts script subprocess with:
   - Environment variables from environment_variables()
   - Wrapped with wrapper_command() if specified
6. Calls on_start() immediately
   - Use to trigger script behavior
   - Returns timeout or None
7. Script runs and sends telemetry to sink
8. After timeout/completion, calls on_stop()
   - Receives Telemetry object with all captured data
   - Run assertions here
9. Saves telemetry to JSON file (script.0.json, script.1.json, etc.)
10. Cleanup and exit
```

## Complete Example: Trace Test

```python
"""
Baseline test: Basic trace export via HTTP with protobuf
"""

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

SERVICE_NAME = "baseline-trace-http"

if __name__ == "__main__":
    # Set up tracing
    provider = TracerProvider()
    exporter = OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
    processor = BatchSpanProcessor(exporter)
    provider.add_span_processor(processor)
    trace.set_tracer_provider(provider)

    # Create test span
    tracer = trace.get_tracer(__name__)
    with tracer.start_as_current_span("test-span") as span:
        span.set_attribute("test.key", "test.value")
        span.set_attribute("test.number", 42)
        span.add_event("test-event", {"event.key": "event.value"})

    # Ensure export
    provider.force_flush()
    provider.shutdown()
    print("✓ Test complete")


class BaselineTraceHttpOtelTest:
    """Test class for oteltest"""

    def requirements(self):
        return [
            "opentelemetry-api==1.27.0",
            "opentelemetry-sdk==1.27.0",
            "opentelemetry-exporter-otlp-proto-http==1.27.0",
        ]

    def environment_variables(self):
        return {"OTEL_SERVICE_NAME": SERVICE_NAME}

    def wrapper_command(self):
        return None  # Direct SDK usage

    def on_start(self):
        return None  # Wait for script to finish naturally

    def on_stop(self, tel, stdout: str, stderr: str, returncode: int):
        """Validate baseline behavior"""
        from oteltest.telemetry import count_spans, first_span

        # Basic checks
        assert returncode == 0, f"Script failed with code {returncode}"
        assert "✓" in stdout, "Script didn't complete successfully"

        # Telemetry validation
        assert tel.traces is not None, "No traces received"
        assert count_spans(tel) == 1, f"Expected 1 span, got {count_spans(tel)}"

        # Span validation
        span = first_span(tel)
        assert span.name == "test-span", f"Wrong span name: {span.name}"

        # Attributes
        attrs = {attr.key: attr.value for attr in span.attributes}
        assert "test.key" in attrs, "Missing test.key attribute"
        assert attrs["test.key"].string_value == "test.value"
        assert attrs["test.number"].int_value == 42

        # Events
        assert len(span.events) == 1, f"Expected 1 event, got {len(span.events)}"
        event = span.events[0]
        assert event.name == "test-event"

        print("✓ All baseline assertions passed")

    def is_http(self):
        return True  # Use HTTP receiver
```

## Complete Example: Client-Server Test

```python
"""
Test Flask server with auto-instrumentation
"""

import time
from flask import Flask

PORT = 8002
HOST = "127.0.0.1"

if __name__ == "__main__":
    app = Flask(__name__)

    @app.route("/")
    def home():
        return "hello"

    app.run(port=PORT, host=HOST)


class FlaskOtelTest:
    """Test class for oteltest"""

    def requirements(self):
        return [
            "flask",
            "opentelemetry-distro",
            "opentelemetry-exporter-otlp-proto-grpc",
            "opentelemetry-instrumentation-flask",
        ]

    def environment_variables(self):
        return {}

    def wrapper_command(self):
        return "opentelemetry-instrument"  # Auto-instrumentation

    def is_http(self):
        return False  # Use gRPC

    def on_start(self):
        import http.client

        # Wait for server startup
        time.sleep(5)

        # Make request to trigger telemetry
        conn = http.client.HTTPConnection(HOST, PORT)
        conn.request("GET", "/")
        response = conn.getresponse()
        print("Response:", response.read().decode())
        conn.close()

        return 30  # Wait up to 30 seconds

    def on_stop(self, tel, stdout: str, stderr: str, returncode: int):
        from oteltest.telemetry import count_spans

        # Validate spans were received
        num_spans = count_spans(tel)
        assert num_spans > 0, f"Expected spans, got {num_spans}"

        print(f"✓ Received {num_spans} spans")
```

## Running Tests

```bash
# Run single test
oteltest path/to/test_script.py

# Run all tests in directory
oteltest path/to/test_directory/

# Results
# - Console output shows pass/fail
# - Telemetry saved to script.0.json, script.1.json, etc.
```

## Key Patterns

### Pattern 1: Direct SDK Usage (No Auto-Instrumentation)

```python
def wrapper_command(self):
    return None

def requirements(self):
    return [
        "opentelemetry-api==1.27.0",
        "opentelemetry-sdk==1.27.0",
        "opentelemetry-exporter-otlp-proto-http==1.27.0",
    ]
```

### Pattern 2: Auto-Instrumentation

```python
def wrapper_command(self):
    return "opentelemetry-instrument"

def requirements(self):
    return [
        "opentelemetry-distro",
        "opentelemetry-exporter-otlp-proto-grpc",
        "opentelemetry-instrumentation-flask",  # or other instrumentation
    ]
```

### Pattern 3: Short-Lived Scripts

```python
def on_start(self):
    return None  # Let script exit naturally

# Script should call shutdown/flush and exit on its own
```

### Pattern 4: Long-Running Servers

```python
def on_start(self):
    import time
    time.sleep(5)  # Wait for startup

    # Trigger behavior (HTTP request, etc.)
    make_http_request()

    return 30  # Force kill after 30 seconds
```

### Pattern 5: Complex Validation

```python
def on_stop(self, tel, stdout, stderr, returncode):
    assert tel.traces is not None, "No traces"

    # Navigate protobuf structure
    resource_span = tel.traces.resource_spans[0]
    scope_span = resource_span.scope_spans[0]
    spans = scope_span.spans

    # Validate each span
    for span in spans:
        assert span.name != "", "Empty span name"
        assert len(span.attributes) > 0, "No attributes"

    print(f"✓ Validated {len(spans)} spans")
```

## Tips & Best Practices

1. **Class naming:** Include "OtelTest" in the class name for automatic discovery
2. **Version pinning:** Pin exact versions in `requirements()` for reproducibility
3. **Assertions:** Use assertions in `on_stop()` to fail tests on validation errors
4. **Timeouts:** Return reasonable timeouts from `on_start()` for long-running servers
5. **Print statements:** Use print() in `on_stop()` to show validation progress
6. **Direct vs wrapped:** Use `wrapper_command()=None` for SDK tests, `"opentelemetry-instrument"` for auto-instrumentation tests
7. **Protocol matching:** Ensure `is_http()` matches your exporter configuration
8. **Helper functions:** Import telemetry helpers in `on_stop()` for cleaner code
