---
title: "Horizon Monitoring"
description: "Prometheus metrics, alerting, and calibration tracking for production Horizon strategies."
icon: "chart-mixed"
---

# Horizon Monitoring

Horizon provides production-grade monitoring with three components: **Prometheus metrics** for observability, **alerts** for real-time notifications, and **calibration tracking** for model evaluation.

<CardGroup cols={3}>
  <Card title="Metrics" icon="gauge-high">
    Thread-safe counters, gauges, and histograms with Prometheus text export and a built-in HTTP server.
  </Card>
  <Card title="Alerts" icon="bell">
    Multi-channel alerting (Webhook, Telegram, Discord, Log) with cooldown throttling per alert type.
  </Card>
  <Card title="Calibration" icon="crosshairs">
    Track prediction accuracy with Brier scores and calibration curves across markets.
  </Card>
</CardGroup>

---

## Prometheus Metrics

### MetricsCollector

The `MetricsCollector` provides thread-safe metric primitives that render to Prometheus text format.

```python
from horizon.monitoring import MetricsCollector

collector = MetricsCollector()
```

#### Counter

A monotonically increasing counter. Use for event counts (orders submitted, fills received, errors).

```python
orders_counter = collector.counter("horizon_orders_submitted_total")
orders_counter.inc()       # Increment by 1
orders_counter.inc()       # Now 2
```

#### Gauge

A value that can go up and down. Use for current state (position size, PnL, active orders).

```python
position_gauge = collector.gauge("horizon_position_size")
position_gauge.set(50.0)   # Set to 50
position_gauge.inc()       # Increment by 1 (now 51)
```

#### Histogram

Records observations and computes distribution statistics. Use for latencies, fill sizes, spreads.

```python
latency_hist = collector.histogram("horizon_tick_latency_seconds")
latency_hist.observe(0.0032)
latency_hist.observe(0.0041)
latency_hist.observe(0.0028)
```

#### Rendering

All metrics render to Prometheus text exposition format:

```python
output = collector.render()
print(output)
```

```
# HELP horizon_orders_submitted_total
# TYPE horizon_orders_submitted_total counter
horizon_orders_submitted_total 2

# HELP horizon_position_size
# TYPE horizon_position_size gauge
horizon_position_size 51.0

# HELP horizon_tick_latency_seconds
# TYPE horizon_tick_latency_seconds histogram
horizon_tick_latency_seconds_bucket{le="0.005"} 3
horizon_tick_latency_seconds_bucket{le="0.01"} 3
...
horizon_tick_latency_seconds_sum 0.0101
horizon_tick_latency_seconds_count 3
```

### MetricsServer

Serve metrics over HTTP for Prometheus scraping.

```python
from horizon.monitoring import MetricsCollector, MetricsServer

collector = MetricsCollector()
server = MetricsServer(collector, port=9090)

server.start()
# Metrics now available at http://localhost:9090/metrics

# When shutting down:
server.stop()
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `collector` | `MetricsCollector` | required | The collector to serve |
| `port` | `int` | `9090` | HTTP port to bind |

<Tip>
  Add `http://localhost:9090/metrics` as a Prometheus scrape target in your `prometheus.yml`. Default scrape interval of 15s works well for trading strategies.
</Tip>

### update_engine_metrics

A helper function that populates standard gauges from the engine's current status.

```python
from horizon.monitoring import MetricsCollector, update_engine_metrics

collector = MetricsCollector()

# Call each tick to update gauges
update_engine_metrics(collector, engine)
```

This updates the following gauges automatically:

| Metric | Description |
|---|---|
| `horizon_engine_active_orders` | Number of active orders |
| `horizon_engine_total_fills` | Total fills processed |
| `horizon_engine_position_count` | Number of open positions |
| `horizon_engine_unrealized_pnl` | Current unrealized PnL |
| `horizon_engine_realized_pnl` | Total realized PnL |
| `horizon_engine_kill_switch` | Kill switch status (0 or 1) |

---

## Alerts

### AlertManager

The `AlertManager` routes alerts to one or more channels with per-type cooldown throttling.

```python
from horizon.monitoring import AlertManager, AlertType, AlertLevel
```

### Alert Channels

<Tabs>
  <Tab title="Webhook">
    Send alerts as JSON POST requests to any HTTP endpoint.

    ```python
    from horizon.monitoring import WebhookChannel

    channel = WebhookChannel(url="https://your-server.com/alerts")
    ```

    Payload format:
    ```json
    {
      "type": "PriceMove",
      "level": "Warning",
      "message": "BTC moved 5% in 1 minute",
      "timestamp": "2025-01-15T12:00:00Z"
    }
    ```
  </Tab>
  <Tab title="Telegram">
    Send alerts to a Telegram chat via the Bot API.

    ```python
    from horizon.monitoring import TelegramChannel

    channel = TelegramChannel(
        bot_token="123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11",
        chat_id="-1001234567890",
    )
    ```
  </Tab>
  <Tab title="Discord">
    Send alerts to a Discord channel via webhook.

    ```python
    from horizon.monitoring import DiscordChannel

    channel = DiscordChannel(
        webhook_url="https://discord.com/api/webhooks/1234567890/abcdef",
    )
    ```
  </Tab>
  <Tab title="Log">
    Write alerts to the Python logger. Useful for development and as a fallback.

    ```python
    from horizon.monitoring import LogChannel

    channel = LogChannel()
    ```
  </Tab>
</Tabs>

### AlertType

| Type | Description |
|---|---|
| `AlertType.PriceMove` | Significant price movement detected |
| `AlertType.PositionLimit` | Position limit approached or breached |
| `AlertType.RiskViolation` | Risk pipeline rejected an order |
| `AlertType.KillSwitch` | Kill switch activated |
| `AlertType.FillReceived` | A fill was received |
| `AlertType.Custom` | User-defined alert |

### AlertLevel

| Level | Description |
|---|---|
| `AlertLevel.Info` | Informational (default) |
| `AlertLevel.Warning` | Requires attention |
| `AlertLevel.Critical` | Immediate action required |

### Sending Alerts

```python
from horizon.monitoring import (
    AlertManager, AlertType, AlertLevel,
    TelegramChannel, DiscordChannel, LogChannel,
)

manager = AlertManager()

# Add multiple channels
manager.add_channel(TelegramChannel(bot_token="...", chat_id="..."))
manager.add_channel(DiscordChannel(webhook_url="..."))
manager.add_channel(LogChannel())

# Send alerts
manager.send(
    AlertType.FillReceived,
    "Filled 50 contracts @ 0.55 on election-winner",
    level=AlertLevel.Info,
)

manager.send(
    AlertType.KillSwitch,
    "Kill switch activated: max drawdown exceeded",
    level=AlertLevel.Critical,
)

manager.send(
    AlertType.Custom,
    "Model retrained with new data",
    level=AlertLevel.Info,
)
```

### Cooldown Throttling

Alerts are throttled per `AlertType` to prevent flooding. The default cooldown is 60 seconds per type.

```python
# These two alerts are 1 second apart, the second will be throttled
manager.send(AlertType.PriceMove, "Price moved +3%")
time.sleep(1)
manager.send(AlertType.PriceMove, "Price moved +4%")  # Throttled, not sent

# Different alert types are independent
manager.send(AlertType.FillReceived, "New fill")       # Sent (different type)
```

<Note>
  Cooldown is per alert type, not per message. If you need to bypass throttling for critical alerts, use `AlertType.Custom` with unique types or set a shorter cooldown via `manager.set_cooldown(AlertType.KillSwitch, seconds=5)`.
</Note>

---

## Calibration Tracking

### CalibrationTracker

Track prediction accuracy over time to evaluate your model's calibration.

```python
from horizon.monitoring import CalibrationTracker

tracker = CalibrationTracker()
```

### Recording Predictions

```python
# Record predictions as events resolve
tracker.record(
    market_id="will-it-rain-tomorrow",
    predicted_prob=0.70,
    outcome=1.0,  # 1.0 = Yes, 0.0 = No
)

tracker.record(
    market_id="btc-above-100k",
    predicted_prob=0.45,
    outcome=0.0,
)

tracker.record(
    market_id="fed-rate-cut",
    predicted_prob=0.80,
    outcome=1.0,
)
```

### Brier Score

The Brier score measures the accuracy of probabilistic predictions. Range is 0.0 (perfect) to 1.0 (worst).

```python
# Overall Brier score
score = tracker.brier_score()
print(f"Brier Score: {score:.4f}")

# Per-market scores
by_market = tracker.brier_by_market()
for market, score in by_market.items():
    print(f"  {market}: {score:.4f}")
```

| Brier Score | Interpretation |
|---|---|
| 0.00 | Perfect predictions |
| 0.10 | Excellent calibration |
| 0.20 | Good calibration |
| 0.25 | Equivalent to always predicting 50% |
| 0.33+ | Poor, model is harmful |

### Calibration Curve

Compute a calibration curve to visualize reliability.

```python
curve = tracker.calibration_curve(n_bins=10)

print("Predicted | Actual")
print("-" * 25)
for predicted, actual in curve:
    print(f"  {predicted:.2f}    |  {actual:.2f}")
```

The calibration curve groups predictions into bins and compares the average predicted probability against the actual outcome frequency. A perfectly calibrated model produces points along the diagonal (predicted == actual).

---

## Full Examples

### Production Monitoring Setup

<Steps>
  <Step title="Initialize monitoring components">
    ```python
    from horizon.monitoring import (
        MetricsCollector,
        MetricsServer,
        AlertManager,
        AlertType,
        AlertLevel,
        CalibrationTracker,
        TelegramChannel,
        LogChannel,
        update_engine_metrics,
    )

    # Metrics
    collector = MetricsCollector()
    server = MetricsServer(collector, port=9090)
    server.start()

    # Alerts
    alerts = AlertManager()
    alerts.add_channel(TelegramChannel(
        bot_token="YOUR_BOT_TOKEN",
        chat_id="YOUR_CHAT_ID",
    ))
    alerts.add_channel(LogChannel())

    # Calibration
    calibration = CalibrationTracker()
    ```
  </Step>
  <Step title="Create custom metrics">
    ```python
    # Strategy-specific metrics
    tick_counter = collector.counter("horizon_ticks_total")
    edge_gauge = collector.gauge("horizon_current_edge")
    fill_size_hist = collector.histogram("horizon_fill_size")
    model_latency = collector.histogram("horizon_model_latency_seconds")
    ```
  </Step>
  <Step title="Wire into the pipeline">
    ```python
    import time
    import horizon as hz
    from horizon import Side, OrderSide, OrderRequest

    def monitored_model(ctx):
        start = time.time()

        # Your model logic
        fair = 0.62

        # Record model latency
        elapsed = time.time() - start
        model_latency.observe(elapsed)

        edge = hz.edge(fair, ctx.feed.price)
        edge_gauge.set(edge)

        return {"fair": fair}

    def monitored_quoter(ctx, fair):
        tick_counter.inc()

        if fair > ctx.feed.price + 0.03:
            return OrderRequest(
                market_id=ctx.market_id,
                side=Side.Yes,
                order_side=OrderSide.Buy,
                price=ctx.feed.price,
                size=10.0,
            )

    def fill_monitor(ctx, engine):
        """Track fills and update metrics."""
        update_engine_metrics(collector, engine)

        status = engine.status()
        if status.kill_switch:
            alerts.send(
                AlertType.KillSwitch,
                f"Kill switch activated. PnL: ${status.realized_pnl:.2f}",
                level=AlertLevel.Critical,
            )
    ```
  </Step>
  <Step title="Run the strategy">
    ```python
    try:
        hz.run(
            name="monitored-strategy",
            markets=["election-winner"],
            feeds={"election-winner": "polymarket_book"},
            pipeline=[monitored_model, monitored_quoter, fill_monitor],
        )
    finally:
        server.stop()
    ```
  </Step>
</Steps>

### Alert-Driven Risk Management

```python
import horizon as hz
from horizon import Side, OrderSide, OrderRequest
from horizon.monitoring import (
    AlertManager, AlertType, AlertLevel,
    DiscordChannel, LogChannel,
)

alerts = AlertManager()
alerts.add_channel(DiscordChannel(webhook_url="https://discord.com/api/webhooks/..."))
alerts.add_channel(LogChannel())

def risk_alerter(ctx, engine):
    """Send alerts based on portfolio state."""
    status = engine.status()
    positions = engine.positions()

    # Alert on large positions
    for market_id, pos in positions.items():
        if abs(pos.net_size) > 200:
            alerts.send(
                AlertType.PositionLimit,
                f"Large position in {market_id}: {pos.net_size:.0f} contracts",
                level=AlertLevel.Warning,
            )

    # Alert on significant drawdown
    if status.unrealized_pnl < -100:
        alerts.send(
            AlertType.RiskViolation,
            f"Unrealized loss: ${status.unrealized_pnl:.2f}",
            level=AlertLevel.Warning,
        )

    # Alert on fills
    recent_fills = engine.recent_fills()
    for fill in recent_fills:
        alerts.send(
            AlertType.FillReceived,
            f"Fill: {fill.order_side.name} {fill.size} @ {fill.price:.4f} in {fill.market_id}",
            level=AlertLevel.Info,
        )

def model(ctx):
    return {"fair": 0.60}

def quoter(ctx, fair):
    if fair > ctx.feed.price + 0.03:
        return OrderRequest(
            market_id=ctx.market_id,
            side=Side.Yes,
            order_side=OrderSide.Buy,
            price=ctx.feed.price,
            size=10.0,
        )

hz.run(
    name="alert-strategy",
    markets=["election-winner"],
    feeds={"election-winner": "polymarket_book"},
    pipeline=[model, quoter, risk_alerter],
)
```

### Calibration Tracking Over Time

```python
from horizon.monitoring import CalibrationTracker

tracker = CalibrationTracker()

# Simulate recording predictions as markets resolve
predictions = [
    ("market-1", 0.80, 1.0),
    ("market-2", 0.30, 0.0),
    ("market-3", 0.65, 1.0),
    ("market-4", 0.90, 1.0),
    ("market-5", 0.20, 1.0),  # Bad prediction: said 20% but it happened
    ("market-6", 0.55, 0.0),
    ("market-7", 0.75, 1.0),
    ("market-8", 0.40, 0.0),
    ("market-9", 0.85, 1.0),
    ("market-10", 0.10, 0.0),
]

for market_id, prob, outcome in predictions:
    tracker.record(market_id, prob, outcome)

# Overall performance
print(f"Overall Brier Score: {tracker.brier_score():.4f}")

# Per-market breakdown
print("\nPer-market Brier Scores:")
for market, score in tracker.brier_by_market().items():
    status = "Good" if score < 0.15 else "Needs work"
    print(f"  {market}: {score:.4f} ({status})")

# Calibration curve
print("\nCalibration Curve:")
print("Predicted | Actual | Deviation")
print("-" * 40)
curve = tracker.calibration_curve(n_bins=5)
for predicted, actual in curve:
    deviation = abs(predicted - actual)
    marker = " ***" if deviation > 0.10 else ""
    print(f"  {predicted:.2f}     | {actual:.2f}   | {deviation:.2f}{marker}")
```

### Integrated Monitoring with Backtesting

```python
import horizon as hz
from horizon.monitoring import CalibrationTracker

tracker = CalibrationTracker()

def model(ctx):
    # Your calibrated model
    fair = 0.62
    return {"fair": fair}

def quoter(ctx, fair):
    from horizon import Side, OrderSide, OrderRequest
    if fair > ctx.feed.price + 0.03:
        return OrderRequest(
            market_id=ctx.market_id,
            side=Side.Yes,
            order_side=OrderSide.Buy,
            price=ctx.feed.price,
            size=10.0,
        )

# Run backtest with outcomes for Brier scoring
data = [
    {"timestamp": t, "price": 0.50 + (t % 10) * 0.01}
    for t in range(1700000000, 1700000500)
]

result = hz.backtest(
    name="calibration-backtest",
    markets=["test-market"],
    data=data,
    pipeline=[model, quoter],
    outcomes={"test-market": 1.0},
)

# Check backtest Brier score
m = result.metrics
if m.brier_score is not None:
    print(f"Backtest Brier Score: {m.brier_score:.4f}")

# Also track in calibration tracker for running statistics
tracker.record("test-market", 0.62, 1.0)
print(f"Tracker Brier Score:  {tracker.brier_score():.4f}")

print(result.summary())
```

---

## Grafana Dashboard

<Accordion title="Example Grafana configuration for Horizon metrics">
  Once the MetricsServer is running, add it as a Prometheus data source in Grafana and create panels:

  ```yaml
  # prometheus.yml
  scrape_configs:
    - job_name: "horizon"
      scrape_interval: 15s
      static_configs:
        - targets: ["localhost:9090"]
  ```

  Suggested panels:
  - **PnL Over Time**: `horizon_engine_realized_pnl` + `horizon_engine_unrealized_pnl`
  - **Order Rate**: `rate(horizon_orders_submitted_total[1m])`
  - **Position Size**: `horizon_engine_position_count`
  - **Tick Latency**: `histogram_quantile(0.95, horizon_tick_latency_seconds_bucket)`
  - **Fill Size Distribution**: `horizon_fill_size_bucket`
  - **Kill Switch Status**: `horizon_engine_kill_switch` (alert on value == 1)
</Accordion>

<Warning>
  Never expose the metrics server to the public internet. Use a reverse proxy with authentication or bind only to localhost. The metrics endpoint does not require authentication by default.
</Warning>
