# Custom Probes API

Module: `citadel_operator.k3s.custom_probes`

Custom probes detect failure modes beyond standard Kubernetes conditions — stuck volumes, GPU errors, unresponsive sidecar health endpoints.

## Enums

```python
class ProbeType(Enum):
    EXEC = "exec"
    HTTP = "http"

class ProbeStatus(Enum):
    PASSING = "passing"
    FAILING = "failing"
    UNKNOWN = "unknown"
```

## CustomProbeSpec

Frozen dataclass defining a probe to execute.

```python
@dataclass(frozen=True)
class CustomProbeSpec:
    name: str
    probe_type: ProbeType
    failure_threshold: int = 3       # Consecutive failures before "failed"
    period_seconds: int = 60         # Execution interval
    exec_config: ExecProbeConfig | None = None
    http_config: HttpProbeConfig | None = None
```

Validates in `__post_init__` that the matching config is provided for the probe type.

### ExecProbeConfig

```python
@dataclass(frozen=True)
class ExecProbeConfig:
    command: list[str]
    timeout_seconds: int = 10
```

### HttpProbeConfig

```python
@dataclass(frozen=True)
class HttpProbeConfig:
    path: str
    port: int
    scheme: str = "http"
    host: str = "localhost"

    @property
    def url(self) -> str: ...  # Full probe URL
```

## ProbeResult

Frozen dataclass representing a single probe execution result.

```python
@dataclass(frozen=True)
class ProbeResult:
    probe_name: str
    probe_type: ProbeType
    status: ProbeStatus
    consecutive_failures: int
    failure_threshold: int
    message: str
    timestamp: float

    @property
    def threshold_exceeded(self) -> bool: ...  # consecutive_failures >= failure_threshold
```

## CustomProbeReport

Frozen dataclass aggregating all probe results for a node.

```python
@dataclass(frozen=True)
class CustomProbeReport:
    node_name: str
    results: list[ProbeResult]
    timestamp: float

    @property
    def any_failed(self) -> bool: ...       # Any probe exceeded threshold
    @property
    def failed_probes(self) -> list[ProbeResult]: ...
    @property
    def passing_probes(self) -> list[ProbeResult]: ...
```

## CustomProbeChecker

Stateful checker that tracks consecutive failures per probe per node.

### Constructor

```python
CustomProbeChecker()
```

### Methods

| Method | Signature | Description |
|--------|-----------|-------------|
| `check` | `(node_name, probes) -> CustomProbeReport` | Execute probes, track failures |
| `reset` | `(node_name=None)` | Reset failure state (all or per-node) |
| `get_failure_count` | `(node_name, probe_name) -> int` | Current consecutive failures |

### Behavior

- Probes only execute if `period_seconds` has elapsed since last check
- Failure count increments on each failed execution
- Failure count resets to 0 on success
- A probe is only marked `FAILING` (threshold exceeded) after `failure_threshold` consecutive failures

## parse_custom_probes

```python
def parse_custom_probes(raw_probes: list[dict]) -> list[CustomProbeSpec]
```

Parse custom probe specs from CRD YAML format.

## Usage

```python
from citadel_operator import CustomProbeChecker, CustomProbeSpec, parse_custom_probes
from citadel_operator.k3s.custom_probes import ExecProbeConfig, ProbeType

# Define probes
probes = [
    CustomProbeSpec(
        name="disk-check",
        probe_type=ProbeType.EXEC,
        exec_config=ExecProbeConfig(command=["df", "-h", "/var/lib/kubelet"]),
        failure_threshold=3,
        period_seconds=60,
    ),
]

# Or parse from YAML
raw = [{"name": "disk-check", "type": "exec", "exec": {"command": ["df", "-h"]}}]
probes = parse_custom_probes(raw)

# Execute
checker = CustomProbeChecker()
report = checker.check("worker-1", probes)

if report.any_failed:
    for probe in report.failed_probes:
        print(f"FAILED: {probe.probe_name} ({probe.consecutive_failures}/{probe.failure_threshold})")
```
