# OCG Operator — Session Handover (2026-02-17)

**Branch**: `feature/temporal-functions`
**Session**: Operator implementation (all 5 Maturity Levels)
**Status**: ✅ Builds clean — `cargo build -p ocg-operator` passes

---

## What Was Built

A full Kubernetes operator for `OCGGraph` inside a new `operator/` workspace crate.
Implements the complete Operator Maturity Model (Levels 1–5).

### Files Changed / Created

| Path | Change |
|------|--------|
| `Cargo.toml` | Fixed repo URL (`nxcypher-rust` → `ocg-persistent`); added `[workspace]` with `members = [".", "operator"]`, `resolver = "2"` |
| `operator/Cargo.toml` | New crate: `ocg-operator v0.1.0` |
| `operator/src/main.rs` | tokio main; `--print-crd` flag; metrics server bootstrap |
| `operator/src/crd/mod.rs` | `OCGGraph` CRD (kube-rs derive); `OCGGraphSpec`, `OCGGraphStatus`, `StorageSpec`, `StorageBackend` (S3/Pvc/Ephemeral/Hybrid), `AutoscalingSpec`, `RebalancerSpec` |
| `operator/src/controller/mod.rs` | `reconcile()`, `error_policy()`, `Context`, finalizer wrapper, rebalance trigger, `run()` |
| `operator/src/controller/configmap.rs` | Level 1: ConfigMap builder (partitions, etcd endpoints, S3 config) |
| `operator/src/controller/services.rs` | Level 1: headless + ClusterIP Service builders |
| `operator/src/controller/statefulset.rs` | Level 1: StatefulSet builder; `scale_down()` for cleanup |
| `operator/src/controller/storage.rs` | Storage volume helpers; PVC, Ephemeral, Hybrid; returns `Vec<PersistentVolumeClaim>` for k8s-openapi 0.22 `volume_claim_templates` |
| `operator/src/controller/upgrade.rs` | Level 2: image-drift detection, rolling upgrade, rollback via annotation `ocg.io/rollback: "true"` |
| `operator/src/controller/backup.rs` | Level 3: CronJob builder (curl health-check loop per partition pod) |
| `operator/src/controller/recovery.rs` | Level 3: StatefulSet ready-replica check → Degraded status + Warning Event |
| `operator/src/controller/hpa.rs` | Level 5: HPA (autoscaling/v2, memory utilization target) |
| `operator/src/controller/pdb.rs` | Level 5: PDB (min_available = ceil(partitions * 0.5)) |
| `operator/src/controller/status.rs` | Status patch; drives metrics set_ready_partitions, set_phase |
| `operator/src/metrics/mod.rs` | Level 4: prometheus-client registry; `reconcile_total`, `reconcile_duration_seconds`, `ocg_graph_ready_partitions`, `ocg_graph_phase`; axum `/metrics` server on port 9090 |
| `operator/deploy/crd.yaml` | Hand-authored CRD manifest (regenerate with `--print-crd`) |
| `operator/deploy/rbac.yaml` | ServiceAccount, ClusterRole, ClusterRoleBinding |
| `operator/deploy/operator-deploy.yaml` | Deployment + metrics Service for the operator |
| `operator/deploy/service-monitor.yaml` | Level 4: ServiceMonitor for Prometheus Operator |
| `operator/deploy/example-ocggraph.yaml` | Example OCGGraph CR (Hybrid storage + autoscaling) |

---

## Key Dependency Decisions

| Dependency | Version | Reason |
|------------|---------|--------|
| `kube` | 0.91 | Latest stable; uses k8s-openapi 0.22 |
| `k8s-openapi` | **0.22** (not 0.23) | Must match kube 0.91 transitive dep; both versions in workspace caused build failure with missing `v1_*` feature |
| `k8s-openapi feature` | `v1_29` | Kubernetes 1.29 minimum API surface |
| `axum` | 0.7 | Metrics HTTP server (Level 4) |
| `prometheus-client` | 0.22 | Metrics exposition |
| `schemars` | 0.8 | JSON Schema generation for CRD |

**k8s-openapi API differences vs. docs:**
- `CrossVersionObjectReference` for HPA is in `k8s_openapi::api::autoscaling::v2`, **not** `apimachinery::pkg::apis::meta::v1`
- `StatefulSetSpec.volume_claim_templates` is `Vec<PersistentVolumeClaim>` (not `PersistentVolumeClaimTemplate`) in k8s-openapi 0.22

---

## Architecture: How Levels Map to Files

```
Level 1 — Install:    configmap.rs + services.rs + statefulset.rs + storage.rs
Level 2 — Upgrade:    upgrade.rs  (image drift + rollback annotation)
Level 3 — Backup:     backup.rs (CronJob) + recovery.rs (Degraded event)
Level 4 — Observability: metrics/mod.rs (prometheus-client + axum /metrics)
Level 5 — Auto-Pilot: hpa.rs + pdb.rs + rebalance trigger in controller/mod.rs
```

All resources are **server-side applied** with field manager `"ocg-operator"`.
All owned resources carry `ownerReferences` → auto-garbage-collected on CR deletion.

---

## Integration Points with Existing Code

The operator orchestrates the `partition_node` binary (from `src/bin/partition_node.rs`).
It does NOT import `nxcypher` lib directly — it manages pods that run it.

- **ConfigMap** injects `PARTITIONS`, `ETCD_ENDPOINTS`, `S3_*` env vars read by `PodConfig`
- **StatefulSet image** = the `partition_node` container image
- **Recovery** watches StatefulSet `ready_replicas` — maps to `RecoveryManager` in `src/distributed/recovery/`
- **Rebalancer trigger** emits a Kubernetes Event; actual rebalance logic lives in `src/distributed/autoscaling/rebalancer.rs`

---

## Upstream Sync Notes

To keep the operator in sync with upstream `kube-rs` changes:

```bash
# Check latest compatible kube + k8s-openapi versions
cargo update -p kube

# Regenerate CRD after any OCGGraphSpec changes
cargo run -p ocg-operator -- --print-crd > operator/deploy/crd.yaml

# Apply updated CRD to cluster
kubectl apply -f operator/deploy/crd.yaml
```

**Watch for:**
- `kube 0.92+` will likely bump to k8s-openapi 0.23/0.24 — update `operator/Cargo.toml` accordingly
- `k8s-openapi 0.23+` may change `StatefulSetSpec.volume_claim_templates` back to `PersistentVolumeClaimTemplate` — check before upgrading
- `prometheus-client 0.23+` changes `EncodeLabelSet` derive requirements

---

## Verification Commands

```bash
# 1. Compile workspace
cargo build --workspace

# 2. Operator-only check
cargo check -p ocg-operator

# 3. Print CRD manifest
cargo run -p ocg-operator -- --print-crd

# 4. Apply to cluster (needs kubectl context)
kubectl apply -f operator/deploy/crd.yaml
kubectl apply -f operator/deploy/rbac.yaml
kubectl apply -f operator/deploy/operator-deploy.yaml
kubectl apply -f operator/deploy/example-ocggraph.yaml

# 5. Watch reconcile
kubectl get ocggraph -w

# 6. Metrics (Level 4)
kubectl port-forward deploy/ocg-operator 9090:9090 -n ocg-system
curl localhost:9090/metrics

# 7. Verify Level 5 resources
kubectl get hpa,pdb -l ocg.io/graph=my-graph
```

---

## Storage-Operations Agent Handover Notes

The operator adds Kubernetes-native storage lifecycle management on top of the existing
Arrow/Parquet/S3 distributed storage stack:

| Existing Layer | Operator Layer |
|---------------|----------------|
| `src/distributed/storage/` — Parquet serialization | PVC volumeClaimTemplates (persistent block storage) |
| `object_store` crate (S3) | S3 StorageSpec → ConfigMap env vars for partition pods |
| In-memory graph (`PropertyGraph` etc.) | Ephemeral volumes for query cache (`/cache` mount) |
| `RecoveryManager` | recovery.rs watches StatefulSet; emits Event on degraded |
| `Rebalancer::plan()` | Level 5 rebalance trigger via Kubernetes Event |

The `StorageBackend::Hybrid` mode gives you persistent Parquet data (PVC `/data`) +
fast ephemeral query cache (emptyDir or ephemeral PVC `/cache`).

---

## Open Items / Next Steps

1. **CRD regeneration**: Run `cargo run -p ocg-operator -- --print-crd` and commit the output to replace the hand-authored `deploy/crd.yaml`
2. **Integration test**: Deploy to a `kind` or `minikube` cluster; apply the example CR
3. **Backup storage target**: The CronJob currently only health-checks pods. The actual Parquet snapshot export (using `object_store`) needs a dedicated backup container image
4. **S3 secret injection**: The `S3Spec.secret_ref` name is put in the ConfigMap but the StatefulSet needs the Secret mounted as env vars — add `envFrom.secretRef` to `statefulset.rs` for S3 backend
5. **Metrics scraping**: Install Prometheus Operator and apply `service-monitor.yaml` to enable metric collection
