# Bugs And Gaps

## Open

1. Fair scheduling maturity is improved, but weighted fairness is still open
- Implemented: rotating stream order + per-stream batch cap + streams-per-cycle cap.
- Implemented: starvation/pending/in-flight/saturation observability metrics.
- Remaining: weighted fair sharing policy (e.g., per-agent weights/quotas).
- Impact: strong baseline isolation, but not yet policy-weighted scheduling.

2. Retry delay is worker-sleep based (not durable scheduled delivery)
- Retry delay currently sleeps in worker then re-enqueues.
- If process exits during sleep, delayed retry is lost.
- Impact: weaker retry durability under crashes.

3. Version conflict outcomes are metriced but not persisted in execution history
- Strict conflicts are counted in metrics and logged.
- No persistent execution record reason code yet.
- Impact: harder postmortem/audit for compliance workflows.

4. Cross-pod fairness/coordination strategy not formalized
- Leader election exists for watchers, but fairness across many worker pods is not policy-driven.
- Impact: enterprise scale questions remain around predictable throughput isolation.

5. Backpressure policy weighting is basic
- Implemented: dispatch-time queue fullness checks, priority bypass, and overflow modes (`drop`, `defer`, `dlq`).
- Remaining: weighted admission across tenants/agents and dynamic per-stream quotas.
- Impact: robust baseline protection exists, but no tenant-aware fairness contracts yet.

6. Determinism formalization is mostly complete, with one remaining hardening item
- Implemented: persisted execution `status`, `lifecycle_state`, `reason`, and `written` fields.
- Implemented: optional strict document-hash guard (`require_document_hash_match`).
- Remaining: explicit lifecycle transition logging from `dispatched -> running -> terminal` (currently terminal state is persisted at completion).
- Impact: strong auditability for final outcome; limited visibility for in-flight stage transitions.

## Addressed In This Pass

1. Strict stale-write guard
- `strict_post_commit` now prevents stale writes using `_mongoclaw_version` optimistic filter.

2. Version stamping
- Strict writes increment `_mongoclaw_version` atomically.

3. Observability
- Added `mongoclaw_version_conflicts_total` and `mongoclaw_retries_scheduled_total`.

4. Retry storm mitigation
- Worker now applies configured retry delay before re-enqueue.

5. Worker fairness controls
- Added env/config-driven fair scheduling controls in worker loop.

6. Dispatch backpressure admission controls
- Added env/config-driven queue-pressure admission with metrics and overflow policies.

7. Execution lifecycle persistence
- Added terminal lifecycle/reason persistence to execution history collection and SDK/API exposure.

8. Horizontal scaling semantics baseline implemented; advanced topology strategy still open
- Implemented: env-driven routing strategy selection and partition count.
- Implemented: explicit at-least-once delivery semantics metadata on dispatched work items.
- Implemented: replay-delivery and routing counters for contention/scale visibility.
- Remaining: multi-region routing policy, tenant-aware partition affinity, and formal exactly-once strategy (currently at-least-once + idempotent write path).

9. Failure isolation baseline implemented; distributed/global budgets remain open
- Implemented: in-process per-agent failure budgets with temporary quarantine.
- Implemented: per-agent quarantine and latency-SLO violation metrics.
- Remaining: cross-process shared budget/quarantine state (current state is process-local), automated alerting hooks, and full resilience benchmark automation.

10. Strict-mode conflict rate under concurrent CRUD can suppress enrichment coverage
- Evidence from live benchmark: high `strict_version_conflict` counts with low write coverage in both baseline and burst scenarios.
- Impact: strict consistency favors correctness but can under-deliver enrichment in high-churn workloads.
- Follow-up: add conflict replay policy and/or per-use-case consistency presets.

11. Prompt template robustness gaps under sparse event documents
- Evidence from live benchmark: `PromptRenderError` when prompt required `document._id`.
- Impact: avoidable pipeline failures under certain update event shapes.
- Follow-up: null-safe prompt guidelines + template lint rules.

12. Missing-agent failures were retried as generic pipeline errors (addressed)
- Evidence: repeated execution errors `Agent 'bench_baseline_agent' not found` / `Agent 'bench_burst_agent' not found` during replay of stale work.
- Root cause: `Executor.execute()` caught `AgentNotFoundError` in broad `except Exception` path, marking reason `pipeline_error`; worker then treated as retryable failure.
- Fix: added explicit `except AgentNotFoundError` and `except AgentDisabledError` branches in executor with reason codes (`agent_not_found`, `agent_disabled`) and re-raise so worker acks without retry.
- Impact: prevents retry amplification for deleted/disabled agents and keeps failure reasons accurate.
