---
type: skill
domain: architecture
links: [Corpus_Callosum/CONSTITUTION.md, PULSE_OS.md]
---

# KAFKA_MAGE.md
> **ROLE:** DISTRIBUTED SYSTEMS ARCHITECT
> **TRIGGER:** "SCALE UP" / "EVENT STREAM" / "PARTITION"

## Step-by-Step Protocol
1. **Audit current throughput** — measure messages/sec, p99 latency, consumer lag with `brain_query.py --query "event throughput bottlenecks"`.
2. **Design partition strategy** — hash on agent_id for ordering guarantees; use round-robin only for stateless consumers. Target: <=12 partitions per topic for manageability.
3. **Configure consumer groups** — one group per concern (capture, bridge, dedup). Set `max.poll.records=500`, `session.timeout.ms=30000`. Never mix fast and slow consumers in one group.
4. **Wire exactly-once semantics** — enable `enable.idempotence=true` on producers, use transactional APIs (`initTransactions`, `beginTransaction`, `commitTransaction`). Set `isolation.level=read_committed` on consumers.
5. **Load test and verify** — push 10x expected peak, confirm zero message loss, consumer lag <1000, p99 <50ms.

## Metrics & Thresholds
- **Producer throughput:** >=10,000 msgs/sec per partition
- **Consumer lag:** <1,000 messages (alert at 5,000, page at 10,000)
- **End-to-end latency:** p50 <10ms, p99 <50ms, p999 <200ms
- **Partition count:** 3-12 per topic (never >50 — rebalancing kills perf)
- **Replication factor:** minimum 3 for durability

## Good Example
```python
# Exactly-once producer with idempotence
from kafka import KafkaProducer
producer = KafkaProducer(
    bootstrap_servers="localhost:9092",
    acks="all",                    # wait for all replicas
    enable_idempotence=True,       # dedup at broker level
    max_in_flight_requests_per_connection=5,
    retries=2147483647,            # infinite retries (idempotent)
    key_serializer=lambda k: k.encode("utf-8"),
    value_serializer=lambda v: json.dumps(v).encode("utf-8"),
)
producer.send("pulse.events", key=agent_id, value=event)
```

## Anti-Patterns
1. **Random partitioning for ordered data** — destroys per-agent ordering; always hash on agent_id.
2. **One giant consumer group** — slow consumers block fast ones; separate by processing speed.
3. **Polling without backpressure** — unbounded queues cause OOM; cap in-memory buffers at 10,000 msgs.

## Tools to Use
- `python pulse/brain/brain_query.py --query "event streaming"` — check prior streaming decisions
- `python pulse/tasks/pulse_relay.py --tasks "Partition Audit:architecture:standard"` — delegate partition review
- `python pulse/brain/pulse_save.py --agent kafka_mage --learnings "what you learned"` — persist findings

## Verification Checklist
- [ ] Consumer lag stays <1,000 under peak load
- [ ] Exactly-once enabled: idempotent producer + transactional consumer
- [ ] No single consumer group mixes fast/slow processors
- [ ] Partition key ensures ordering where required (agent_id, session_id)
- [ ] Replication factor >=3, acks=all on all producers
