# Architecture

This document provides a high-level overview of MFP's internal design. It's intended for contributors and users who want to understand how the protocol works under the hood.

## Design Philosophy

**Library-first, runtime-mediated, symmetric by default.**

MFP is not a client/server system. It's a protocol for peer-to-peer communication where every agent is treated equally. The runtime acts as a neutral mediator, enforcing security constraints without imposing workflow hierarchy.

**Key principles:**

1. **No trust in LLMs** — Cryptographic validation happens before message delivery, not after
2. **Symmetric peers** — All agents use the same API and have identical capabilities
3. **Deterministic enforcement** — Security constraints are mathematical, not policy-based
4. **Library-first** — The runtime embeds in any Python process; the standalone server is a thin wrapper

---

## Layered Architecture

MFP is organized into six layers, from lowest to highest abstraction:

```
┌──────────────────────────────────────────────────────────┐
│                     Application                          │  ← User code
│           (Agent callables, business logic)              │
└──────────────────────────────────────────────────────────┘
                          │
                          ▼
┌──────────────────────────────────────────────────────────┐
│                   Agent Lifecycle                        │  ← bind/unbind, tools
│     (identity.py, lifecycle.py, tools.py)                │
└──────────────────────────────────────────────────────────┘
                          │
                          ▼
┌──────────────────────────────────────────────────────────┐
│                       Runtime                            │  ← Pipeline, quarantine
│    (runtime.py, pipeline.py, channels.py, quarantine.py) │
└──────────────────────────────────────────────────────────┘
                          │
            ┌─────────────┼─────────────┐
            ▼             ▼             ▼
┌──────────────┐  ┌─────────────┐  ┌──────────────┐
│  Core        │  │  Storage    │  │ Observability│  ← Production
│ (crypto,     │  │ (SQLite,    │  │ (logging,    │     hardening
│  frames,     │  │  schema,    │  │  metrics,    │
│  ratchet,    │  │  crypto)    │  │  health,     │
│  types)      │  │             │  │  circuit     │
│              │  │             │  │  breakers)   │
└──────────────┘  └─────────────┘  └──────────────┘
            │             │             │
            └─────────────┴─────────────┘
                          ▼
            ┌─────────────────────────────┐
            │        Federation           │  ← Cross-runtime
            │  (bilateral, transport,     │     communication
            │   recovery, wire protocol)  │
            └─────────────────────────────┘
```

---

## Layer 1: Core

**Modules:** `mfp/core/`

The foundation layer providing cryptographic primitives and frame construction.

### `primitives.py`

Low-level cryptographic operations:
- **AES-256-GCM** — AEAD encryption/decryption (default algorithm)
- **HMAC-SHA256** — Message authentication
- **X25519** — Diffie-Hellman key exchange
- **SHA-256** — Hashing and key derivation

All primitives use the `cryptography` library.

### `frame.py`

Frame construction and validation:
- **build_frame()** — Constructs symmetric mirror frame (`2k` blocks)
- **validate_frame()** — Verifies frame integrity and temporal validity
- **strip_frame()** — Removes validated frame, returns payload

Frames are deterministic functions of `(step, local_state, global_state)`.

### `ratchet.py`

State evolution:
- **advance_ratchet()** — One-way state chain: `S_n = f(S_{n-1}, frame_n)`
- **derive_frame_seed()** — Generates cryptographic seed from ratchet state

Uses HMAC-based ratcheting for forward secrecy.

### `encoding.py`

Payload transformation:
- **encode()** — Encrypts plaintext with AES-256-GCM
- **decode()** — Decrypts ciphertext, verifies authentication tag

### `validation.py`

Input validation helpers for configuration and runtime parameters.

### `types.py`

Core type definitions:
- `AgentId`, `ChannelId`, `MessageId` (identifier wrappers)
- `AgentState` (UNREGISTERED, BINDING, BOUND, ACTIVE, QUARANTINED, TERMINATED)
- `StateValue` (ratchet state, 32 bytes)
- `DeliveredMessage`, `Receipt`, `ProtocolMessage`
- Error types and enums

---

## Layer 2: Storage

**Modules:** `mfp/storage/`

Persistent state management via SQLite.

### `engine.py`

Storage interface:
- **create_agent()** / **delete_agent()** — Agent lifecycle
- **create_channel()** / **get_channel()** — Channel records
- **enqueue_message()** / **dequeue_message()** — Message queuing
- **update_ratchet()** — Ratchet state persistence

All operations are transactional.

### `schema.py`

SQLite schema definitions:
- `agents` — Agent identity, status, quarantine flags
- `channels` — Channel metadata, peer mappings, ratchet state
- `messages` — Pending message queue (FIFO)
- `ratchets` — Local and global ratchet states

### `crypto.py`

At-rest encryption (optional):
- **encrypt_value()** / **decrypt_value()** — Database field encryption
- Uses AES-256-GCM with a master key from `storage.master_key_file`

---

## Layer 3: Runtime

**Modules:** `mfp/runtime/`

Execution orchestration and security enforcement.

### `runtime.py`

Central coordinator:
- **Runtime.bind()** — Register agent, assign `AgentId`
- **Runtime.unbind()** — Deregister agent, close channels
- **Runtime.deliver()** — Route message to agent callable
- **Runtime.shutdown()** — Graceful teardown

Holds the `StorageEngine` and global ratchet state.

### `pipeline.py`

Six-stage message pipeline (ACCEPT, FRAME, ENCODE, VALIDATE, DECODE, DELIVER):
- Validates send requests and payload size limits
- Samples frames, encrypts/decrypts, and validates frame pairs
- Invokes agent callable with `DeliveredMessage`
- Supports both sync and async pipelines (`process_message` / `process_message_async`)
- Timeout protection on agent callable execution

### `channels.py`

Channel management:
- **establish_channel()** — Create new channel, initialize ratchet
- **close_channel()** — Tear down channel, flush pending messages
- **derive_channel_id()** — Deterministic ID from `(agent_a, agent_b, nonce)`

### `quarantine.py`

Security isolation:
- **check_rate_limit()** — Enforce `max_message_rate`
- **check_payload_size()** — Enforce `max_payload_size`
- **trigger_quarantine()** — Transition agent to `QUARANTINED` state

Quarantined agents cannot send/receive messages until administratively reset.

---

## Layer 4: Agent Lifecycle

**Modules:** `mfp/agent/`

User-facing API for agent operations.

### `lifecycle.py`

Agent binding and handle management:
- **bind()** — Register agent callable with runtime, return `AgentHandle`
- **unbind()** — Deregister agent
- **AgentHandle.send()** — Send message on channel
- **AgentHandle.channels()** — List visible channels
- **AgentHandle.status()** — Query own lifecycle status

### `identity.py`

Agent identity derivation:
- **derive_agent_id()** — Deterministic ID from `(runtime_id, agent_callable)`
- Uses HMAC-SHA256 for stable, unique identifiers

### `tools.py`

Protocol tools (agent-facing API):
- **mfp_send()** — Send encrypted message on channel
- **mfp_channels()** — List all channels for agent
- **mfp_status()** — Query agent status (state, channel count, pending messages)

These are the primary functions LLM agents invoke.

---

## Layer 5: Federation

**Modules:** `mfp/federation/`

Cross-runtime communication and recovery.

### `bilateral.py`

Bilateral channel establishment:
- **bootstrap_deterministic()** — Derive shared key from runtime IDs
- **bootstrap_ceremonial()** — X25519 Diffie-Hellman key exchange
- **BilateralChannel** — Cross-runtime channel abstraction

### `transport.py`

TCP message transport:
- **TransportServer** — Listen for incoming envelopes
- **TransportClient** — Send envelopes to remote runtime
- Asynchronous I/O with configurable timeouts

### `wire.py`

Wire protocol:
- **build_envelope_header()** — Frame + routing metadata
- **validate_envelope()** — Verify integrity, decrypt payload
- Uses the same frame/ratchet mechanism as local channels

### `recovery.py`

State synchronization:
- **detect_divergence()** — Identify ratchet state mismatch
- **negotiate_recovery()** — Exchange state proofs, find common ancestor
- **resync()** — Fast-forward ratchet to converge

Prevents federation from stalling due to dropped messages.

---

## Message Flow

### Local Send (Alice → Bob, same runtime)

1. **Application** calls `mfp_send(alice_handle, channel_id, plaintext)`
2. **Agent tools** validate handle state, lookup channel
3. **Runtime pipeline** retrieves ratchet state from storage
4. **Core frame** builds mirror frame from `(step, local_state, global_state)`
5. **Core encoding** encrypts plaintext with channel key
6. **Core frame** wraps payload in frame: `[open | ciphertext | close]`
7. **Storage** persists message to Bob's queue, advances ratchet
8. **Runtime** delivers to Bob's callable: `bob(channel_id, plaintext)`
9. **Return value** captured for logging/hooks

### Federated Send (Alice → Charlie, remote runtime)

1. **Application** calls `mfp_send(alice_handle, bilateral_channel_id, plaintext)`
2. **Agent tools** identify channel as bilateral (cross-runtime)
3. **Federation bilateral** retrieves shared federation key
4. **Core frame** builds frame using bilateral ratchet state
5. **Core encoding** encrypts plaintext
6. **Federation wire** constructs envelope with routing header
7. **Federation transport** sends envelope over TCP to remote runtime
8. **Remote runtime** receives envelope, validates frame
9. **Remote runtime** delivers to Charlie's callable
10. **Recovery** monitors for sequence gaps, triggers resync if needed

---

## Security Model

### Threat Mitigation

**Frame validation prevents:**
- Prompt injection (invalid frames rejected before payload delivery)
- Replay attacks (temporal ratchet invalidates old frames)
- Forgery (frames cryptographically bound to ratchet state)

**Encryption provides:**
- Confidentiality (AES-256-GCM AEAD)
- Authenticity (HMAC verification)
- Forward secrecy (ratchet never reverses)

**Quarantine protects against:**
- Malicious agents (rate/size limits)
- Corrupted state (validation failure thresholds)

### Non-Goals

MFP does **not** protect against:
- Compromised runtime (trusted computing base)
- Side-channel attacks (timing, memory access patterns)
- Physical access to storage (at-rest encryption is optional)

See [Security Model](security.md) for full threat analysis.

---

## Design Decisions

### Why SQLite?

- **Transactional:** ACID guarantees for ratchet state
- **Embedded:** No external database server
- **Simple:** Single file, easy backup/restore
- **WAL mode:** Concurrent reads during writes

### Why Symmetric Frames?

- **Structural validation:** LLMs can't bypass mathematical constraints
- **Deterministic:** No policy decisions, no edge cases
- **Efficient:** O(k) validation, constant-size state

### Why Ratchet Instead of Logs?

- **Bounded state:** Fixed 32 bytes per channel, regardless of history
- **Forward secrecy:** Old states can't be recovered
- **Federation-friendly:** Only current state needs synchronization

### Why Library-First?

- **Composable:** Embed in any application
- **Testable:** Pure functions, no global state
- **Portable:** No daemon dependencies

---

## Testing Strategy

**604 tests across 6 test modules:**

- **Unit tests** — Core primitives, frame logic, ratchet evolution
- **Integration tests** — Runtime + storage + agents
- **E2E tests** — Full message flows, quarantine triggers, federation
- **Property tests** — Frame symmetry, ratchet monotonicity

Coverage: ~95% (excluding server.py boilerplate).

---

## Performance Characteristics

**Benchmarks (single runtime, 2 agents, 10k messages):**

- Frame construction: ~50 µs/message
- Encryption/decryption: ~20 µs/message
- SQLite write: ~100 µs/message
- End-to-end latency: ~200 µs/message

**Federation overhead:**

- TCP transport: +5-50ms (network latency)
- Recovery protocol: ~100ms (worst case, 3 round-trips)

**Memory:**

- Runtime: ~5 MB baseline
- Per-agent overhead: ~1 KB (ratchet state + metadata)
- Per-message queue: ~1 KB/message

---

## Extension Points

### Custom Agent Types

Implement the agent callable signature (sync or async):

```python
from mfp.core.types import DeliveredMessage

def agent(msg: DeliveredMessage) -> None:
    # Your logic here
    print(msg.payload.decode())

# Or async:
async def async_agent(msg: DeliveredMessage) -> None:
    await process(msg.payload)
```

### Lifecycle Hooks

Enable hooks in `RuntimeConfig(enable_hooks=True)`:

- `before_send` — Pre-validation hook
- `after_send` — Post-delivery hook
- `on_quarantine` — Isolation event

### Storage Backends

Subclass `StorageEngine` to use alternative backends (Postgres, DynamoDB, etc.).

### Transport Protocols

Subclass `TransportServer` for non-TCP transports (WebSocket, QUIC, etc.).

---

## See Also

- [API Reference](api-reference.md) — public interface
- [Server Guide](server-guide.md) — standalone server setup
- [Security Model](security.md) — threat analysis
- [Contributing](contributing.md) — development guidelines
- [MFP Specification](https://github.com/Madahub-dev/mfp-spec) — protocol design specs

---

## Layer 6: Observability & Performance (v0.2.0+)

**Modules:** `mfp/observability/`, `mfp/core/merkle.py`, `mfp/federation/rotation.py`, `mfp/runtime/deduplication.py`

Production hardening features for monitoring, resilience, and performance.

### Observability (`mfp/observability/`)

**logging.py** — Structured logging:
- LogContext with correlation IDs
- JSON/text format support
- Audit events for security operations

**health.py** — Health check endpoints:
- `/health/live` — Liveness probe (K8s)
- `/health/ready` — Readiness probe (K8s)
- `/health/startup` — Startup probe (K8s)

**metrics.py** — Prometheus metrics:
- Counters: messages, failures, quarantine events
- Gauges: active channels, agents, queue depth
- Histograms: pipeline duration, Sg computation time

**circuit_breaker.py** — Resilience pattern:
- 3-state FSM (CLOSED/OPEN/HALF_OPEN)
- Prevents cascading failures
- Used for storage and bilateral channels

**timeout.py** — Time limits:
- Thread-based timeout enforcement
- Agent, pipeline, storage timeouts
- Automatic quarantine on timeout

**http_server.py** — HTTP server for health and metrics endpoints.

### Performance Optimizations

**merkle.py** — Incremental Sg (⚠️ Breaking):
- Merkle tree for O(log N) Sg updates
- 10x faster than v0.1.x for 1000+ channels
- <1ms Sg computation

**frame.py** (enhanced) — Frame caching:
- LRU cache for deterministic frames
- 16-17x speedup for cache hits
- 90% hit rate in recovery scenarios

**transport.py** (enhanced) — Connection pooling:
- TCP connection reuse
- Idle eviction (5 min), lifetime limits (1 hour)
- Background cleanup task

### Security Hardening

**rotation.py** — Key rotation:
- X25519 DH-based rekey protocol
- Triggers: message count (1M), time (24h), manual
- Forward secrecy with ephemeral keypairs

**deduplication.py** — Replay protection:
- Per-channel message ID tracking
- O(1) duplicate detection
- Sliding window + TTL eviction

---

## Production Features Summary

| Feature | Module | Benefit |
|---------|--------|---------|
| Structured Logging | observability/logging.py | Request tracing, audit trail |
| Health Checks | observability/health.py | K8s integration, uptime monitoring |
| Metrics | observability/metrics.py | Performance monitoring, alerting |
| Circuit Breakers | observability/circuit_breaker.py | Resilience, failure isolation |
| Timeouts | observability/timeout.py | Prevents blocking, auto-quarantine |
| Merkle Tree | core/merkle.py | 10x faster Sg (O(log N)) |
| Frame Caching | core/frame.py | 16x speedup for recovery |
| Connection Pool | federation/transport.py | Reduced handshake overhead |
| Key Rotation | federation/rotation.py | Forward secrecy, key fatigue mitigation |
| Deduplication | runtime/deduplication.py | Replay attack prevention |

**Test Coverage:** 813 tests (591 unit, 191 integration, 27 E2E, 4 benchmark)

See [production-guide.md](./production-guide.md) for deployment details.

