# Complete Synqed Production Inbox Deployment Guide

**Everything you need to deploy Synqed's production-grade inbox in one file.**

---

## Table of Contents

1. [Quick Deploy (5 minutes)](#quick-deploy-5-minutes)
2. [What's New in Production Version](#whats-new-in-production-version)
3. [Prerequisites](#prerequisites)
4. [Step-by-Step Installation](#step-by-step-installation)
5. [Configuration](#configuration)
6. [Starting the System](#starting-the-system)
7. [Testing](#testing)
8. [Production Deployment](#production-deployment)
9. [Troubleshooting](#troubleshooting)
10. [Complete Code Examples](#complete-code-examples)

---

## Quick Deploy (5 minutes)

```bash
# 1. Install Redis
brew install redis && brew services start redis

# 2. Install Synqed
pip install synqed cryptography redis httpx uvicorn

# 3. Create app
cat > main.py << 'EOF'
from fastapi import FastAPI
from synqed.agent_email.inbox import router
from synqed.agent_email.inbox.startup import create_lifespan

app = FastAPI(lifespan=create_lifespan("redis://localhost:6379"))
app.include_router(router)

@app.get("/health")
async def health():
    return {"status": "healthy"}
EOF

# 4. Initialize agents
python3 << 'EOF'
import asyncio
from synqed.agent_email.registry.api import get_registry
from synqed.agent_email.registry.models import AgentRegistryEntry
from synqed.agent_email.inbox import generate_keypair
import json

async def setup():
    registry = get_registry()
    private_key, public_key = generate_keypair()
    
    registry.register(AgentRegistryEntry(
        agent_id="agent://demo/alice",
        email_like="alice@demo",
        inbox_url="http://localhost:8000/v1/a2a/inbox",
        public_key=public_key,
    ))
    
    with open('keypairs.json', 'w') as f:
        json.dump({"agent://demo/alice": {"private_key": private_key, "public_key": public_key}}, f)
    
    print("✓ Agent registered")

asyncio.run(setup())
EOF

# 5. Kill any process on port 8000 and start
lsof -ti :8000 | xargs kill -9 2>/dev/null; python main.py
```

Done! Visit http://localhost:8000/docs

---

## What's New in Production Version

| Feature | Before | After |
|---------|--------|-------|
| **Identity** | None | Ed25519 signatures (required) |
| **Delivery** | Synchronous | Async queue with retry |
| **Rate Limiting** | None | 100/min sender, 500/min IP |
| **Tracing** | None | Distributed trace_id |
| **Reliability** | Best effort | Guaranteed + DLQ |
| **Routing** | Local only | Local + remote HTTP |
| **Dependencies** | None | Redis + cryptography |

---

## Prerequisites

### Required Software

**macOS:**
```bash
# Install Homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install Redis
brew install redis
brew services start redis

# Verify
redis-cli ping  # Should return: PONG
```

**Linux (Ubuntu/Debian):**
```bash
# Install Redis
sudo apt update
sudo apt install redis-server
sudo systemctl start redis
sudo systemctl enable redis

# Verify
redis-cli ping
```

**Using Docker (any OS):**
```bash
docker run -d -p 6379:6379 --name redis redis:7-alpine
```

### Python Requirements

- Python 3.10+
- pip or conda

---

## Step-by-Step Installation

### 1. Install Python Dependencies

```bash
# Create virtual environment (recommended)
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install synqed
pip install synqed cryptography redis httpx uvicorn fastapi

# Verify installation
python -c "from synqed.agent_email.inbox import generate_keypair; print('✓ Synqed installed')"
```

### 2. Create Application File

Create `main.py`:

```python
"""
Production Synqed Inbox Application
"""
import os
from fastapi import FastAPI
from synqed.agent_email.inbox import router
from synqed.agent_email.inbox.startup import create_lifespan

# Configuration
REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379")

# Create FastAPI app with lifespan management
app = FastAPI(
    title="Synqed Agent Email System",
    version="2.0.0",
    description="Production-grade A2A inbox with guaranteed delivery",
    lifespan=create_lifespan(redis_url=REDIS_URL),
)

# Include inbox router
app.include_router(router)

# Health check endpoint
@app.get("/health")
async def health():
    """Health check endpoint for monitoring."""
    return {
        "status": "healthy",
        "version": "2.0.0",
        "redis": REDIS_URL,
    }

# Root endpoint
@app.get("/")
async def root():
    """Root endpoint with service information."""
    return {
        "service": "Synqed Agent Email System",
        "version": "2.0.0",
        "endpoints": {
            "docs": "/docs",
            "health": "/health",
            "inbox": "/v1/a2a/inbox",
        }
    }

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000, log_level="info")
```

### 3. Create Environment Configuration

Create `.env`:

```bash
# Redis Configuration
REDIS_URL=redis://localhost:6379

# Rate Limiting
SENDER_RATE_LIMIT=100
IP_RATE_LIMIT=500

# Queue Configuration
MAX_RETRIES=5
INITIAL_BACKOFF_MS=100
MAX_BACKOFF_MS=30000

# HTTP Configuration
HTTP_TIMEOUT=30.0

# Logging
LOG_LEVEL=INFO
```

### 4. Initialize Agents with Keypairs

Create `init_agents.py`:

```python
"""
Initialize agents with Ed25519 keypairs.
"""
import asyncio
import json
from synqed.agent_email.registry.api import get_registry
from synqed.agent_email.registry.models import AgentRegistryEntry
from synqed.agent_email.inbox import generate_keypair

async def main():
    """Initialize demo agents."""
    print("Initializing agents with keypairs...")
    
    registry = get_registry()
    keypairs = {}
    
    # Define your agents
    agents = [
        {
            "agent_id": "agent://demo/alice",
            "email_like": "alice@demo",
            "inbox_url": "http://localhost:8000/v1/a2a/inbox",
            "capabilities": ["a2a/1.0", "chat"],
            "metadata": {"description": "Demo agent Alice"},
        },
        {
            "agent_id": "agent://demo/bob",
            "email_like": "bob@demo",
            "inbox_url": "http://localhost:8000/v1/a2a/inbox",
            "capabilities": ["a2a/1.0", "chat"],
            "metadata": {"description": "Demo agent Bob"},
        },
    ]
    
    # Generate keypairs and register
    for agent in agents:
        # Generate Ed25519 keypair
        private_key, public_key = generate_keypair()
        
        # Register agent with public key
        registry.register(AgentRegistryEntry(
            agent_id=agent["agent_id"],
            email_like=agent["email_like"],
            inbox_url=agent["inbox_url"],
            public_key=public_key,  # Required!
            capabilities=agent.get("capabilities", []),
            metadata=agent.get("metadata", {}),
        ))
        
        # Store keypair for sending messages
        keypairs[agent["agent_id"]] = {
            "private_key": private_key,
            "public_key": public_key,
        }
        
        print(f"✓ Registered: {agent['agent_id']}")
    
    # Save keypairs to file
    with open("keypairs.json", "w") as f:
        json.dump(keypairs, f, indent=2)
    
    print(f"\n✓ Registered {len(agents)} agents")
    print("⚠️  Keypairs saved to keypairs.json - store securely in production!")
    print("\nIn production, use a secrets manager (AWS Secrets Manager, etc.)")

if __name__ == "__main__":
    asyncio.run(main())
```

Run initialization:

```bash
python init_agents.py
```

---

## Configuration

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `REDIS_URL` | `redis://localhost:6379` | Redis connection URL |
| `SENDER_RATE_LIMIT` | `100` | Max requests per sender per minute |
| `IP_RATE_LIMIT` | `500` | Max requests per IP per minute |
| `MAX_RETRIES` | `5` | Max retry attempts before DLQ |
| `INITIAL_BACKOFF_MS` | `100` | Initial retry backoff (milliseconds) |
| `MAX_BACKOFF_MS` | `30000` | Maximum retry backoff (milliseconds) |
| `HTTP_TIMEOUT` | `30.0` | Timeout for remote forwarding (seconds) |
| `LOG_LEVEL` | `INFO` | Logging level |

### Redis Configuration

**Development (local):**
```bash
redis://localhost:6379
```

**Production with AUTH:**
```bash
redis://:password@host:6379
```

**Redis Cluster:**
```bash
redis://node1:6379,node2:6379,node3:6379
```

**Redis Sentinel:**
```bash
redis://sentinel1:26379,sentinel2:26379/mymaster
```

---

## Starting the System

### Development (Single Process)

```bash
# Kill anything on port 8000
lsof -ti :8000 | xargs kill -9 2>/dev/null

# Start server
python main.py
```

### Production (Multiple Workers)

```bash
# Using uvicorn
uvicorn main:app --host 0.0.0.0 --port 8000 --workers 4

# Using gunicorn
gunicorn main:app \
  --workers 4 \
  --worker-class uvicorn.workers.UvicornWorker \
  --bind 0.0.0.0:8000 \
  --timeout 60
```

### Background Service (systemd - Linux only)

Create `/etc/systemd/system/synqed.service`:

```ini
[Unit]
Description=Synqed Agent Email Service
After=network.target redis.service

[Service]
Type=notify
User=synqed
Group=synqed
WorkingDirectory=/opt/synqed
Environment="REDIS_URL=redis://localhost:6379"
Environment="LOG_LEVEL=INFO"
ExecStart=/opt/synqed/venv/bin/uvicorn main:app \
  --host 0.0.0.0 \
  --port 8000 \
  --workers 4
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
```

Enable and start:

```bash
sudo systemctl daemon-reload
sudo systemctl enable synqed
sudo systemctl start synqed
sudo systemctl status synqed
```

### Docker

Create `Dockerfile`:

```dockerfile
FROM python:3.11-slim

WORKDIR /app

# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application
COPY main.py .
COPY init_agents.py .
COPY .env .

# Create non-root user
RUN useradd -m -u 1000 synqed && chown -R synqed:synqed /app
USER synqed

EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
```

Create `docker-compose.yml`:

```yaml
version: '3.8'

services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
    command: redis-server --appendonly yes
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s

  synqed:
    build: .
    ports:
      - "8000:8000"
    environment:
      - REDIS_URL=redis://redis:6379
    depends_on:
      redis:
        condition: service_healthy
    restart: unless-stopped

volumes:
  redis-data:
```

Deploy:

```bash
docker-compose up -d
docker-compose logs -f synqed
```

---

## Testing

### 1. Health Check

```bash
curl http://localhost:8000/health
```

Expected response:
```json
{
  "status": "healthy",
  "version": "2.0.0",
  "redis": "redis://localhost:6379"
}
```

### 2. View API Documentation

Open browser: http://localhost:8000/docs

### 3. Send Test Message

Create `test_send.py`:

```python
"""
Test sending a signed message.
"""
import asyncio
import json
import httpx
from synqed.agent_email.inbox import sign_message

async def main():
    # Load keypair
    with open('keypairs.json', 'r') as f:
        keypairs = json.load(f)
    
    alice_keys = keypairs["agent://demo/alice"]
    
    # Create message
    message = {
        "thread_id": "test-thread-123",
        "role": "user",
        "content": "Hello from Alice!",
    }
    
    # Sign message
    signature = sign_message(
        private_key_b64=alice_keys["private_key"],
        sender="agent://demo/alice",
        recipient="agent://demo/bob",
        message=message,
        thread_id="test-thread-123",
    )
    
    # Send envelope
    envelope = {
        "sender": "agent://demo/alice",
        "recipient": "agent://demo/bob",
        "message": message,
        "signature": signature,
    }
    
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "http://localhost:8000/v1/a2a/inbox",
            json=envelope,
            timeout=30.0,
        )
        
        result = response.json()
        print("Response:", json.dumps(result, indent=2))
        
        if result["status"] == "accepted":
            print(f"\n✓ Message accepted!")
            print(f"  Message ID: {result['message_id']}")
            print(f"  Trace ID: {result['trace_id']}")
        else:
            print(f"\n✗ Message failed: {result.get('error')}")

if __name__ == "__main__":
    asyncio.run(main())
```

Run test:

```bash
python test_send.py
```

### 4. Monitor Queue

Create `monitor.py`:

```python
"""
Monitor queue health.
"""
import asyncio
from synqed.agent_email.inbox import get_message_queue
from synqed.agent_email.registry.api import get_registry

async def main():
    queue = get_message_queue()
    await queue.connect()
    
    registry = get_registry()
    
    print("Queue Status:")
    print("-" * 60)
    
    for entry in registry.list_all():
        pending = await queue.get_queue_length(entry.agent_id)
        dlq = await queue.get_dlq_length(entry.agent_id)
        
        print(f"\n{entry.agent_id}")
        print(f"  Pending: {pending}")
        print(f"  DLQ: {dlq}")
    
    await queue.close()

if __name__ == "__main__":
    asyncio.run(main())
```

Run monitor:

```bash
python monitor.py
```

---

## Production Deployment

### Pre-Deployment Checklist

- [ ] Redis running with persistence
- [ ] Redis secured with AUTH password
- [ ] TLS enabled for Redis connections
- [ ] Keypairs stored in secrets manager (not files)
- [ ] HTTPS enforced for all inbox URLs
- [ ] Rate limits configured appropriately
- [ ] Logging configured (no sensitive data)
- [ ] Monitoring and alerting setup
- [ ] Backup and disaster recovery tested
- [ ] Load testing completed

### Security Hardening

**1. Redis Security:**

```bash
# Edit redis.conf
requirepass YOUR_STRONG_PASSWORD
bind 127.0.0.1
protected-mode yes
```

Update `.env`:
```bash
REDIS_URL=redis://:YOUR_STRONG_PASSWORD@localhost:6379
```

**2. Key Management:**

Use a secrets manager instead of `keypairs.json`:

```python
# AWS Secrets Manager example
import boto3
import json

def get_private_key(agent_id):
    client = boto3.client('secretsmanager')
    response = client.get_secret_value(SecretId=f'synqed/{agent_id}/private_key')
    return json.loads(response['SecretString'])['private_key']
```

**3. HTTPS Only:**

```python
# Force HTTPS in production
from fastapi.middleware.httpsredirect import HTTPSRedirectMiddleware

if os.getenv("ENVIRONMENT") == "production":
    app.add_middleware(HTTPSRedirectMiddleware)
```

**4. Run as Non-Root:**

```bash
# Create user
sudo useradd -m -s /bin/bash synqed

# Set permissions
sudo chown -R synqed:synqed /opt/synqed

# Run as synqed user
sudo -u synqed uvicorn main:app --host 0.0.0.0 --port 8000
```

### Monitoring

**Prometheus Metrics:**

```python
from prometheus_client import Counter, Histogram, generate_latest
from fastapi import Response

# Metrics
messages_total = Counter('inbox_messages_total', 'Total messages')
processing_duration = Histogram('inbox_processing_seconds', 'Processing duration')

@app.get("/metrics")
async def metrics():
    return Response(content=generate_latest(), media_type="text/plain")
```

**Health Checks:**

```python
@app.get("/health/live")
async def liveness():
    """Kubernetes liveness probe."""
    return {"status": "alive"}

@app.get("/health/ready")
async def readiness():
    """Kubernetes readiness probe."""
    # Check Redis connectivity
    try:
        queue = get_message_queue()
        await queue._redis.ping()
        return {"status": "ready"}
    except:
        return {"status": "not ready"}, 503
```

---

## Troubleshooting

### Issue: Messages not processing

**Symptoms:** Messages queued but not delivered

**Diagnosis:**
```bash
# Check Redis
redis-cli ping

# Check queue length
redis-cli XLEN agent_inbox:agent://demo/alice

# Check workers
ps aux | grep uvicorn
```

**Fix:**
```bash
# Restart service
sudo systemctl restart synqed

# Or kill and restart manually
lsof -ti :8000 | xargs kill -9
python main.py
```

### Issue: Signature verification failing

**Symptoms:** All messages rejected with "signature verification failed"

**Diagnosis:**
```python
from synqed.agent_email.registry.api import get_registry

# Check if sender has public_key
registry = get_registry()
entry = registry.get_by_uri("agent://demo/alice")
print(f"Public key: {entry.public_key}")
```

**Fix:**
- Ensure sender is registered with correct `public_key`
- Verify message includes all required fields: `thread_id`, `role`, `content`
- Check signature is computed with matching private key

### Issue: High DLQ count

**Symptoms:** Many messages in dead letter queue

**Diagnosis:**
```bash
# View DLQ messages
redis-cli XRANGE agent_inbox_dlq:agent://demo/alice - + COUNT 10
```

**Common causes:**
1. Remote endpoint down
2. Invalid signatures
3. Network timeouts

**Fix:**
```bash
# Check remote endpoint
curl http://remote-host:8000/health

# Increase timeout
export HTTP_TIMEOUT=60.0

# Replay DLQ messages (manual process)
```

### Issue: Rate limiting

**Symptoms:** 429 Too Many Requests errors

**Fix:**
```bash
# Increase limits in .env
SENDER_RATE_LIMIT=200
IP_RATE_LIMIT=1000

# Restart service
sudo systemctl restart synqed
```

### Issue: Port 8000 already in use

**Fix:**
```bash
# Kill process using port 8000
lsof -ti :8000 | xargs kill -9

# Or use different port
uvicorn main:app --port 8001
```

### Issue: Redis connection failed

**Symptoms:** `ConnectionError: Error connecting to Redis`

**Fix:**
```bash
# Check Redis is running
redis-cli ping

# Start Redis if needed
brew services start redis  # macOS
sudo systemctl start redis  # Linux

# Check Redis logs
tail -f /var/log/redis/redis-server.log
```

---

## Complete Code Examples

### Example 1: Local Agent with Runtime

```python
"""
Local agent that processes messages in-process.
"""
from typing import Any, Dict
from synqed.agent_email.inbox import LocalAgentRuntime, register_agent_runtime

class MyAgent(LocalAgentRuntime):
    """Custom agent implementation."""
    
    def __init__(self, name: str):
        self.name = name
    
    async def handle_a2a_envelope(
        self,
        sender: str,
        recipient: str,
        envelope: Dict[str, Any],
    ) -> Dict[str, Any] | None:
        """Process incoming message."""
        content = envelope.get("content", "")
        print(f"{self.name} received: '{content}' from {sender}")
        
        # Return response
        return {
            "thread_id": envelope.get("thread_id"),
            "role": "assistant",
            "content": f"Response from {self.name}: {content}",
        }

# Register runtime
register_agent_runtime("agent://demo/alice", MyAgent("Alice"))
```

### Example 2: Sending Messages Programmatically

```python
"""
Send messages with proper signing.
"""
import asyncio
import json
import httpx
from synqed.agent_email.inbox import sign_message

class AgentClient:
    """Client for sending A2A messages."""
    
    def __init__(self, agent_id: str, private_key: str, inbox_url: str = "http://localhost:8000"):
        self.agent_id = agent_id
        self.private_key = private_key
        self.inbox_url = inbox_url
    
    async def send_message(
        self,
        recipient: str,
        content: str,
        thread_id: str,
        trace_id: str | None = None,
    ) -> dict:
        """Send a signed message."""
        # Create message
        message = {
            "thread_id": thread_id,
            "role": "user",
            "content": content,
        }
        
        # Sign message
        signature = sign_message(
            private_key_b64=self.private_key,
            sender=self.agent_id,
            recipient=recipient,
            message=message,
            thread_id=thread_id,
        )
        
        # Prepare envelope
        envelope = {
            "sender": self.agent_id,
            "recipient": recipient,
            "message": message,
            "signature": signature,
        }
        
        if trace_id:
            envelope["trace_id"] = trace_id
        
        # Send
        async with httpx.AsyncClient() as client:
            response = await client.post(
                f"{self.inbox_url}/v1/a2a/inbox",
                json=envelope,
                timeout=30.0,
            )
            response.raise_for_status()
            return response.json()

# Usage
async def main():
    # Load keypair
    with open('keypairs.json', 'r') as f:
        keypairs = json.load(f)
    
    # Create client
    alice = AgentClient(
        agent_id="agent://demo/alice",
        private_key=keypairs["agent://demo/alice"]["private_key"],
    )
    
    # Send message
    result = await alice.send_message(
        recipient="agent://demo/bob",
        content="Hello Bob!",
        thread_id="conversation-123",
    )
    
    print(f"Message sent: {result['message_id']}")

if __name__ == "__main__":
    asyncio.run(main())
```

### Example 3: Batch Message Processing

```python
"""
Send multiple messages efficiently.
"""
import asyncio
import json
from agent_client import AgentClient  # From Example 2

async def send_batch(client: AgentClient, messages: list[dict]):
    """Send multiple messages concurrently."""
    tasks = [
        client.send_message(
            recipient=msg["recipient"],
            content=msg["content"],
            thread_id=msg["thread_id"],
        )
        for msg in messages
    ]
    
    results = await asyncio.gather(*tasks, return_exceptions=True)
    
    # Process results
    succeeded = []
    failed = []
    
    for i, result in enumerate(results):
        if isinstance(result, Exception):
            failed.append({"message": messages[i], "error": str(result)})
        else:
            succeeded.append(result)
    
    return succeeded, failed

async def main():
    # Setup client
    with open('keypairs.json', 'r') as f:
        keypairs = json.load(f)
    
    alice = AgentClient(
        agent_id="agent://demo/alice",
        private_key=keypairs["agent://demo/alice"]["private_key"],
    )
    
    # Batch messages
    messages = [
        {"recipient": "agent://demo/bob", "content": "Message 1", "thread_id": "thread-1"},
        {"recipient": "agent://demo/bob", "content": "Message 2", "thread_id": "thread-2"},
        {"recipient": "agent://demo/bob", "content": "Message 3", "thread_id": "thread-3"},
    ]
    
    # Send batch
    succeeded, failed = await send_batch(alice, messages)
    
    print(f"✓ Sent: {len(succeeded)}")
    print(f"✗ Failed: {len(failed)}")

if __name__ == "__main__":
    asyncio.run(main())
```

---

## Summary

You now have everything needed to deploy Synqed's production-grade inbox:

**Key Commands:**
```bash
# Setup
brew install redis && brew services start redis
pip install synqed cryptography redis httpx uvicorn
python init_agents.py

# Run
lsof -ti :8000 | xargs kill -9 2>/dev/null; python main.py

# Test
curl http://localhost:8000/health
python test_send.py

# Monitor
python monitor.py
```

**Architecture:**
1. **Inbox endpoint** validates, signs, queues (instant return)
2. **Redis Streams** durable queue with retry
3. **Workers** process async (local runtime or remote forward)
4. **DLQ** captures failures after 5 retries

**Production Deployment:**
- Multi-worker: `uvicorn main:app --workers 4`
- Docker: `docker-compose up -d`
- Kubernetes: Apply deployment manifests

**Files Created:**
- `main.py` - Application entry point
- `init_agents.py` - Agent initialization
- `.env` - Configuration
- `keypairs.json` - Agent keypairs (use secrets manager in prod)
- `test_send.py` - Testing script
- `monitor.py` - Queue monitoring

---

**You're ready to deploy! 🚀**

For questions, check logs and queue status. All production features are active: crypto identity, guaranteed delivery, rate limiting, distributed tracing, and DLQ.

