# Distributed Systems Integration Review: Akosha Ecosystem

**Date**: 2026-01-31
**Reviewer**: Workflow Orchestrator Agent
**Scope**: Integration architecture across Session-Buddy, Mahavishnu, Oneiric, and Akosha
**Focus**: Production reality, failure modes, operational complexity

______________________________________________________________________

## Executive Summary

**Overall Assessment**: **7.5/10** - Solid foundation with critical gaps in failure handling

The Akosha ecosystem demonstrates thoughtful service decomposition and clear responsibility boundaries. However, several **HIGH** and **CRITICAL** concerns exist around dependency cascades, single points of failure, and missing failure modes that will cause production incidents.

**Key Findings**:

- ✅ **Excellent**: Service boundaries and pull model decoupling
- ✅ **Excellent**: Clear separation of concerns (4 services)
- ⚠️ **Concerning**: No clear Mahavishnu failure strategy
- ⚠️ **Concerning**: MCP as primary orchestration protocol is non-standard
- ❌ **Critical**: Missing ingestion backlog monitoring/alerting
- ❌ **Critical**: No documented partial failure scenarios

______________________________________________________________________

## Architecture Overview

### Service Topology

```
┌─────────────────┐      ┌─────────────────┐
│ Session-Buddy   │      │   Mahavishnu    │
│ (100-100k       │      │ (Orchestrator)  │
│  instances)     │      │                 │
└────────┬────────┘      └────────┬────────┘
         │                        │
         │ S3 Upload              │ MCP Trigger
         ▼                        ▼
    ┌─────────────────────────────────┐
    │         Cloud Storage           │
    │    (S3/Azure/GCS via Oneiric)   │
    └──────────────┬──────────────────┘
                   │ Pull Model
                   ▼
    ┌─────────────────────────────────┐
    │          Akosha                 │
    │  • Hot Store (DuckDB in-memory) │
    │  • Warm Store (DuckDB on-disk)  │
    │  • Cold Store (Parquet + R2)    │
    │  • Ingestion Workers            │
    │  • Query APIs (MCP + REST)      │
    └─────────────────────────────────┘
```

### Communication Matrix

| Source → Target | Protocol | Purpose | SPOF Risk |
|-----------------|-----------|---------|-----------|
| **Session-Buddy → Cloud Storage** | Oneiric Adapter | Upload memories | Low (storage is decoupled) |
| **Mahavishnu → Akosha** | MCP | Trigger workflows | **HIGH** (Mahavishnu required) |
| **Akosha → Mahavishnu** | MCP | Health/metrics | **HIGH** (reporting dependency) |
| **Akosha → Cloud Storage** | Oneiric Adapter | Pull/process data | Low (storage is decoupled) |
| **Consumers → Akosha** | REST/MCP | Query APIs | Low (multiple consumers) |

______________________________________________________________________

## Strengths

### ✅ 1. Service Decomposition (Excellent)

**Rating**: 9/10

The 4-service architecture has **excellent** separation of concerns:

| Service | Responsibility | Boundaries |
|---------|----------------|------------|
| **Session-Buddy** | User memory collection | Uploads to cloud, no Akosha dependency |
| **Mahavishnu** | Workflow orchestration | Triggers workflows, scales pods |
| **Oneiric** | Storage abstraction | Multi-cloud adapters, stack_level routing |
| **Akosha** | Aggregation and query | Pulls from storage, serves queries |

**Why this works**:

- Session-Buddy can evolve independently (no Akosha API dependency)
- Akosha can be down without blocking Session-Buddy uploads
- Clear responsibility ownership per team
- Each service has independent deployment cadence

**Evidence from code**:

```python
# Session-Buddy: Upload and forget
await oneiric_storage.upload(
    bucket="session-buddy-memories",
    path=f"systems/{system_id}/memory.db",
    data=memory_db
)
# No Akosha dependency - excellent decoupling

# Akosha: Pull independently
uploads = await oneiric_storage.list_prefixes(
    bucket="session-buddy-memories",
    prefix=f"systems/{system_id}/"
)
```

### ✅ 2. Pull Model Design (Excellent)

**Rating**: 9/10

The pull model (Session-Buddy → S3 → Akosha) is **production-grade** decoupling:

**Benefits**:

- ✅ **Failure isolation**: Akosha down doesn't block Session-Buddy
- ✅ **No write-path latency**: User experience unaffected
- ✅ **Replay capability**: Can reprocess historical uploads
- ✅ **Independent scaling**: Scale Akosha without touching Session-Buddy

**Trade-offs**:

- ⚠️ **Eventual consistency**: 30-second to 5-minute delay
- ⚠️ **Duplicate storage**: Uploads stored twice temporarily
- ⚠️ **Discovery needed**: Need polling or S3 events for new uploads

**Verdict**: Pull model is the **right choice** for this use case. Push model would create tight coupling and failure cascades.

### ✅ 3. Three-Tier Storage (Excellent)

**Rating**: 8.5/10

Hot/Warm/Cold tiering is **well-designed** for cost optimization:

| Tier | Retention | Storage | Cost Factor | Use Case |
|------|-----------|---------|-------------|----------|
| **Hot** | 0-7 days | DuckDB in-memory | 100x | Recent searches |
| **Warm** | 7-90 days | DuckDB on-disk | 10x | Historical analytics |
| **Cold** | 90+ days | Parquet + R2 | 1x | Compliance/archive |

**Why this works**:

- ✅ **80% cost reduction** for cold data (no embeddings)
- ✅ **Sub-second latency** for hot tier (most queries)
- ✅ **Automatic aging** reduces operational overhead
- ✅ **Clear upgrade path** to Milvus/TimescaleDB at scale

**Concern**: Automatic aging service is **not yet implemented** (see "Critical Issues" below).

### ✅ 4. Oneiric Storage Abstraction (Good)

**Rating**: 7.5/10

Oneiric provides **multi-cloud flexibility** via `stack_level` prioritization:

```python
# Hot tier (stack_level=100)
hot_storage = await bridge.use("storage-s3-hot")

# Warm tier (stack_level=50)
warm_storage = await bridge.use("storage-s3-warm")

# Cold tier (stack_level=10)
cold_storage = await bridge.use("storage-s3-cold")
```

**Benefits**:

- ✅ **Vendor portability**: Switch S3/Azure/GCS via config
- ✅ **Tier abstraction**: `stack_level` for priority routing
- ✅ **Unified interface**: Single API for all storage backends

**Concerns**:

- ⚠️ **Dependency risk**: Oneiric is a shared dependency
- ⚠️ **Version coupling**: Akosha tied to Oneiric API changes
- ⚠️ **Debugging complexity**: Oneiric failures obscure root cause

### ✅ 5. Circuit Breaker Pattern (Good)

**Rating**: 8/10

Akosha has **comprehensive circuit breakers** for external calls:

```python
# /Users/les/Projects/akosha/akosha/resilience/circuit_breaker.py
class CircuitBreaker:
    """Circuit breaker for protecting external service calls."""
    def __init__(self, service_name: str, config: CircuitBreakerConfig):
        self.failure_threshold = 5  # Failures before opening
        self.timeout = 60.0  # Seconds before half-open
        self.call_timeout = 30.0  # Max seconds per call
```

**Why this works**:

- ✅ **Cascading failure prevention**: Stops bouncing failures
- ✅ **Graceful degradation**: Falls back to next tier
- ✅ **Auto-recovery**: Half-open state tests service recovery
- ✅ **Observable**: Statistics and state exposed via metrics

**Evidence**: 98.78% test coverage for circuit breaker module.

______________________________________________________________________

## Concerns

### 🔴 CRITICAL: No Mahavishnu Failure Strategy

**Severity**: **CRITICAL**
**Likelihood**: **HIGH** (will happen in first 6 months of production)
**Impact**: **Akosha cannot scale, trigger workflows, or report health**

**Problem**:

Akosha is **fully dependent** on Mahavishnu for:

1. Workflow triggering (scheduled/cron/manual)
1. Pod scaling based on backlog metrics
1. Health check monitoring
1. Alert escalation

**What happens when Mahavishnu is down?**

```python
# Akosha startup - registers with Mahavishnu
await register_with_mahavishnu(
    workflows=["akosha-daily-ingest", "akosha-tier-transition"]
)

# If Mahavishnu is down:
# ❌ Akosha cannot register workflows
# ❌ No scheduled ingestion triggers
# ❌ No automatic pod scaling
# ❌ No health check reporting
# ❌ No alert escalation
```

**Missing documentation**: What should Akosha do when Mahavishnu is unreachable?

**Recommendations**:

1. **Bootstrap mode**: Akosha should have "safe mode" when Mahavishnu is down

   ```python
   # Akosha should work without Mahavishnu
   if not await mahavishnu_healthy():
       logger.warning("Mahavishnu unreachable - entering bootstrap mode")
       # Use local cron for critical workflows
       # Use HPA metrics instead of Mahavishnu-based scaling
       # Buffer metrics until Mahavishnu recovers
   ```

1. **Dual health reporting**: Report to both Mahavishnu AND Prometheus

   ```python
   # Push metrics to Prometheus (always works)
   prometheus_client.push_metrics(akosha_metrics)

   # Report to Mahavishnu (best-effort)
   try:
       await mahavishnu.report_health(metrics)
   except MahavishnuUnavailable:
       logger.warning("Mahavishnu unavailable - metrics buffered")
   ```

1. **Document fail-fast behavior**:

   ```markdown
   ## Mahavishnu Failure Mode

   If Mahavishnu is down at Akosha startup:

   1. Akosha enters "bootstrap mode"
   2. Local cron triggers daily ingestion (12:00 AM UTC)
   3. HPA scales based on CPU/memory (not backlog)
   4. Metrics buffered to local disk (max 1GB)
   5. Alerts sent via PagerDuty (not Mahavishnu)

   When Mahavishnu recovers:
   1. Buffer replayed
   2. Workflows re-registered
   3. Normal operation resumes
   ```

### 🔴 CRITICAL: Missing Ingestion Backlog Alerting

**Severity**: **CRITICAL**
**Likelihood**: **HIGH** (backlog will grow during incidents)
**Impact**: **Data becomes stale, SLA breaches**

**Problem**:

Akosha has **no documented alert** for ingestion backlog:

```python
# config/akosha.yaml
ingestion:
  workers: 3
  poll_interval_seconds: 30
  batch_size: 100

# ❌ No alert configured for backlog > 10,000 uploads
# ❌ No automatic scaling trigger defined
# ❌ No SLA breach detection
```

**What happens**:

1. Upload rate spikes (e.g., 10x normal traffic)
1. 3 workers cannot keep up
1. Backlog grows to 50,000 uploads
1. Data becomes 12+ hours stale
1. **No alert fired** - ops team unaware until users complain

**Evidence**: ADR-001 mentions "Backlog > 10,000 uploads" as a failure scenario, but no alert is configured.

**Recommendations**:

1. **Add Prometheus alert** (urgent):

   ```yaml
   # monitoring/prometheus/alerts.yml
   - alert: AkoshaIngestionBacklogHigh
     expr: akosha_ingestion_backlog_count > 10000
     for: 5m
     labels:
       severity: warning
     annotations:
       summary: "Ingestion backlog above 10,000"
       description: "{{ $value }} uploads pending ingestion"

   - alert: AkoshaIngestionBacklogCritical
     expr: akosha_ingestion_backlog_count > 50000
     for: 2m
     labels:
       severity: critical
     annotations:
       summary: "Critical ingestion backlog"
       description: "{{ $value }} uploads pending - scale pods immediately"
   ```

1. **Auto-scale trigger**:

   ```yaml
   # k8s/hpa.yaml
   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   spec:
     scaleTargetRef:
       name: akosha-ingestion
     minReplicas: 3
     maxReplicas: 50
     metrics:
     - type: Pods
       pods:
         metric:
           name: akosha_ingestion_backlog_count
         target:
           type: AverageValue
           averageValue: "1000"  # Scale up if backlog > 1000 per pod
   ```

1. **Backlog metric implementation**:

   ```python
   # akosha/ingestion/worker.py
   from prometheus_client import Gauge

   ingestion_backlog = Gauge(
       "akosha_ingestion_backlog_count",
       "Number of uploads pending ingestion",
   )

   async def _discover_uploads(self) -> list[dict]:
       """Discover and count pending uploads."""
       uploads = await self._list_pending_uploads()
       ingestion_backlog.set(len(uploads))
       return uploads
   ```

### 🟠 HIGH: MCP as Orchestration Protocol (Non-Standard)

**Severity**: **HIGH**
**Likelihood**: **MEDIUM**
**Impact**: **Operational complexity, debugging difficulty**

**Problem**:

Using MCP (Model Context Protocol) for orchestration is **unconventional**:

```python
# Mahavishnu → Akosha via MCP
@mahavishnu_server.tool()
async def trigger_akosha_ingest(source_system: str):
    akosha_client = get_akosha_mcp_client()
    result = await akosha_client.call_tool(
        "akosha_start_ingestion",
        {"source_system": source_system}
    )
```

**Why this is concerning**:

1. **Non-standard**: MCP is designed for LLM tool calling, not service orchestration

   - Standard orchestration: gRPC, REST, message queues (SQS/Kafka)
   - Chosen protocol: MCP (JSON-RPC over stdio/HTTP)

1. **Debugging complexity**: MCP tool calls are harder to trace than REST

   - No standardized request/response logging
   - No OpenAPI/Swagger documentation
   - Fewer debugging tools available

1. **Operational unfamiliarity**: Ops teams don't know MCP

   - Standard ops tools: curl, Postman, AWS X-Ray
   - MCP requires specialized tooling

1. **Vendor lock-in**: FastMCP is a custom library

   - Smaller ecosystem than gRPC/REST
   - Fewer client libraries

**When MCP makes sense**:

- ✅ LLM-to-service communication (designed use case)
- ✅ Human-in-the-loop workflows (Claude Code calling Akosha)

**When MCP doesn't make sense**:

- ❌ Service-to-service orchestration (Mahavishnu → Akosha)
- ❌ Health checks and metrics (use REST/HTTP)
- ❌ High-frequency triggers (use message queue)

**Recommendations**:

1. **Dual protocol**: Keep MCP for LLM use, add gRPC for service orchestration

   ```python
   # Keep MCP for LLM workflows
   @akosha.mcp.tool()
   async def akosha_search(query: str):
       """Search via MCP (for Claude Code)."""
       return await search_conversations(query)

   # Add gRPC for service orchestration
   class AkoshaService(akosha_pb2_grpc.AkoshaServiceServicer):
       async def TriggerIngestion(self, request, context):
           """Trigger ingestion via gRPC (for Mahavishnu)."""
           return await start_ingestion(request.system_id)
   ```

1. **Document MCP limitations**:

   ```markdown
   ## MCP Protocol Constraints

   MCP is used for:
   - ✅ LLM-triggered workflows (Claude Code)
   - ✅ Interactive tool exploration

   MCP is NOT used for:
   - ❌ High-frequency triggers (>10 req/sec) → Use gRPC
   - ❌ Binary data transfer → Use REST/gRPC
   - ❌ Service mesh integration → Use gRPC
   ```

1. **Add REST fallback** for critical operations:

   ```python
   # Akosha REST API (always available)
   @app.post("/api/v1/ingestion/trigger")
   async def trigger_ingestion_rest(system_id: str):
       """REST endpoint for ingestion triggering."""
       return await start_ingestion(system_id)
   ```

### 🟠 HIGH: Oneiric Dependency Risk

**Severity**: **HIGH**
**Likelihood**: **MEDIUM**
**Impact**: **Storage failures affect multiple services**

**Problem**:

Akosha depends on Oneiric for **all cloud storage operations**:

```python
# Akosha → Oneiric → S3
from oneiric.adapters import StorageAdapter

storage = await bridge.use("storage-s3-cold")
await storage.upload(bucket="akosha-cold", path=data.id, data=data)
```

**Failure scenarios**:

1. **Oneiric adapter bug**: All storage operations fail

   - Example: Oneiric S3 adapter has multipart upload bug
   - Impact: Akosha cannot write to cold tier
   - Recovery: Wait for Oneiric fix OR bypass with direct S3 client

1. **Oneiric version skew**: Breaking API changes

   - Example: Oneiric v2.0 changes `upload()` signature
   - Impact: Akosha deployment fails
   - Recovery: Pin Oneiric version OR update Akosha code

1. **Oneiric deployment dependency**: Shared runtime

   - Example: Oneiric deployed as sidecar
   - Impact: Oneiric pod crash = Akosha storage unavailable
   - Recovery: Restart Oneiric sidecar

**Current mitigation**: **None documented**.

**Recommendations**:

1. **Circuit break Oneiric calls** (already implemented, good):

   ```python
   # /Users/les/Projects/akosha/akosha/resilience/circuit_breaker.py
   oneiric_breaker = CircuitBreaker(
       service_name="oneiric-s3-cold",
       config=CircuitBreakerConfig(
           failure_threshold=5,
           timeout=60.0,
       )
   )

   @oneiric_breaker.call
   async def upload_to_cold(data: bytes):
       return await cold_storage.upload(data)
   ```

1. **Add fallback to direct S3 client**:

   ```python
   async def upload_with_fallback(data: bytes, path: str):
       """Upload with Oneiric fallback."""
       try:
           # Try Oneiric first
           return await oneiric_storage.upload(path, data)
       except OneiricUnavailable:
           logger.warning("Oneiric unavailable - using direct S3 client")
           # Fallback to boto3
           return await boto3_client.upload_fileobj(data, path)
   ```

1. **Version pinning**:

   ```toml
   # pyproject.toml
   [dependencies]
   oneiric = ">=1.2.0,<2.0.0"  # Pin to major version
   ```

1. **Document Oneiric failure mode**:

   ````markdown
   ## Oneiric Failure Recovery

   If Oneiric adapters fail:

   1. Check Oneiric service health
   2. Check circuit breaker state: GET /api/v1/circuit_breakers
   3. If OPEN: Wait 60 seconds for auto-recovery
   4. If still failing: Enable direct S3 fallback
      ```bash
      export AKOSHA_STORAGE_DIRECT_S3=true
      kubectl rollout restart deployment/akosha-ingestion
   ````

   ```

   ```

### 🟡 MEDIUM: No Message Queue for Event-Driven Ingestion

**Severity**: **MEDIUM**
**Likelihood**: **MEDIUM**
**Impact**: **Inefficient polling, delayed ingestion**

**Problem**:

Current ingestion uses **polling** instead of event-driven architecture:

```python
# Current: Poll every 30 seconds
class IngestionWorker:
    async def run(self):
        while self._running:
            uploads = await self._discover_uploads()  # S3 LIST operation
            await asyncio.sleep(30)  # Poll interval
```

**Why polling is problematic**:

1. **Wasteful**: S3 LIST operations are expensive ($0.005 per 1,000 requests)

   - 3 workers × 2 polls/min = 8,640 polls/day = $0.13/day
   - At scale: 50 workers × 2 polls/min = 144,000 polls/day

1. **Delayed ingestion**: 30-second average delay

   - Upload arrives at t=0
   - Worker polls at t=15 (misses)
   - Worker polls at t=45 (detects)
   - **45-second average delay**

1. **No push scaling**: Can't scale workers based on incoming events

   - Must over-provision workers
   - Can't use event-driven autoscaling

**Event-driven alternative** (mentioned in ROADMAP but not implemented):

```yaml
# Event-driven ingestion via S3 + SQS
Session-Buddy → S3 upload → S3 Event → SQS queue → Akosha workers
```

**Benefits of event-driven**:

- ✅ **Real-time**: Sub-second ingestion latency
- ✅ **Efficient**: No polling overhead
- ✅ **Auto-scaling**: Scale based on SQS queue depth

**Trade-offs**:

- ⚠️ **Complexity**: Additional SQS infrastructure
- ⚠️ **Cost**: SQS request costs ($0.40 per 1M requests)
- ⚠️ **Failure handling**: Need DLQ for failed messages

**Recommendations**:

1. **Keep polling for Phase 1** (100 systems): Acceptable trade-off

1. **Add event-driven for Phase 2** (1,000+ systems):

   ```python
   # Phase 2: Event-driven ingestion
   import boto3

   sqs_client = boto3.client('sqs')

   async def listen_for_events():
       """Listen for S3 upload events via SQS."""
       while True:
           messages = await sqs_client.receive_message(
               QueueUrl=os.getenv['AKOSHA_INGESTION_QUEUE'],
               MaxNumberOfMessages=10,
               WaitTimeSeconds=20,  # Long polling
           )
           for msg in messages:
               await process_s3_event(msg)
   ```

1. **Hybrid approach**: Polling + event-driven

   ```python
   # Use SQS when available, fall back to polling
   if os.getenv('AKOSHA_INGESTION_QUEUE'):
       await listen_for_events()  # Event-driven
   else:
       await poll_s3()  # Fallback polling
   ```

### 🟡 MEDIUM: Missing Graceful Shutdown Handling

**Severity**: **MEDIUM**
**Likelihood**: **HIGH** (happens during every deployment)
**Impact**: **Data loss, interrupted queries**

**Problem**:

Akosha ingestion worker has **no graceful shutdown** logic:

```python
# /Users/les/Projects/akosha/akosha/ingestion/worker.py
class IngestionWorker:
    def stop(self) -> None:
        """Stop the worker."""
        self._running = False
        logger.info("Ingestion worker stopped")
```

**What happens during deployment**:

1. Kubernetes sends SIGTERM to Akosha pod
1. Worker's `stop()` called immediately
1. **Current upload processing aborted mid-way**
1. Partial data written to hot store
1. Next worker re-processes upload (duplicate work)
1. Orphaned records in hot store

**Missing logic**:

- ❌ No in-flight upload completion
- ❌ No checkpoint/save state
- ❌ No drain period before shutdown
- ❌ No connection cleanup (DuckDB, Redis)

**Recommendations**:

1. **Add graceful shutdown handler**:

   ```python
   class IngestionWorker:
       def __init__(self):
           self._running = False
           self._shutting_down = False
           self._in_flight_uploads = set()

       async def run(self):
           """Main worker loop with graceful shutdown."""
           self._running = True

           # Register signal handlers
           loop = asyncio.get_running_loop()
           loop.add_signal_handler(signal.SIGTERM, self._shutdown)

           while self._running:
               try:
                   uploads = await self._discover_uploads()

                   for upload in uploads:
                       if self._shutting_down:
                           break  # Stop accepting new work

                       self._in_flight_uploads.add(upload['id'])
                       try:
                           await self._process_upload(upload)
                       finally:
                           self._in_flight_uploads.remove(upload['id'])

                   if self._shutting_down:
                       # Wait for in-flight uploads to complete
                       await self._wait_for_in_flight()
                       break

               except Exception as e:
                   logger.error(f"Ingestion error: {e}")

       def _shutdown(self):
           """Initiate graceful shutdown."""
           logger.info("SIGTERM received - initiating graceful shutdown")
           self._shutting_down = True
           self._running = False

       async def _wait_for_in_flight(self, timeout: int = 30):
           """Wait for in-flight uploads to complete."""
           deadline = time.time() + timeout
           while self._in_flight_uploads and time.time() < deadline:
               logger.info(f"Waiting for {len(self._in_flight_uploads)} in-flight uploads")
               await asyncio.sleep(1)

           if self._in_flight_uploads:
               logger.warning(f"Shutdown timeout - {len(self._in_flight_uploads)} uploads may be incomplete")

       async def close(self):
           """Clean up resources."""
           await self.hot_store.close()
           await self.redis.close()
           logger.info("Worker shutdown complete")
   ```

1. **Kubernetes preStop hook**:

   ```yaml
   # k8s/deployment.yaml
   lifecycle:
     preStop:
       exec:
         command: ["/bin/sh", "-c", "curl -X POST http://localhost:8000/api/v1/shutdown && sleep 30"]
   ```

1. **Add shutdown endpoint**:

   ```python
   @app.post("/api/v1/shutdown")
   async def shutdown_endpoint():
       """Initiate graceful shutdown."""
       worker._shutdown()
       return {"status": "shutting_down"}
   ```

### 🟡 MEDIUM: No Documented Partial Failure Scenarios

**Severity**: **MEDIUM**
**Likelihood**: **100%** (will happen in production)
**Impact**: **Ops team unprepared, extended MTTR**

**Problem**:

ADR-001 documents failure scenarios but **doesn't provide runbooks**:

```markdown
# ADR-001 Decision 8: Failure Handling & Resilience

| Failure | Detection | Response | Recovery |
|---------|-----------|----------|----------|
| Akosha ingestion down | Mahavishnu health check | Pause uploads, queue in S3 | Scale up pods |
| Hot store corrupted | Health check fails | Failover to warm tier | Restore from backup |
```

**What's missing**:

- ❌ Step-by-step recovery procedures
- ❌ Command examples for ops to run
- ❌ Expected recovery times
- ❌ Rollback procedures
- ❌ Escalation paths

**Example runbook (should be documented)**:

````markdown
## Runbook: Akosha Hot Store Corruption

### Detection
```bash
# Hot store health check failing
kubectl get pods -n akosha
# akosha-ingestion-5d7f8b9c-abc12   0/1     CrashLoopBackOff
````

### Symptoms

- Search queries returning errors
- Ingestion failing with "Database locked"
- Logs show: `duckdb.Error: IO Error: Corruption`

### Recovery Steps

1. **Verify corruption**

   ```bash
   kubectl logs -n akosha deployment/akosha-ingestion --tail=100 | grep -i corrupt
   ```

1. **Scale to zero (stop writes)**

   ```bash
   kubectl scale deployment/akosha-ingestion --replicas=0 -n akosha
   ```

1. **Enable warm tier fallback**

   ```bash
   kubectl set env deployment/akosha-ingestion AKOSHA_HOT_FALLBACK=WARM -n akosha
   ```

1. **Restore from backup**

   ```bash
   # Restore from latest S3 backup
   aws s3 sync s3://akosha-backups/hot/$(date +%Y-%m-%d) /data/akosha/hot
   ```

1. **Resume ingestion**

   ```bash
   kubectl scale deployment/akosha-ingestion --replicas=3 -n akosha
   ```

### Expected Recovery Time

- 5 minutes (if backup available)
- 2 hours (if rebuild from warm tier required)

### Escalation

- If recovery fails after 2 hours → Page storage team
- If data loss suspected → Page security team

```

**Recommendations**:

1. **Create runbook directory**:
```

/docs/runbooks/
├── 001-ingestion-backlog.md
├── 002-hot-store-corruption.md
├── 003-mahavishnu-unreachable.md
├── 004-s3-slow-upload.md
├── 005-high-latency-queries.md
└── 006-circuit-breaker-open.md

````

2. **Runbook template**:
```markdown
# Runbook: [Title]

**Severity**: [Critical/High/Medium/Low]
**Estimated MTTR**: [X minutes]
**Escalation**: [Team/Role]

## Detection
[Commands to detect issue]

## Symptoms
[Observable symptoms]

## Recovery Steps
1. [Step 1]
2. [Step 2]
3. [Step 3]

## Rollback
[How to rollback if recovery fails]

## Prevention
[How to prevent recurrence]
````

### 🟢 LOW: Service Count Complexity

**Severity**: **LOW**
**Likelihood**: **N/A** (architectural decision)
**Impact**: **Operational overhead**

**Observation**:

4 services (Session-Buddy, Mahavishnu, Oneiric, Akosha) is **manageable** but adds complexity:

**Complexity factors**:

- 4 codebases to maintain
- 4 deployment pipelines
- 4 monitoring dashboards
- 12 service-to-service integration points
- 4 on-call rotations (or 1 complex rotation)

**When 4 services is too many**:

- Small team (\<5 engineers): 2-3 services better
- Low traffic (\<100 req/sec): Monolith simpler
- Limited ops maturity: Fewer services = fewer alerts

**When 4 services is justified**:

- Team size: 10+ engineers (can specialize)
- Scale: 1,000+ systems (need horizontal scaling)
- Complexity: Each service has distinct domain logic

**Verdict**: 4 services is **appropriate** for the stated scale (100-10,000 systems).

**Recommendation**: No changes needed, but document **why 4 services**:

```markdown
## Why 4 Services?

We chose 4 services over a monolith because:

1. **Team specialization**: Each service owned by different sub-team
2. **Independent deployment**: Session-Buddy updates don't require Akosha deployment
3. **Horizontal scaling**: Mahavishnu orchestrates 100+ Akosha pods independently
4. **Technology diversity**: Oneiric supports multi-cloud, Akosha optimized for analytics

**Trade-offs accepted**:
- Operational complexity (4 services vs 1)
- Debugging complexity (distributed traces vs single process)
- Latency (service-to-service calls vs in-process)

**If starting over**: Would choose 4 services again.
```

______________________________________________________________________

## Failure Mode Analysis

### Scenario 1: Mahavishnu Outage (Duration: 2 hours)

**Trigger**: Mahavishnu deployment failure, all pods CrashLoopBackOff

**Impact**:
| Component | Impact | Severity |
|-----------|--------|----------|
| Akosha workflow triggers | ❌ None scheduled | **HIGH** |
| Akosha pod autoscaling | ❌ Manual only | **MEDIUM** |
| Health check reporting | ❌ Not reported | **LOW** |
| Akosha query serving | ✅ Unaffected | **NONE** |

**Current behavior**:

- Akosha continues serving queries (✅ good)
- No scheduled workflows run (❌ bad)
- Ingestion workers continue polling (✅ good)
- No pod autoscaling (❌ bad)

**What should happen**:

1. Akosha detects Mahavishnu unreachable (timeout: 10s)
1. Enters "bootstrap mode" with degraded functionality:
   - Local cron schedules daily ingestion (12:00 AM UTC)
   - HPA scales based on CPU/memory (not backlog)
   - Metrics buffered to local file (max 1 hour)
1. Alert sent: "Mahavishnu unreachable - Akosha in bootstrap mode"
1. When Mahavishnu recovers:
   - Buffer replayed
   - Workflows re-registered
   - Normal operation resumes

**Gap**: Bootstrap mode **not implemented**.

### Scenario 2: S3 Outage (Duration: 30 minutes)

**Trigger**: AWS S3 us-east-1 outage (https://status.aws.amazon.com/)

**Impact**:
| Component | Impact | Severity |
|-----------|--------|----------|
| Session-Buddy uploads | ❌ Queued locally | **MEDIUM** |
| Akosha ingestion | ⚠️ Backlog grows | **MEDIUM** |
| Akosha hot tier | ✅ Unaffected | **NONE** |
| Akosha queries | ✅ Serve from hot | **NONE** |

**Current behavior**:

- Session-Buddy buffers uploads locally (✅ good)
- Akosha workers retry S3 operations (✅ good)
- Circuit breaker opens after 5 failures (✅ good)
- Queries continue serving from hot tier (✅ excellent)

**Graceful degradation score**: **9/10** (excellent)

**Improvement needed**: Document S3 outage runbook

````markdown
## Runbook: S3 Outage

### Detection
```bash
# Circuit breaker OPEN
kubectl get configmap akosha-config -o jsonpath='{.data.circuit_breakers}'
# {"oneiric-s3-cold": "open"}

# S3 health check failing
aws s3 ls s3://akosha-cold --summarize
# An error occurred (503) when calling the ListObjectsV2 operation
````

### Symptoms

- Ingestion backlog growing
- Circuit breaker OPEN for S3
- Logs show: `ConnectionError: S3 unavailable`

### Recovery

1. **Wait for AWS recovery** (check https://status.aws.amazon.com/)
1. **Circuit breaker auto-recovers** after 60 seconds
1. **Backlog drains** automatically
1. **No manual intervention needed**

### Prevention

- Multi-region S3 (replication to us-west-2)
- Local buffering for 1 hour of uploads

````

### Scenario 3: Akosha Hot Store Memory Exhaustion

**Trigger**: Memory leak or query spike causes OOM

**Impact**:
| Component | Impact | Severity |
|-----------|--------|----------|
| Hot store | ❌ Crash/restart | **HIGH** |
| Warm store | ✅ Unaffected | **NONE** |
| Queries | ⚠️ Fallback to warm | **MEDIUM** |
| Ingestion | ⚠️ Queued until restart | **LOW** |

**Current behavior**:
- Pod OOMKilled
- 30-second restart (Kubernetes)
- Hot store data **lost** (in-memory)
- Queries failover to warm tier (✅ good)

**Graceful degradation score**: **6/10** (acceptable, but data loss)

**Improvement needed**:
1. **Write-ahead log** for hot store (mentioned in config but not implemented):
   ```python
   # config/akosha.yaml
   storage:
     hot:
       write_ahead_log: true
       wal_path: "/data/akosha/wal"

   # Should implement:
   class HotStore:
       def __init__(self, wal_path: Path):
           self.wal = open(wal_path, 'a')

       async def insert(self, record: HotRecord):
           # Write to WAL first (durability)
           await self.wal.write(record.json())
           await self.wal.flush()

           # Then write to DuckDB (performance)
           self.conn.execute("INSERT INTO conversations ...")
````

2. **Warm tier promotion** on hot store failure:
   ```python
   async def search_with_fallback(query_embedding: list[float]):
       try:
           return await hot_store.search(query_embedding)
       except HotStoreUnavailable:
           logger.warning("Hot store unavailable - failing over to warm tier")
           return await warm_store.search(query_embedding)
   ```

______________________________________________________________________

## Integration Pattern Assessment

### 1. Pull Model vs Push Model

**Verdict**: ✅ **Pull model is correct choice**

| Criteria | Pull Model (Chosen) | Push Model (Alternative) |
|----------|---------------------|--------------------------|
| **Failure isolation** | ✅ Excellent (Akosha down doesn't block Session-Buddy) | ❌ Poor (Session-Buddy must retry) |
| **Latency** | ⚠️ 30s - 5min delay | ✅ Immediate |
| **Complexity** | ✅ Simple (no retry logic) | ❌ Complex (exponential backoff, dead letter queue) |
| **Scalability** | ✅ Horizontal (add workers) | ✅ Horizontal (scale Akosha) |
| **Replay capability** | ✅ Re-process historical data | ❌ Only current data |
| **Infrastructure** | ⚠️ S3 + polling (or S3 + SQS) | ❌ Load balancer + retry logic |

**Recommendation**: Keep pull model, add **event-driven** (S3 → SQS → Akosha) for Phase 2 to reduce latency.

### 2. Service Mesh: Needed or Not?

**Current state**: No service mesh (plain Kubernetes Service)

**Service mesh benefits**:

- ✅ mTLS encryption (automatic)
- ✅ Traffic shifting (canary deployments)
- ✅ Observability (distributed tracing)
- ✅ Retry logic (application-level)

**Service mesh costs**:

- ⚠️ Complexity (Istio/Linkerd ops overhead)
- ⚠️ Latency (sidecar proxy overhead: 2-5ms)
- ⚠️ Resource usage (sidecar CPU/memory)

**Verdict**: 🟡 **Not needed for Phase 1, evaluate for Phase 3**

**Recommendation**:

- Phase 1-2 (100-1,000 systems): **No service mesh**
  - Use application-level mTLS (if needed)
  - Use application-level retries
  - Use OpenTelemetry tracing (already implemented)
- Phase 3+ (10,000+ systems): **Consider Istio**
  - Multi-cluster traffic management
  - Advanced traffic shifting (blue/green)
  - Policy enforcement (RBAC)

### 3. Message Queue: SQS vs Polling

**Current state**: Polling (30-second interval)

**Message queue benefits**:

- ✅ Real-time ingestion (sub-second latency)
- � Efficient (no polling overhead)
- ✅ Auto-scaling (scale based on queue depth)
- ✅ Dead letter queue (failed messages)

**Message queue costs**:

- ⚠️ Infrastructure (SQS + S3 event configuration)
- ⚠️ Cost ($0.40 per 1M requests)
- ⚠️ Failure handling (DLQ monitoring)

**Verdict**: 🟡 **Keep polling for Phase 1, add SQS for Phase 2**

**Recommendation**:

```python
# Hybrid approach
class IngestionWorker:
    def __init__(self):
        self.use_sqs = bool(os.getenv('AKOSHA_INGESTION_QUEUE'))

    async def run(self):
        if self.use_sqs:
            await self._listen_for_events()  # Phase 2: Event-driven
        else:
            await self._poll_s3()  # Phase 1: Polling
```

### 4. API Versioning Strategy

**Current state**: No versioning documented

**REST API**:

```python
# Current: No versioning
@app.post("/api/v1/search")  # Hardcoded v1
async def search(request: SearchRequest):
    pass
```

**MCP API**:

```python
# No versioning mechanism
@akosha.mcp.tool()
async def akosha_search(query: str):
    pass
```

**Risk**: Breaking API changes will break consumers

**Recommendation**: Document versioning strategy

```markdown
## API Versioning

### REST API

URL-based versioning (recommended):
- `/api/v1/search` (current)
- `/api/v2/search` (future, with breaking changes)

Deprecation timeline:
- v1 supported for 12 months after v2 release
- v1 deprecated after 6 months
- v1 sunset after 12 months

### MCP API

Tool-based versioning:
- `akosha_search_v1` (current)
- `akosha_search_v2` (future, with breaking changes)

Both versions supported indefinitely (no breaking changes to existing tools).
```

______________________________________________________________________

## Recommendations Summary

### Critical (Must Fix Before Production)

1. **Add Mahavishnu failure handling** (bootstrap mode)

   - [ ] Implement local cron fallback for scheduled workflows
   - [ ] Buffer metrics when Mahavishnu unreachable
   - [ ] Document bootstrap mode behavior

1. **Add ingestion backlog alerting**

   - [ ] Create Prometheus alert: `akosha_ingestion_backlog_count > 10000`
   - [ ] Create HPA trigger: Scale pods when backlog > 1000/pod
   - [ ] Implement `akosha_ingestion_backlog_count` metric

1. **Document partial failure scenarios** (runbooks)

   - [ ] Create `/docs/runbooks/` directory
   - [ ] Write 6 runbooks for common failures
   - [ ] Include step-by-step recovery commands

### High (Should Fix in Next Sprint)

4. **Evaluate MCP for orchestration**

   - [ ] Audit all Mahavishnu → Akosha MCP calls
   - [ ] Add gRPC endpoint for high-frequency triggers
   - [ ] Keep MCP for LLM workflows only

1. **Add Oneiric fallback**

   - [ ] Implement direct S3 client fallback
   - [ ] Add Oneiric health check
   - [ ] Document Oneiric failure recovery

1. **Implement graceful shutdown**

   - [ ] Add in-flight upload completion
   - [ ] Add 30-second drain period
   - [ ] Add Kubernetes preStop hook

### Medium (Nice to Have)

7. **Add event-driven ingestion** (Phase 2)

   - [ ] Configure S3 → SQS event notifications
   - [ ] Implement SQS listener
   - [ ] Keep polling as fallback

1. **Add hot store write-ahead log**

   - [ ] Implement WAL for in-memory hot store
   - [ ] Add WAL replay on startup
   - [ ] Document recovery procedures

1. **Document API versioning strategy**

   - [ ] Write API versioning guide
   - [ ] Add deprecation timeline
   - [ ] Version MCP tools (v1/v2)

### Low (Future Consideration)

10. **Evaluate service mesh** (Phase 3+)
    - [ ] Assess multi-cluster needs
    - [ ] Compare Istio vs Linkerd
    - [ ] Pilot service mesh in staging

______________________________________________________________________

## Conclusion

The Akosha ecosystem demonstrates **solid distributed systems fundamentals**:

- ✅ **Excellent service boundaries** (4 services with clear responsibilities)
- ✅ **Correct pull model** (decouples Session-Buddy from Akosha)
- ✅ **Well-designed storage tiering** (cost-effective hot/warm/cold)
- ✅ **Comprehensive circuit breakers** (prevents cascading failures)

**Critical gaps** must be addressed before production:

1. 🔴 **Mahavishnu failure handling** - Add bootstrap mode
1. 🔴 **Backlog alerting** - Add Prometheus alerts and HPA triggers
1. 🔴 **Runbook documentation** - Document failure recovery procedures

**High-priority improvements**:

4. 🟠 **MCP protocol evaluation** - Consider gRPC for orchestration
1. 🟠 **Oneiric fallback** - Add direct S3 client
1. 🟠 **Graceful shutdown** - Complete in-flight work

**Overall readiness**: **7.5/10** - Address critical gaps before production deployment.

______________________________________________________________________

**Next steps**:

1. Prioritize critical issues (1-3) for next sprint
1. Create runbooks for common failure scenarios
1. Add integration tests for failure modes
1. Conduct chaos engineering test (Mahavishnu outage simulation)

______________________________________________________________________

**Document version**: 1.0
**Last updated**: 2026-01-31
**Review frequency**: Quarterly (or after major architecture changes)
