ALACTIC AGI Framework - Docker Deployment Guide

# ALACTIC AGI Framework

Enterprise AI Dataset Processing Platform
Docker Deployment & Orchestration Guide
Version 1.0.0 | Docker Deployment v1.0

## 🐳 Docker Deployment Overview

Enterprise Docker Deployment: ALACTIC AGI Framework provides complete containerization with enterprise-grade monitoring, service discovery, and automated scaling capabilities.

### Deployment Benefits

- Production Ready: Pre-configured for enterprise environments
- Auto-Scaling: Dynamic resource allocation based on load
- Complete Observability: Prometheus + Grafana + AlertManager stack
- Service Discovery: Automatic component communication
- High Availability: Built-in redundancy and failover
- Security First: Network isolation and secure communication

### Container Architecture

Service | Container | Purpose | Port | Dependencies
--- | --- | --- | --- | ---
ALACTIC AGI Core | alactic-agi | Main processing framework | 5000 | Solr, Redis
API Gateway | alactic-api | REST API interface | 3000 | Core framework
Apache Solr | solr | Search & indexing | 8983 | None
Prometheus | prometheus | Metrics collection | 9090 | All services
Grafana | grafana | Monitoring dashboards | 3001 | Prometheus
AlertManager | alertmanager | Alert routing | 9093 | Prometheus
Node Exporter | node-exporter | System metrics | 9100 | None

## 📋 Prerequisites

#### 🐳 Docker Requirements

- Docker Engine 20.10.0+ (Latest recommended)
- Docker Compose v2.0+ (Plugin version)
- Docker Desktop (Windows/Mac)
- Minimum 8GB RAM allocated to Docker

#### 💻 System Resources

- CPU: 4+ cores (8+ recommended)
- Memory: 16GB+ RAM (32GB for production)
- Storage: 200GB+ available space
- Network: Internet connectivity

#### 🌐 Network Configuration

- Ports 3000-3001, 5000, 8983, 9090-9100 available
- Docker network: 172.20.0.0/16
- External access for monitoring dashboards
- DNS resolution for container communication

#### 🔧 Prerequisites Installation

PowerShell
# Install Docker Desktop (Windows) winget install Docker.DockerDesktop # Or download from: https://docker.com/products/docker-desktop # Verify installation docker --version docker-compose --version
Production Deployment: For enterprise production environments, allocate minimum 32GB RAM and ensure SSD storage for optimal performance with 1M+ documents.

## 🏗️ Docker Architecture

### Service Architecture Diagram

ASCII
┌─────────────────────────────────────────────────────────────────┐ │ ALACTIC AGI Docker Stack │ ├─────────────────────────────────────────────────────────────────┤ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Grafana │ │ AlertManager│ │ Prometheus │ │ │ │ :3001 │◄───┤ :9093 │◄───┤ :9090 │ │ │ └─────────────┘ └─────────────┘ └─────┬───────┘ │ │ │ │ │ ┌─────────────┐ ┌─────────────┐ │ │ │ │ ALACTIC API │ │ALACTIC Core │◄────────┼────────────────┤ │ │ :3000 │◄───┤ :5000 │ │ │ │ └─────────────┘ └──────┬──────┘ │ │ │ │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Solr │◄───┤ Redis │ │Node Exporter│ │ │ │ :8983 │ │ :6379 │ │ :9100 │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ └─────────────────────────────────────────────────────────────────┘ External Access Ports: 3000 (API), 3001 (Grafana), 9090 (Prometheus)

### Network Configuration

YAML
# Docker Compose Network Definition networks: alactic-network: driver: bridge ipam: config: - subnet: 172.20.0.0/16 # Service IP Assignment services: alactic-core: networks: alactic-network: ipv4_address: 172.20.0.10 solr: networks: alactic-network: ipv4_address: 172.20.0.20 prometheus: networks: alactic-network: ipv4_address: 172.20.0.30

### Storage Volumes

Volume | Mount Point | Purpose | Persistence
--- | --- | --- | ---
solr-data | /var/solr | Solr index and configuration | Persistent
prometheus-data | /prometheus | Metrics time-series data | Persistent
grafana-data | /var/lib/grafana | Dashboards and configurations | Persistent
alactic-logs | /app/logs | Application logs | Persistent

## 🚀 Deployment Guide

### Environment Preparation

PowerShell
# Clone the repository git clone https://github.com/AlacticAI/alactic-agi.git cd alactic-agi # Verify Docker is running docker info # Check available resources docker system df docker system events --since 5m
Resource Check: Ensure Docker has sufficient resources allocated before proceeding with deployment.

### Configuration Setup

#### Docker Compose Configuration

YAML
# docker-compose-monitoring-test.yml version: '3.8' services: # ALACTIC AGI Core Service alactic-agi: build: . container_name: alactic-agi-core ports: - "5000:5000" - "8080:8080" environment: - ALACTIC_DEBUG=false - ALACTIC_SOLR_URL=http://solr:8983/solr/super_rag - PROMETHEUS_ENABLED=true volumes: - ./data:/app/data - alactic-logs:/app/logs networks: - alactic-network depends_on: - solr - redis restart: unless-stopped healthcheck: test: ["CMD", "curl", "-f", "http://localhost:5000/health"] interval: 30s timeout: 10s retries: 3 # Apache Solr solr: image: solr:9-slim container_name: alactic-solr ports: - "8983:8983" volumes: - solr-data:/var/solr - ./solr_config:/opt/solr/server/solr/configsets/alactic_config networks: - alactic-network restart: unless-stopped command: solr-precreate super_rag /opt/solr/server/solr/configsets/alactic_config volumes: solr-data: prometheus-data: grafana-data: alactic-logs: networks: alactic-network: driver: bridge

#### Environment Variables Configuration

Bash
# Create .env file for production cat > .env << 'EOF' # ALACTIC AGI Configuration ALACTIC_DEBUG=false ALACTIC_LOG_LEVEL=INFO ALACTIC_MAX_WORKERS=8 # Database Configuration ALACTIC_SOLR_URL=http://solr:8983/solr/super_rag ALACTIC_REDIS_URL=redis://redis:6379/0 # Monitoring Configuration PROMETHEUS_ENABLED=true GRAFANA_ADMIN_PASSWORD=alactic_secure_admin_2024 ALERTMANAGER_EMAIL=alerts@alacticai.com # Security Configuration JWT_SECRET_KEY=your_jwt_secret_key_here API_RATE_LIMIT=1000 EOF

### Stack Deployment

#### Basic Deployment

PowerShell
# Deploy the complete stack docker-compose -f docker-compose-monitoring-test.yml up -d # Check deployment status docker-compose -f docker-compose-monitoring-test.yml ps # View logs docker-compose -f docker-compose-monitoring-test.yml logs -f

#### Production Deployment

PowerShell
# Production deployment with resource limits docker-compose -f docker-compose-monitoring.yml up -d # Scale specific services docker-compose -f docker-compose-monitoring.yml up -d --scale alactic-agi=3 # Deploy with specific environment docker-compose -f docker-compose-monitoring.yml --env-file .env.production up -d
Deployment Complete! All services should now be running. Check the status with docker-compose ps

### Service Verification

PowerShell
# Comprehensive service health check python validate_docker_stack.py # Individual service checks curl http://localhost:5000/health # ALACTIC AGI Core curl http://localhost:3000/api/health # API Gateway curl http://localhost:8983/solr/admin/ping # Solr curl http://localhost:9090/-/healthy # Prometheus curl http://localhost:3001/api/health # Grafana # Check service connectivity docker exec alactic-agi-core curl http://solr:8983/solr/admin/ping docker exec prometheus curl http://alactic-agi:8080/metrics

#### Expected Service Status

Service | URL | Expected Response | Status
--- | --- | --- | ---
ALACTIC Core | http://localhost:5000/health | {"status": "healthy"} | ✓ Running
API Gateway | http://localhost:3000/api/health | {"status": "ok"} | ✓ Running
Apache Solr | http://localhost:8983/solr/admin/ping | {"status": "OK"} | ✓ Running
Prometheus | http://localhost:9090/-/healthy | Prometheus is Healthy | ✓ Running
Grafana | http://localhost:3001/api/health | {"database": "ok"} | ✓ Running

## 📊 Enterprise Monitoring Stack

Complete Observability: ALACTIC AGI includes enterprise-grade monitoring with Prometheus, Grafana, and AlertManager for production observability.

### Monitoring Components

#### 📈 Prometheus Configuration

YAML
# monitoring/prometheus-simple.yml global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'alactic-agi' static_configs: - targets: ['alactic-agi:8080'] scrape_interval: 5s metrics_path: '/metrics' - job_name: 'node-exporter' static_configs: - targets: ['node-exporter:9100'] rule_files: - "/etc/prometheus/alerts.yml" alerting: alertmanagers: - static_configs: - targets: - alertmanager:9093

#### 🚨 AlertManager Setup

YAML
# monitoring/alertmanager/alertmanager-simple.yml global: smtp_smarthost: 'localhost:587' smtp_from: 'alerts@alacticai.com' route: group_by: ['alertname'] group_wait: 10s group_interval: 10s repeat_interval: 1h receiver: 'web.hook' receivers: - name: 'web.hook' email_configs: - to: 'support@alacticai.com' subject: 'ALACTIC AGI Alert: {{ .GroupLabels.alertname }}' body: | Alert: {{ .GroupLabels.alertname }} Instance: {{ .CommonLabels.instance }} Severity: {{ .CommonLabels.severity }} Description: {{ .CommonAnnotations.description }}

#### 📊 Grafana Dashboards

Pre-configured dashboards include:

- ALACTIC AGI Overview: Business metrics and KPIs
- System Performance: CPU, memory, disk usage
- Application Metrics: Request rates, response times
- Solr Monitoring: Query performance, index status
- Error Tracking: Error rates and failure analysis

Default Credentials:
Username: admin
Password: alactic_secure_admin_2024

#### 🔍 Node Exporter Metrics

System-level monitoring includes:

- CPU utilization and load averages
- Memory usage and swap activity
- Disk I/O and filesystem usage
- Network traffic and connections
- Process and service status

### Monitoring URLs

Service | URL | Purpose | Credentials
--- | --- | --- | ---
Grafana Dashboards | http://localhost:3001 | Visual monitoring dashboards | admin / alactic_secure_admin_2024
Prometheus UI | http://localhost:9090 | Metrics exploration and alerting | None required
AlertManager UI | http://localhost:9093 | Alert management and routing | None required
ALACTIC Metrics | http://localhost:8080/metrics | Raw Prometheus metrics | None required

### Custom Metrics Integration

Python
# Example: Adding custom business metrics from monitoring import MetricsCollector # Initialize metrics in your application metrics = MetricsCollector() # Record business metrics metrics.record_counter("documents_processed_total", 1, {"source": "web_crawler", "status": "success"}) metrics.record_gauge("processing_queue_size", queue_size) metrics.record_timer("document_processing_duration_seconds", processing_time, {"document_type": "research_paper"}) # Metrics automatically exposed at /metrics endpoint

## 🔧 Stack Management

### Daily Operations

#### Service Management Commands

PowerShell
# Start the entire stack docker-compose -f docker-compose-monitoring-test.yml up -d # Stop the entire stack docker-compose -f docker-compose-monitoring-test.yml down # Restart specific service docker-compose -f docker-compose-monitoring-test.yml restart alactic-agi # View service logs docker-compose -f docker-compose-monitoring-test.yml logs -f alactic-agi # Check service status docker-compose -f docker-compose-monitoring-test.yml ps # Update and restart services docker-compose -f docker-compose-monitoring-test.yml pull docker-compose -f docker-compose-monitoring-test.yml up -d

#### Resource Monitoring

PowerShell
# Monitor resource usage docker stats # Check disk usage docker system df # View container details docker inspect alactic-agi-core # Check network connectivity docker network ls docker network inspect alactic-agi_alactic-network # Backup volumes docker run --rm -v alactic-agi_solr-data:/data -v ${PWD}:/backup alpine tar czf /backup/solr-backup.tar.gz -C /data .

### Data Management

#### Backup Operations

PowerShell
# Create backup script cat > backup_alactic.ps1 << 'EOF' $BackupDir = "C:\ALACTIC_Backups\$(Get-Date -Format 'yyyy-MM-dd_HH-mm-ss')" New-Item -ItemType Directory -Path $BackupDir -Force # Backup Solr data docker run --rm -v alactic-agi_solr-data:/data -v "${BackupDir}:/backup" alpine tar czf /backup/solr-data.tar.gz -C /data . # Backup Prometheus data docker run --rm -v alactic-agi_prometheus-data:/data -v "${BackupDir}:/backup" alpine tar czf /backup/prometheus-data.tar.gz -C /data . # Backup Grafana data docker run --rm -v alactic-agi_grafana-data:/data -v "${BackupDir}:/backup" alpine tar czf /backup/grafana-data.tar.gz -C /data . # Backup configuration files Copy-Item -Path ".\docker-compose-monitoring-test.yml" -Destination "${BackupDir}\" Copy-Item -Path ".\config.ini" -Destination "${BackupDir}\" Copy-Item -Path ".\.env" -Destination "${BackupDir}\" Write-Host "Backup completed: $BackupDir" EOF # Run backup .\backup_alactic.ps1

#### Restore Operations

PowerShell
# Restore from backup $BackupDir = "C:\ALACTIC_Backups\2024-01-15_14-30-00" # Stop services docker-compose -f docker-compose-monitoring-test.yml down # Restore volumes docker run --rm -v alactic-agi_solr-data:/data -v "${BackupDir}:/backup" alpine tar xzf /backup/solr-data.tar.gz -C /data docker run --rm -v alactic-agi_prometheus-data:/data -v "${BackupDir}:/backup" alpine tar xzf /backup/prometheus-data.tar.gz -C /data docker run --rm -v alactic-agi_grafana-data:/data -v "${BackupDir}:/backup" alpine tar xzf /backup/grafana-data.tar.gz -C /data # Restart services docker-compose -f docker-compose-monitoring-test.yml up -d

### Security Management

#### Container Security

PowerShell
# Security audit docker run --rm -v /var/run/docker.sock:/var/run/docker.sock aquasec/docker-bench # Check for vulnerabilities docker run --rm -v /var/run/docker.sock:/var/run/docker.sock quay.io/coreos/clair-local-scan # Update container security docker-compose -f docker-compose-monitoring-test.yml pull docker-compose -f docker-compose-monitoring-test.yml up -d # Review security logs docker-compose -f docker-compose-monitoring-test.yml logs | grep -i "security\|error\|fail"

#### Network Security

PowerShell
# Check exposed ports netstat -tlnp | grep -E "(3000|3001|5000|8983|9090)" # Firewall configuration (Ubuntu/CentOS) ufw allow 3000/tcp comment "ALACTIC API" ufw allow 3001/tcp comment "Grafana Dashboard" ufw allow 9090/tcp comment "Prometheus" # Block internal ports from external access ufw deny 5000/tcp comment "ALACTIC Core - Internal Only" ufw deny 8983/tcp comment "Solr - Internal Only"

## 📈 Scaling & Performance

Enterprise Scaling: ALACTIC AGI supports horizontal and vertical scaling for enterprise workloads processing millions of documents.

### Horizontal Scaling

#### Multi-Instance Deployment

PowerShell
# Scale ALACTIC AGI core service docker-compose -f docker-compose-monitoring-test.yml up -d --scale alactic-agi=3 # Scale with load balancer cat > docker-compose-scaled.yml << 'EOF' version: '3.8' services: nginx: image: nginx:alpine ports: - "80:80" volumes: - ./nginx.conf:/etc/nginx/nginx.conf depends_on: - alactic-agi-1 - alactic-agi-2 - alactic-agi-3 networks: - alactic-network alactic-agi-1: extends: file: docker-compose-monitoring-test.yml service: alactic-agi container_name: alactic-agi-1 alactic-agi-2: extends: file: docker-compose-monitoring-test.yml service: alactic-agi container_name: alactic-agi-2 alactic-agi-3: extends: file: docker-compose-monitoring-test.yml service: alactic-agi container_name: alactic-agi-3 EOF # Deploy scaled configuration docker-compose -f docker-compose-scaled.yml up -d

#### Load Balancer Configuration

Nginx
# nginx.conf for load balancing events { worker_connections 1024; } http { upstream alactic_backend { server alactic-agi-1:5000; server alactic-agi-2:5000; server alactic-agi-3:5000; # Health checks keepalive 32; } server { listen 80; location / { proxy_pass http://alactic_backend; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # Timeouts proxy_connect_timeout 30s; proxy_send_timeout 30s; proxy_read_timeout 30s; } location /health { access_log off; return 200 "healthy\n"; add_header Content-Type text/plain; } } }

### Vertical Scaling

#### Resource Limits Configuration

YAML
# Production resource limits services: alactic-agi: deploy: resources: limits: cpus: '4.0' memory: 8G reservations: cpus: '2.0' memory: 4G environment: - ALACTIC_MAX_WORKERS=8 - JAVA_OPTS=-Xmx4g -Xms2g solr: deploy: resources: limits: cpus: '2.0' memory: 4G reservations: cpus: '1.0' memory: 2G environment: - SOLR_HEAP=3g

### Performance Optimization

#### Docker Performance Tuning

PowerShell
# Docker daemon optimization (daemon.json) { "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" }, "storage-driver": "overlay2", "storage-opts": [ "overlay2.override_kernel_check=true" ], "default-ulimits": { "nofile": { "Name": "nofile", "Hard": 64000, "Soft": 64000 } } } # Restart Docker daemon Restart-Service docker

#### Application Performance Tuning

INI
# config.ini for high-performance deployment [DEFAULT] debug = false log_level = WARNING max_workers = 16 batch_size = 1000 [DATABASE] solr_url = http://solr:8983/solr/super_rag solr_timeout = 60 connection_pool_size = 20 [SCRAPING] max_pages = 100000 delay = 0.5 concurrent_requests = 16 download_timeout = 30 [MONITORING] metrics_enabled = true high_performance_mode = true metrics_buffer_size = 10000

### Auto-Scaling with Docker Swarm

PowerShell
# Initialize Docker Swarm docker swarm init # Deploy stack with auto-scaling docker stack deploy -c docker-compose-swarm.yml alactic # Configure auto-scaling rules docker service update --replicas 5 alactic_alactic-agi docker service update --limit-cpu 2 --limit-memory 4g alactic_alactic-agi # Monitor service scaling docker service ls docker service ps alactic_alactic-agi

## 🔧 Troubleshooting

### Common Docker Issues

Before troubleshooting: Always run python validate_docker_stack.py for comprehensive diagnostics.

Issue | Symptoms | Solution | Prevention
--- | --- | --- | ---
Container fails to start | Exit code 1, restart loops | Check logs: `docker logs container_name` | Validate configuration before deployment
Service communication failure | Connection refused, timeouts | Verify network: `docker network inspect` | Use service names for inter-container communication
Out of memory errors | OOMKilled, performance degradation | Increase memory limits or optimize application | Monitor memory usage with `docker stats`
Port binding conflicts | Port already in use errors | Check port usage: `netstat -tlnp` | Use Docker networks instead of host ports

### Diagnostic Commands

#### Comprehensive Stack Diagnosis

PowerShell
# Complete stack validation python validate_docker_stack.py # Service-specific diagnostics docker-compose -f docker-compose-monitoring-test.yml ps docker-compose -f docker-compose-monitoring-test.yml logs --tail=100 # Network connectivity testing docker exec alactic-agi-core ping solr docker exec alactic-agi-core curl http://solr:8983/solr/admin/ping docker exec prometheus curl http://alactic-agi:8080/metrics # Resource usage monitoring docker stats --no-stream docker system df docker system events --since 10m

#### Container Health Checks

PowerShell
# Individual container health checks docker exec alactic-agi-core curl -f http://localhost:5000/health docker exec alactic-solr curl -f http://localhost:8983/solr/admin/ping docker exec prometheus curl -f http://localhost:9090/-/healthy docker exec grafana curl -f http://localhost:3001/api/health # Check container processes docker exec alactic-agi-core ps aux docker exec alactic-agi-core netstat -tlnp # Inspect container configuration docker inspect alactic-agi-core | jq '.[] | {State, NetworkSettings, Mounts}'

### Performance Issues

#### Memory and CPU Optimization

PowerShell
# Monitor resource usage docker stats --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}" # Optimize memory usage docker-compose -f docker-compose-monitoring-test.yml restart alactic-agi # Check for memory leaks docker exec alactic-agi-core python -c " import psutil process = psutil.Process() print(f'Memory: {process.memory_info().rss / 1024 / 1024:.2f} MB') print(f'CPU: {process.cpu_percent()}%') " # Garbage collection docker exec alactic-agi-core python -c "import gc; gc.collect(); print('GC completed')"

#### Storage Issues

PowerShell
# Check disk usage docker system df -v # Clean up unused resources docker system prune -f docker volume prune -f docker image prune -f # Check volume usage docker exec alactic-solr df -h /var/solr docker exec prometheus df -h /prometheus # Monitor log file sizes docker exec alactic-agi-core du -sh /app/logs/* docker-compose -f docker-compose-monitoring-test.yml logs --tail=1 | wc -l

### Network Troubleshooting

#### Service Discovery Issues

PowerShell
# Check DNS resolution docker exec alactic-agi-core nslookup solr docker exec alactic-agi-core nslookup prometheus # Network connectivity testing docker exec alactic-agi-core telnet solr 8983 docker exec prometheus telnet alactic-agi 8080 # Inspect network configuration docker network ls docker network inspect alactic-agi_alactic-network # Check iptables rules (Linux) docker exec alactic-agi-core iptables -L -n

### Recovery Procedures

#### Emergency Recovery

PowerShell
# Emergency stack restart docker-compose -f docker-compose-monitoring-test.yml down docker system prune -f docker-compose -f docker-compose-monitoring-test.yml up -d # Restore from backup $BackupDir = "C:\ALACTIC_Backups\latest" docker-compose -f docker-compose-monitoring-test.yml down -v docker run --rm -v alactic-agi_solr-data:/data -v "${BackupDir}:/backup" alpine tar xzf /backup/solr-data.tar.gz -C /data docker-compose -f docker-compose-monitoring-test.yml up -d # Validate recovery python validate_docker_stack.py

### Getting Support

Enterprise Support Channels:

- 📧 Email: support@alacticai.com
- 🎫 Support Portal: support.alacticai.com
- 📞 Emergency Hotline: +1-800-ALACTIC (24/7 Enterprise Support)
- 💬 Slack: #alactic-support (Enterprise customers)