# Augint Zero-Touch Billing System - Complete CLI Guide

## Executive Summary

The Augint Billing System implements **Zero-Touch Architecture** - a fully automated billing infrastructure where APIs are discovered and attached to usage plans without manual intervention. This guide covers the complete CLI toolset for managing, monitoring, and troubleshooting this automated system.

## Core Philosophy: Zero-Touch Operations

### What Zero-Touch Means
- **No Manual API Registration**: APIs are discovered automatically every 5 minutes
- **No Manual Plan Assignment**: APIs attach to usage plans based on tags
- **No Manual User Management**: Stripe handles all customer operations
- **No Manual Billing**: Usage reporting and invoicing are fully automated

### The Three Pillars

```mermaid
flowchart TB
    subgraph Discovery["🔍 Auto-Discovery (Every 5 min)"]
        A1[EventBridge Schedule] --> B1[ApiDiscovery Lambda]
        B1 --> C1[Scan for REST APIs]
        C1 --> D1[Auto-attach to Plans]
    end

    subgraph Events["⚡ Event Processing (Real-time)"]
        A2[Stripe Webhook] --> B2[EventBridge Partner]
        B2 --> C2[StripeEventHandler]
        C2 --> D2[Move Keys Between Plans]
    end

    subgraph Usage["📊 Usage Reporting (Hourly)"]
        A3[EventBridge Schedule] --> B3[UsageReporter Lambda]
        B3 --> C3[Query Metered Usage]
        C3 --> D3[Report to Stripe]
    end

    style B1 fill:#e8f5e9
    style C2 fill:#cfe2ff
    style B3 fill:#fff3cd
```

## Customer Journey

```mermaid
stateDiagram-v2
    [*] --> FreeUser: Signs Up (Gets API Key)

    FreeUser --> PayingCustomer: Adds Payment Method
    PayingCustomer --> ChurnedCustomer: Payment Fails/Cancels
    ChurnedCustomer --> PayingCustomer: Re-subscribes

    note right of FreeUser
        • API Key in FREE_10K plan
        • 10K calls/month limit
        • No Stripe record
    end note

    note right of PayingCustomer
        • API Key in METERED plan
        • Unlimited usage (pay-as-you-go)
        • Linked in DynamoDB
        • Usage reported to Stripe
    end note

    note right of ChurnedCustomer
        • API Key back in FREE_10K
        • Still in DynamoDB
        • Usage stops reporting
    end note
```

## CLI Command Structure

The CLI is organized by operational domain:

```
ai-billing/
├── infra/         # Infrastructure management
├── monitor/       # Real-time monitoring
├── aws/           # AWS service operations
├── billing/       # Billing operations
├── validate/      # Configuration validation
├── test/          # Testing utilities
├── setup/         # Initial configuration
└── admin/         # Administrative tasks
```

## Part 1: Infrastructure Management (`infra/`)

These commands manage the Zero-Touch infrastructure components.

### Check Infrastructure Status
```bash
# View complete infrastructure health
ai-billing infra status

# Output includes:
# - CloudFormation stack status
# - Lambda function states
# - EventBridge rule status
# - Usage plan configurations
```

### Manually Trigger Discovery
```bash
# Trigger API discovery immediately (normally runs every 5 min)
ai-billing infra trigger-discovery

# Wait for completion and see results
ai-billing infra trigger-discovery --wait --show-output
```

### List Discovered APIs
```bash
# View all discovered APIs and their attachment status
ai-billing infra list-apis

# Filter by specific plan
ai-billing infra list-apis --plan METERED

# Show detailed information
ai-billing infra list-apis --detailed
```

### Verify API Attachments
```bash
# Verify all APIs are correctly attached
ai-billing infra verify

# Attempt to fix issues automatically
ai-billing infra verify --fix

# Output as JSON for automation
ai-billing infra verify --json
```

### Emergency Manual Operations
```bash
# Manual attachment (emergency use only!)
ai-billing infra attach --api-id abc123 --stage prod --plan METERED

# Manual detachment
ai-billing infra detach --api-id abc123 --stage prod

# Pause auto-discovery
ai-billing infra pause --component discovery

# Resume auto-discovery
ai-billing infra resume --component discovery
```

## Part 2: Monitoring (`monitor/`)

Real-time visibility into Zero-Touch operations.

### Live Dashboard
```bash
# Real-time monitoring dashboard
ai-billing monitor dashboard

# Auto-refresh every 30 seconds
ai-billing monitor dashboard --refresh 30

# The dashboard shows:
# - EventBridge rule states (ENABLED/DISABLED)
# - Lambda invocation counts (last hour)
# - Recent API discoveries (last hour)
# - API Gateway metrics
# - Active alerts
```

Example Dashboard Output:
```
================================================================================
Zero-Touch Monitoring Dashboard - augint-billing-prod
================================================================================
Last Updated: 2024-01-15 10:30:45 UTC

📅 EventBridge Rules:
  • API Discovery (5 min)              ENABLED
  • Usage Reporting (hourly)           ENABLED
  • Stripe Events                      ENABLED

⚡ Lambda Activity (last hour):
  • API Discovery                      12 invocations
  • Usage Reporting                    1 invocation
  • Stripe Events                      3 invocations

🔍 Recent API Discoveries (last hour):
  • api-xyz123 (prod) → METERED (2m ago)
  • api-abc456 (staging) → FREE_10K (7m ago)

📊 API Gateway Metrics:
  • Total API calls (last hour): 45,234
  • Active API keys: 127
  • Keys in METERED plan: 23
```

## Part 3: AWS Operations (`aws/`)

Direct interaction with AWS services.

### Lambda Operations
```bash
# List all Lambda functions
ai-billing aws lambda list

# Invoke a specific function
ai-billing aws lambda invoke --function augint-billing-ApiDiscovery

# View recent logs
ai-billing aws lambda logs --function augint-billing-UsageReporter --since 1h

# Check for errors
ai-billing aws lambda errors --function augint-billing-StripeEventHandler
```

### EventBridge Management
```bash
# List all rules
ai-billing aws eventbridge list-rules

# View rule details
ai-billing aws eventbridge describe --rule ApiDiscoverySchedule

# Disable a rule (emergency)
ai-billing aws eventbridge disable --rule ApiDiscoverySchedule

# Enable a rule
ai-billing aws eventbridge enable --rule ApiDiscoverySchedule
```

### API Gateway Operations
```bash
# List all REST APIs
ai-billing aws apigateway list-apis

# View usage plans
ai-billing aws apigateway list-plans

# Check API attachments
ai-billing aws apigateway list-attachments --plan-id METERED
```

### CloudWatch Monitoring
```bash
# View Lambda logs
ai-billing aws cloudwatch logs --group /aws/lambda/augint-billing --since 30m

# Query specific errors
ai-billing aws cloudwatch logs --group /aws/lambda/augint-billing --filter ERROR

# View metrics
ai-billing aws cloudwatch metrics --namespace AWS/Lambda --metric Invocations
```

## Part 4: Billing Operations (`billing/`)

Core billing functionality.

### Process Stripe Events
```bash
# Process a webhook event from file (testing)
ai-billing billing process --file events/customer.subscription.created.json

# Show what would happen without executing
ai-billing billing process --file events/payment_method.attached.json --dry-run
```

### Sync Usage to Stripe
```bash
# Report usage for the last hour
ai-billing billing sync-usage

# Report for a specific time window
ai-billing billing sync-usage --since "2024-01-15T00:00:00Z" --until "2024-01-15T01:00:00Z"

# Dry run to see what would be reported
ai-billing billing sync-usage --dry-run
```

### Environment Configuration
```bash
# Display current configuration
ai-billing billing env-dump

# Output:
# STACK_NAME=augint-billing-prod
# TABLE_NAME=augint-billing-prod-customer-links
# FREE_USAGE_PLAN_ID=FREE_10K
# METERED_USAGE_PLAN_ID=METERED
# STRIPE_SECRET_ARN=arn:aws:secretsmanager:...
```

## Part 5: Validation (`validate/`)

Ensure system configuration is correct.

### Validate Configuration
```bash
# Complete system validation
ai-billing validate config

# Validate specific components
ai-billing validate config --component stripe
ai-billing validate config --component dynamodb
ai-billing validate config --component apigateway

# Fix issues automatically where possible
ai-billing validate config --fix
```

### Health Checks
```bash
# Run all health checks
ai-billing validate health

# Check specific subsystem
ai-billing validate health --component billing
```

## Part 6: Testing (`test/`)

Test system behavior without affecting production.

### Integration Testing
```bash
# Run a complete billing cycle test
ai-billing test integration cycle

# Test webhook processing
ai-billing test integration webhook --event customer.subscription.created

# Test API key linking
ai-billing test integration link --api-key test_key_123 --customer cus_test
```

### Simulation
```bash
# Simulate customer journey
ai-billing test simulate journey --scenario new-customer

# Simulate usage reporting
ai-billing test simulate usage --api-key test_key --calls 5000
```

## Part 7: Initial Setup (`setup/`)

One-time configuration tasks.

### Create Stripe Products
```bash
# Create API usage product in Stripe
ai-billing setup stripe-product

# Verify Stripe configuration
ai-billing setup verify --stripe
```

### Configure EventBridge
```bash
# Set up Stripe partner event source
ai-billing setup eventbridge --stripe-account acct_123

# Verify EventBridge configuration
ai-billing setup verify --eventbridge
```

## Part 8: Administrative Operations (`admin/`)

Advanced administrative tasks.

### Manage Customer Links
```bash
# List all customer links
ai-billing admin list-links

# Remove orphaned link
ai-billing admin remove-link --api-key old_key_123

# Migrate customer to new API key
ai-billing admin migrate-customer --from old_key --to new_key
```

### Audit Operations
```bash
# View audit log
ai-billing admin audit --since 7d

# Export audit data
ai-billing admin audit --export audit-report.json
```

## Common Scenarios and Solutions

### Scenario 1: API Not Being Discovered
```bash
# 1. Check if discovery is running
ai-billing infra status

# 2. Manually trigger discovery
ai-billing infra trigger-discovery --wait --show-output

# 3. Check Lambda logs for errors
ai-billing aws lambda errors --function ApiDiscovery

# 4. Verify API is a REST API (not HTTP API)
ai-billing aws apigateway list-apis --type REST
```

### Scenario 2: Usage Not Reporting to Stripe
```bash
# 1. Check customer link exists
ai-billing admin list-links --api-key key_123

# 2. Verify API key is in METERED plan
ai-billing infra list-apis --api-key key_123

# 3. Check usage reporter logs
ai-billing aws lambda logs --function UsageReporter --filter key_123

# 4. Manually sync usage
ai-billing billing sync-usage --api-key key_123
```

### Scenario 3: Customer Stuck in Wrong Plan
```bash
# 1. Check current status
ai-billing validate customer --stripe-id cus_123

# 2. Process latest Stripe events
ai-billing billing process --customer cus_123 --latest

# 3. Emergency manual move (last resort!)
ai-billing infra attach --api-key key_123 --plan METERED
```

### Scenario 4: System Health Check
```bash
# Complete health assessment
ai-billing validate health
ai-billing infra status
ai-billing monitor dashboard --refresh 10
```

## Environment Variables

### Required
- `STACK_NAME`: CloudFormation stack name
- `AWS_REGION`: AWS region (default: us-east-1)
- `STRIPE_SECRET_KEY` or `STRIPE_SECRET_ARN`: Stripe authentication

### Optional
- `TABLE_NAME`: DynamoDB table (default: `{STACK_NAME}-customer-links`)
- `FREE_USAGE_PLAN_ID`: Free tier plan (default: FREE_10K)
- `METERED_USAGE_PLAN_ID`: Paid tier plan (default: METERED)
- `LOG_LEVEL`: Logging verbosity (DEBUG, INFO, WARNING, ERROR)

## Best Practices

### 1. Monitoring
- Always run `monitor dashboard` during deployments
- Set up CloudWatch alarms for Lambda errors
- Review audit logs weekly

### 2. Troubleshooting
- Start with `validate config` for configuration issues
- Check `infra status` for infrastructure problems
- Use `--dry-run` before making changes

### 3. Emergency Response
- Use manual operations (`infra attach/detach`) only as last resort
- Always document manual interventions
- Re-enable automation ASAP after manual fixes

### 4. Testing
- Test all webhook events in staging first
- Use `--dry-run` for usage sync testing
- Validate configuration after any changes

## Architecture Diagrams

### System Components
```mermaid
graph TB
    subgraph External
        S[Stripe Platform]
        C[Customers]
        A[API Clients]
    end

    subgraph AWS
        EB[EventBridge]
        L1[ApiDiscovery Lambda]
        L2[StripeEventHandler Lambda]
        L3[UsageReporter Lambda]
        DDB[(DynamoDB)]
        APIGW[API Gateway]
        SM[Secrets Manager]
    end

    S -->|Webhooks| EB
    EB -->|Schedule| L1
    EB -->|Events| L2
    EB -->|Schedule| L3
    L1 -->|Attach| APIGW
    L2 -->|Update| DDB
    L3 -->|Read| DDB
    L3 -->|Report| S
    A -->|Calls| APIGW
    APIGW -->|Usage| L3
```

### Data Flow
```mermaid
sequenceDiagram
    participant Customer
    participant Stripe
    participant EventBridge
    participant Handler
    participant DynamoDB
    participant APIGateway

    Customer->>Stripe: Add payment method
    Stripe->>EventBridge: setup_intent.succeeded
    EventBridge->>Handler: Process event
    Handler->>DynamoDB: Create/update link
    Handler->>APIGateway: Move to METERED plan
    APIGateway-->>Customer: Higher rate limits
```

## Troubleshooting Guide

### Lambda Not Executing
1. Check EventBridge rule is ENABLED: `ai-billing infra status`
2. Verify Lambda has execution role: `ai-billing aws lambda describe`
3. Check CloudWatch logs for errors: `ai-billing aws lambda errors`

### APIs Not Discovered
1. Ensure APIs are REST type (not HTTP)
2. Check discovery Lambda logs
3. Verify IAM permissions for API Gateway access
4. Manually trigger discovery: `ai-billing infra trigger-discovery`

### Usage Not Reporting
1. Verify customer link exists in DynamoDB
2. Check API key is in METERED plan
3. Ensure usage reporter Lambda is running
4. Look for idempotency key conflicts in logs

### Stripe Events Not Processing
1. Verify EventBridge partner connection
2. Check webhook endpoint in Stripe dashboard
3. Review StripeEventHandler logs
4. Test with manual event: `ai-billing billing process --file event.json`

## Migration from Manual Operations

If you're migrating from the old manual promote/demote system:

1. **Stop using manual commands** - They no longer exist
2. **Deploy Zero-Touch infrastructure** - EventBridge rules and Lambdas
3. **Let discovery run** - APIs will be found automatically
4. **Monitor the transition** - Use `monitor dashboard`
5. **Validate final state** - Run `infra verify`

## Support and Resources

### Getting Help
- Check system status: `ai-billing validate health`
- View recent errors: `ai-billing aws lambda errors --all`
- Review audit log: `ai-billing admin audit --since 1d`

### Documentation
- API Documentation: See Stripe API docs
- AWS Services: Lambda, EventBridge, API Gateway docs
- Architecture: See system design documents

### Emergency Contacts
- On-call: Check PagerDuty
- Stripe Support: dashboard.stripe.com/support
- AWS Support: AWS Console Support Center

## Appendix: Complete Command Reference

### Infrastructure Commands
```bash
ai-billing infra status [--json]
ai-billing infra trigger-discovery [--wait] [--show-output]
ai-billing infra list-apis [--plan PLAN] [--detailed]
ai-billing infra verify [--fix] [--json]
ai-billing infra attach --api-id ID --stage STAGE --plan PLAN
ai-billing infra detach --api-id ID --stage STAGE
ai-billing infra pause --component COMPONENT
ai-billing infra resume --component COMPONENT
```

### Monitoring Commands
```bash
ai-billing monitor dashboard [--refresh SECONDS]
```

### AWS Commands
```bash
ai-billing aws lambda list
ai-billing aws lambda invoke --function NAME [--payload JSON]
ai-billing aws lambda logs --function NAME [--since TIME]
ai-billing aws lambda errors --function NAME
ai-billing aws eventbridge list-rules
ai-billing aws eventbridge describe --rule NAME
ai-billing aws apigateway list-apis
ai-billing aws apigateway list-plans
ai-billing aws cloudwatch logs --group GROUP [--filter PATTERN]
```

### Billing Commands
```bash
ai-billing billing process --file FILE [--dry-run]
ai-billing billing sync-usage [--since TIME] [--until TIME] [--dry-run]
ai-billing billing env-dump
```

---

*Last Updated: 2024-01-15*
*Version: 2.0.0 - Zero-Touch Architecture*
