# Storage Infrastructure Guide

## Storage Types Overview

| Type | Best For | Examples | Latency | Cost |
|------|----------|----------|---------|------|
| **Object Storage** | Files, media, backups | S3, R2, GCS | Medium | Low |
| **Block Storage** | Databases, VMs | EBS, Persistent Disks | Low | Medium |
| **File Storage** | Shared file systems | EFS, Filestore | Medium | Medium |
| **Database Storage** | Structured data | RDS, Supabase, PlanetScale | Very Low | Varies |
| **Cache/KV** | Session data, hot data | Redis, Cloudflare KV | Ultra-Low | Low |

## Object Storage

### Cloudflare R2 (Recommended)

**Why R2:**
- **Zero egress fees** (vs S3's $0.09/GB)
- S3-compatible API
- Global distribution
- Significantly cheaper at scale

**Pricing (2025-01):**
- Storage: $0.015/GB/month
- Class A operations (writes): $4.50/million
- Class B operations (reads): $0.36/million
- Egress: **$0** (this is the killer feature)

**Pulumi Setup:**

```python
import pulumi
import pulumi_cloudflare as cloudflare

# Create R2 bucket
uploads_bucket = cloudflare.R2Bucket(
    "uploads",
    account_id=cloudflare_account_id,
    name=f"{project_name}-uploads-{stack}",
)

# Create bucket for backups
backups_bucket = cloudflare.R2Bucket(
    "backups",
    account_id=cloudflare_account_id,
    name=f"{project_name}-backups-{stack}",
)

pulumi.export("uploads_bucket", uploads_bucket.name)
```

**Access from Workers:**

```typescript
// workers/upload.ts
export interface Env {
  UPLOADS: R2Bucket;
}

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    if (request.method === 'PUT') {
      const key = new URL(request.url).pathname.slice(1);
      await env.UPLOADS.put(key, request.body);
      return new Response('Uploaded', { status: 201 });
    }

    if (request.method === 'GET') {
      const key = new URL(request.url).pathname.slice(1);
      const object = await env.UPLOADS.get(key);

      if (!object) {
        return new Response('Not Found', { status: 404 });
      }

      return new Response(object.body, {
        headers: {
          'Content-Type': object.httpMetadata?.contentType || 'application/octet-stream',
        },
      });
    }

    return new Response('Method Not Allowed', { status: 405 });
  },
};
```

### AWS S3

**When to use S3 instead of R2:**
- Deep AWS integration needed
- S3-specific features (S3 Select, Glacier)
- Existing AWS infrastructure

**Pulumi Setup:**

```python
import pulumi
import pulumi_aws as aws

bucket = aws.s3.Bucket(
    "uploads",
    bucket=f"{project_name}-uploads-{stack}",
    versioning=aws.s3.BucketVersioningArgs(enabled=True),
    server_side_encryption_configuration=aws.s3.BucketServerSideEncryptionConfigurationArgs(
        rule=aws.s3.BucketServerSideEncryptionConfigurationRuleArgs(
            apply_server_side_encryption_by_default=aws.s3.BucketServerSideEncryptionConfigurationRuleApplyServerSideEncryptionByDefaultArgs(
                sse_algorithm="AES256",
            ),
        ),
    ),
    lifecycle_rules=[
        aws.s3.BucketLifecycleRuleArgs(
            enabled=True,
            expiration=aws.s3.BucketLifecycleRuleExpirationArgs(days=90),
            prefix="temp/",
        ),
        aws.s3.BucketLifecycleRuleArgs(
            enabled=True,
            noncurrent_version_expiration=aws.s3.BucketLifecycleRuleNoncurrentVersionExpirationArgs(
                days=30,
            ),
        ),
    ],
    tags=get_standard_tags(),
)

# Block public access (security best practice)
aws.s3.BucketPublicAccessBlock(
    "uploads-public-block",
    bucket=bucket.id,
    block_public_acls=True,
    block_public_policy=True,
    ignore_public_acls=True,
    restrict_public_buckets=True,
)
```

### Supabase Storage

**Best for:** User uploads integrated with Supabase Auth

```sql
-- Create storage bucket
INSERT INTO storage.buckets (id, name, public)
VALUES ('avatars', 'avatars', true);

-- Create private bucket
INSERT INTO storage.buckets (id, name, public)
VALUES ('documents', 'documents', false);

-- RLS policy: Users can upload to their own folder
CREATE POLICY "Users can upload own avatar"
ON storage.objects FOR INSERT
WITH CHECK (
  bucket_id = 'avatars' AND
  auth.uid()::text = (storage.foldername(name))[1]
);

-- RLS policy: Users can read their own documents
CREATE POLICY "Users can read own documents"
ON storage.objects FOR SELECT
USING (
  bucket_id = 'documents' AND
  auth.uid()::text = (storage.foldername(name))[1]
);
```

**Usage from application:**

```typescript
import { createClient } from '@supabase/supabase-js'

const supabase = createClient(url, anonKey)

// Upload file
const { data, error } = await supabase.storage
  .from('avatars')
  .upload(`${userId}/avatar.png`, file)

// Get public URL
const { data: { publicUrl } } = supabase.storage
  .from('avatars')
  .getPublicUrl(`${userId}/avatar.png`)

// Download private file
const { data, error } = await supabase.storage
  .from('documents')
  .download(`${userId}/report.pdf`)
```

## Database Storage

### Supabase (PostgreSQL)

**Best for:** Full-stack apps needing Auth + Storage + Real-time

See: [supabase.md](./supabase.md) for complete guide

**Quick setup:**

```bash
# Create project
supabase projects create myapp --region us-east-1

# Run migrations
supabase db push
```

### PlanetScale (MySQL)

**Best for:** Serverless MySQL with branching

```bash
# Install CLI
brew install planetscale/tap/pscale

# Create database
pscale database create myapp --region us-east

# Create branch for development
pscale branch create myapp feature-xyz

# Deploy schema changes
pscale deploy-request create myapp feature-xyz
```

### AWS RDS (PostgreSQL/MySQL)

**Best for:** Full control, enterprise compliance

```python
import pulumi_aws as aws

# Create RDS instance
database = aws.rds.Instance(
    "main",
    instance_class=db_instance_class,  # db.t3.micro for dev
    allocated_storage=20,
    engine="postgres",
    engine_version="15",
    db_name="myapp",
    username="admin",
    password=db_password,  # Pulumi secret
    skip_final_snapshot=stack != "production",
    backup_retention_period=30 if stack == "production" else 7,
    multi_az=stack == "production",
    storage_encrypted=True,
    vpc_security_group_ids=[db_security_group.id],
    db_subnet_group_name=db_subnet_group.name,
    tags=get_standard_tags(),
)
```

## Cache and KV Storage

### Cloudflare KV

**Best for:** Global key-value storage at the edge

```typescript
// wrangler.toml
[[kv_namespaces]]
binding = "CACHE"
id = "xxx"

// Worker code
export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const key = 'user:123'

    // Read
    const cached = await env.CACHE.get(key)
    if (cached) return new Response(cached)

    // Write
    const data = await fetchFromDatabase()
    await env.CACHE.put(key, JSON.stringify(data), {
      expirationTtl: 3600, // 1 hour
    })

    return new Response(JSON.stringify(data))
  },
}
```

### Redis (Upstash)

**Best for:** Serverless Redis with persistence

```python
# With Pulumi (using Upstash provider)
import pulumi_upstash as upstash

redis = upstash.RedisDatabase(
    "cache",
    database_name=f"{project_name}-{stack}",
    region="us-east-1",
    tls=True,
)

pulumi.export("redis_url", redis.endpoint)
```

**Usage:**

```python
import redis
import os

r = redis.from_url(os.getenv("REDIS_URL"))

# Set with expiration
r.setex("session:abc", 3600, "user_data")

# Get
data = r.get("session:abc")
```

## Storage Patterns

### Pattern 1: Upload Flow with Signed URLs

```python
"""Secure file upload with signed URLs."""

import boto3
from datetime import datetime, timedelta


def generate_upload_url(bucket: str, key: str, expires_in: int = 3600) -> str:
    """Generate presigned URL for upload.

    Args:
        bucket: S3/R2 bucket name
        key: Object key (path)
        expires_in: URL expiration in seconds

    Returns:
        Presigned URL for PUT request
    """
    s3 = boto3.client('s3')

    url = s3.generate_presigned_url(
        'put_object',
        Params={'Bucket': bucket, 'Key': key},
        ExpiresIn=expires_in,
    )

    return url


# API endpoint
@app.post("/upload-url")
async def get_upload_url(filename: str, user_id: str):
    """Get presigned URL for file upload."""
    key = f"uploads/{user_id}/{datetime.now().isoformat()}/{filename}"
    url = generate_upload_url("my-bucket", key)
    return {"upload_url": url, "key": key}
```

### Pattern 2: Multi-Tier Storage

```python
"""Multi-tier storage for cost optimization."""

# Tier 1: Hot storage (frequent access)
# - Cloudflare R2 or S3 Standard
# - Use for recent uploads, active files

# Tier 2: Warm storage (occasional access)
# - S3 Infrequent Access
# - Use for older files, archives

# Tier 3: Cold storage (rare access)
# - S3 Glacier
# - Use for compliance, long-term backups

# Lifecycle policy example
lifecycle_rules = [
    # Move to IA after 30 days
    aws.s3.BucketLifecycleRuleArgs(
        enabled=True,
        prefix="uploads/",
        transition=[
            aws.s3.BucketLifecycleRuleTransitionArgs(
                days=30,
                storage_class="STANDARD_IA",
            ),
        ],
    ),
    # Move to Glacier after 90 days
    aws.s3.BucketLifecycleRuleArgs(
        enabled=True,
        prefix="archives/",
        transition=[
            aws.s3.BucketLifecycleRuleTransitionArgs(
                days=90,
                storage_class="GLACIER",
            ),
        ],
    ),
]
```

### Pattern 3: CDN Integration

```python
"""Serve files through CDN."""

import pulumi_cloudflare as cloudflare

# R2 bucket for storage
bucket = cloudflare.R2Bucket("assets", ...)

# Custom domain for CDN access
cloudflare.Record(
    "cdn-cname",
    zone_id=zone.id,
    name="cdn",
    type="CNAME",
    value=f"{bucket.name}.r2.cloudflarestorage.com",
    proxied=True,  # Enable Cloudflare CDN
)

# Cache rules
cloudflare.PageRule(
    "cache-assets",
    zone_id=zone.id,
    target="cdn.example.com/*",
    actions=cloudflare.PageRuleActionsArgs(
        cache_level="cache_everything",
        edge_cache_ttl=86400,  # 1 day
    ),
)
```

## Cost Comparison

### Object Storage (per 1TB/month + 1TB egress)

| Provider | Storage | Egress | Total |
|----------|---------|--------|-------|
| Cloudflare R2 | $15 | $0 | **$15** |
| AWS S3 | $23 | $90 | $113 |
| GCP Cloud Storage | $20 | $120 | $140 |
| Azure Blob | $18 | $87 | $105 |

**R2 is ~7x cheaper than S3 for egress-heavy workloads.**

### Database Storage

| Provider | Small (10GB) | Medium (100GB) | Large (500GB) |
|----------|--------------|----------------|---------------|
| Supabase Pro | $25 base | ~$37 | ~$87 |
| PlanetScale | $29/month | ~$50 | ~$150 |
| RDS PostgreSQL | ~$30 | ~$100 | ~$300 |

## Best Practices

### Security

- [ ] Enable encryption at rest
- [ ] Block public access by default
- [ ] Use signed URLs for uploads/downloads
- [ ] Implement RLS for user data
- [ ] Regular backup verification

### Cost Optimization

- [ ] Use R2 for egress-heavy workloads
- [ ] Implement lifecycle policies
- [ ] Monitor storage growth
- [ ] Clean up orphaned files
- [ ] Right-size database instances

### Performance

- [ ] Use CDN for public assets
- [ ] Implement caching layers
- [ ] Compress before upload
- [ ] Use appropriate storage class
- [ ] Index database queries

## Related Documentation

- **Cloudflare**: [cloudflare.md](./cloudflare.md) - R2 and CDN
- **Supabase**: [supabase.md](./supabase.md) - Storage and database
- **Secrets Management**: [secrets-management.md](./secrets-management.md) - Credentials
- **Financial Governance**: [financial-governance.md](./financial-governance.md) - Cost management
- **Database Migrations**: [database-migrations.md](./database-migrations.md) - Schema management

---

**Key Takeaway**: Choose storage based on access patterns and cost. Use Cloudflare R2 for object storage to eliminate egress fees. Use Supabase for integrated database + storage + auth. Implement lifecycle policies to manage costs over time.
