# Evictor Configuration Guide

## ⚠️ CRITICAL SAFETY WARNINGS

**READ THIS ENTIRE GUIDE BEFORE USING THE EVICTOR TOOL**

The evictor is a CAST.AI component that optimizes cluster resources by evicting and rescheduling pods to better-fitting nodes. **Misconfiguration can cause production service disruptions.**

**BEFORE YOU START:**
- ✅ Understand what each parameter does (read this guide)
- ✅ Configure PodDisruptionBudgets for critical applications
- ✅ Test in a non-production cluster first
- ✅ Always start with `dry_run=true`
- ✅ Monitor your cluster closely after changes
- ✅ Have a rollback plan ready

---

## Understanding Evictor Parameters

### enabled (boolean)
**What it does**: Master switch for the evictor
- `true`: Evictor is active (respects `dry_run` setting)
- `false`: Evictor is completely disabled
- **Default**: `false`
- **Risk**: LOW when `dry_run=true`, HIGH when `dry_run=false`

**When to use**:
- Set `true` to enable optimization
- Set `false` for emergency stop of all evictions

### dry_run (boolean)
**What it does**: Controls whether evictions are simulated or executed
- `true`: Simulates evictions, logs what would happen, but doesn't execute
- `false`: Performs actual pod evictions
- **Default**: `true`
- **Risk**: LOW when true, HIGH when false
- **⚠️ CRITICAL**: Always start with `dry_run=true`

**When to use**:
- `true` for testing and validation (ALWAYS START HERE)
- `false` only after confirming dry-run behaves as expected

### aggressive_mode (boolean)
**What it does**: Controls whether single-replica workloads can be evicted
- `true`: Evicts single-replica Deployments, StatefulSets, and Jobs
- `false`: Only evicts multi-replica workloads (safer)
- **Default**: `false`
- **Risk**: CRITICAL when combined with `dry_run=false`
- **⚠️ DANGER**: Can interrupt single-instance applications

**When to use**:
- Almost NEVER in production
- Only with extensive PodDisruptionBudgets
- Only after thorough testing in staging

### scoped_mode (boolean)
**What it does**: Limits evictor to CAST.AI-managed nodes only
- `true`: Only evicts pods from CAST.AI-provisioned nodes
- `false`: Evicts pods from any node in the cluster
- **Default**: `false`
- **Risk**: MEDIUM

**When to use**:
- `true` during gradual CAST.AI adoption
- `false` for full cluster optimization

### cycle_interval (string)
**What it does**: Time between evictor optimization cycles
- Format: Go duration (e.g., "30s", "5m", "1h")
- **Default**: "5m"
- **Risk**: LOW (but very short intervals cause churn)

**Recommended values**:
- Production: "5m" to "10m"
- Aggressive: "1m" to "3m"
- Conservative: "15m" to "30m"

### node_grace_period_minutes (integer)
**What it does**: How long to wait before new nodes are eligible for eviction
- Range: 1-60 minutes
- **Default**: 5
- **Risk**: LOW

**When to adjust**:
- Lower (1-3): More aggressive optimization
- Higher (10-15): More stable, less churn

---

## Protection Mechanisms

### What the Evictor ALWAYS Respects

1. **PodDisruptionBudgets (PDBs)** - ALWAYS honored
2. **DaemonSet pods** - Automatically skipped
3. **System namespaces** - kube-system pods protected by default
4. **Pod annotations**:
   - `autoscaling.cast.ai/removal-disabled: "true"`
   - `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"`

### How to Protect Critical Workloads

**Method 1: PodDisruptionBudget (Recommended)**
```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-critical-app-pdb
  namespace: production
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: my-critical-app
```

**Method 2: Pod Annotation**
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-critical-app
spec:
  template:
    metadata:
      annotations:
        autoscaling.cast.ai/removal-disabled: "true"
```

---

## Recommended Workflows

### Workflow 1: First-Time Setup (Non-Production)

```
Step 1: Enable with dry-run
  Ask Claude: "Enable the evictor on my staging cluster with dry_run=true"

Step 2: Monitor logs (15-30 minutes)
  - Check CAST.AI console → Cluster → Events
  - Look for simulated eviction logs
  - Verify expected pods would be evicted

Step 3: Review and protect critical workloads
  - Add PodDisruptionBudgets where needed
  - Add protection annotations if necessary

Step 4: Disable dry-run (staging only!)
  Ask Claude: "Disable dry_run for the evictor on my staging cluster"

Step 5: Monitor for disruptions (1+ hour)
  - Watch application health
  - Check for unexpected restarts
  - Verify service availability

Step 6: Rollback if needed
  Ask Claude: "Disable the evictor on my staging cluster"
```

### Workflow 2: Production Enablement

**Prerequisites**:
- ✅ Successfully tested in staging
- ✅ PodDisruptionBudgets configured
- ✅ Protection annotations added
- ✅ Monitoring in place
- ✅ Rollback plan documented

```
Step 1: Enable dry-run in production
  Ask Claude: "Enable the evictor on my production cluster with dry_run=true and scoped_mode=true"

Step 2: Extended monitoring (2-4 hours)
  - Verify only CAST.AI nodes would be affected (scoped_mode)
  - Confirm no critical workloads would be disrupted
  - Check logs for unexpected behavior

Step 3: Disable dry-run (carefully!)
  Ask Claude: "Disable dry_run for the evictor on my production cluster, keeping scoped_mode enabled"

Step 4: Active monitoring (minimum 4 hours)
  - Set up alerts for pod eviction rate
  - Watch application metrics
  - Have team available for rollback

Step 5: Expand scope (optional)
  If all is stable, disable scoped_mode:
  Ask Claude: "Disable scoped_mode for the evictor on my production cluster"
```

### Workflow 3: Emergency Rollback

**Symptoms of Issues**:
- Unexpected pod evictions
- Service degradation or outages
- Excessive pod restarts
- Application errors

**Immediate Action**:
```
Ask Claude: "Disable the evictor on my [cluster] immediately with confirmation"
```

This sets `enabled=false`, stopping ALL eviction activity.

**Verification**:
```
Ask Claude: "Show me the evictor configuration for my [cluster]"
```

Confirm `enabled: false` in the response.

---

## Usage Examples

### Example 1: Enable with Maximum Safety

**You**: "Enable the evictor on my production-gke cluster with dry_run mode and scoped mode"

**Claude will**:
1. Show current configuration
2. Show proposed changes with risk assessment
3. Ask for confirmation
4. Apply changes: `enabled=true`, `dry_run=true`, `scoped_mode=true`
5. Show before/after comparison
6. Provide monitoring recommendations

### Example 2: Adjust Cycle Interval

**You**: "Change the evictor cycle interval to 10 minutes for my cluster"

**Claude will**:
1. Read current configuration
2. Update only `cycle_interval` to "10m"
3. Preserve all other settings
4. Show what changed
5. Confirm update succeeded

### Example 3: Disable Dry-Run (After Testing)

**You**: "Disable dry_run for the evictor on my staging cluster"

**Claude will**:
1. Show MEDIUM/HIGH risk warning
2. Require explicit confirmation
3. Apply change: `dry_run=false`
4. Recommend monitoring for 15-30 minutes
5. Provide rollback instructions

### Example 4: Emergency Stop

**You**: "Stop all evictor activity on my cluster RIGHT NOW"

**Claude will**:
1. Set `enabled=false` with confirmation
2. Confirm all evictions stopped
3. Show current (disabled) state
4. Explain how to re-enable when ready

---

## Monitoring After Changes

### Metrics to Watch

**In CAST.AI Console**:
- Navigate to: Cluster → Events → Filter by "Evictor"
- Look for: Pod eviction events
- Check for: Errors or warnings

**In Kubernetes**:
```bash
# Watch eviction events
kubectl get events --field-selector reason=Evicted --all-namespaces -w

# Check pod restart counts
kubectl get pods --all-namespaces --sort-by='.status.containerStatuses[0].restartCount'

# Monitor node count
kubectl get nodes -w
```

**Application Metrics**:
- Service availability/uptime
- Response times
- Error rates
- Request success rate

### What's Normal

- **Gradual optimization**: Pods evicted over time, not all at once
- **Improved node utilization**: More pods per node
- **Slight pod restarts**: As pods move to better nodes
- **Node count reduction**: Unused nodes removed

### What's NOT Normal (Rollback Immediately)

- **Sudden pod eviction waves**: Many pods evicted simultaneously
- **Service outages**: Applications become unavailable
- **Continuous restarts**: Pods keep getting evicted repeatedly
- **Cascading failures**: One eviction triggers more problems

---

## Troubleshooting

### Issue: Too Many Evictions

**Symptoms**: High pod eviction rate, applications restarting frequently

**Solution**:
1. Increase `cycle_interval`: "5m" → "10m" or "15m"
2. Or temporarily enable `dry_run=true` to stop evictions
3. Review which pods are being evicted and add protection

### Issue: Critical Application Interrupted

**Symptoms**: Important service experienced downtime due to eviction

**Immediate Action**:
```
Ask Claude: "Disable the evictor on my cluster immediately"
```

**Prevention**:
1. Add PodDisruptionBudget for the application
2. Or add protection annotation to pods
3. Or enable `scoped_mode` to limit scope

**Re-enable**:
Only after protection is in place and tested in staging

### Issue: Not Enough Optimization

**Symptoms**: Cluster still has unused nodes, costs aren't optimizing

**Solutions**:
1. Verify `dry_run=false` (evictions must be enabled)
2. Check if `enabled=true`
3. Reduce `cycle_interval` for more frequent optimization
4. Review pod/node protection annotations (might be over-protecting)
5. Consider disabling `scoped_mode` to allow full cluster optimization

### Issue: Configuration Not Applying

**Symptoms**: Update succeeds but configuration doesn't change

**Solutions**:
1. Verify API key has write permissions (not read-only)
2. Check policy ID is correct: use `list_cluster_policies`
3. Confirm cluster is accessible
4. Check CAST.AI console for policy lock or conflicts

---

## Best Practices

### 1. Testing First
✅ Always test in non-production
✅ Use dry-run mode extensively
✅ Monitor for at least 1 hour before production

### 2. Protection
✅ Configure PodDisruptionBudgets for all critical apps
✅ Add annotations for single-replica critical workloads
✅ Use scoped_mode during migration/testing

### 3. Monitoring
✅ Watch CAST.AI console events
✅ Monitor Kubernetes eviction events
✅ Set up alerts for high eviction rates
✅ Track application health metrics

### 4. Changes
✅ Make one change at a time
✅ Wait and monitor between changes
✅ Document all configuration changes
✅ Save before/after configurations

### 5. Rollback Readiness
✅ Know how to disable immediately
✅ Have the previous configuration saved
✅ Test rollback procedure in staging
✅ Communicate changes to team

---

## Quick Reference

### Enable Dry-Run Testing
```
enabled: true
dry_run: true
aggressive_mode: false
scoped_mode: true  (optional, for safety)
```

### Production Configuration (Conservative)
```
enabled: true
dry_run: false
aggressive_mode: false
scoped_mode: false
cycle_interval: "10m"
node_grace_period_minutes: 10
```

### Production Configuration (Aggressive)
```
⚠️ Use ONLY after extensive testing!

enabled: true
dry_run: false
aggressive_mode: true
scoped_mode: false
cycle_interval: "5m"
node_grace_period_minutes: 5
```

### Emergency Disable
```
enabled: false
```

### Switch to Simulation
```
dry_run: true
```

---

## Getting Help

**If you encounter issues:**

1. **Check Logs**: CAST.AI Console → Cluster → Events
2. **Review Configuration**: Use `get_cluster_policy_details`
3. **Contact Support**: support@cast.ai with cluster ID and timestamp
4. **Emergency**: Disable evictor immediately, investigate later

**Support Information**:
- Email: support@cast.ai
- Documentation: https://docs.cast.ai/docs/evictor
- Status: https://status.cast.ai

---

## Additional Resources

- [CAST.AI Evictor Documentation](https://docs.cast.ai/docs/evictor)
- [Kubernetes PodDisruptionBudgets](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)
- [CAST.AI Support](https://cast.ai/support)

---

**Remember**: When in doubt, use `dry_run=true` and monitor carefully! 🛡️