# Self-Healing CI/CD Pipeline

**Compliance**: T-13 (Verify Before Done), P-005 (Self-Healing CI/CD)

## Overview

LLMHosts.com uses a **zero-tolerance, self-healing CI pipeline** that automatically fixes lint and format errors, ensures all tests pass, and provides comprehensive verification before allowing merges.

## Key Features

### 1. Zero-Tolerance Policy

- **Zero errors**: All type checks, lint checks, and tests must pass
- **Zero warnings**: Build warnings are treated as failures
- **Zero console errors**: E2E tests monitor browser console for errors

### 2. Self-Healing

When lint or format checks fail, the CI will:
1. Attempt auto-fix (`ruff check --fix`, `ruff format`, or `eslint --fix`)
2. Re-check to verify fixes worked
3. **If fixes succeed**: Commit changes with `[skip ci]` message
4. **If fixes fail**: Mark job as failed, require human intervention

### 3. Two-Phase Architecture

**Phase 1: Parallel Checks** (11 jobs run in parallel)

| Category | Job | Purpose |
|----------|-----|---------|
| Python | `python-typecheck` | MyPy type checking |
| Python | `python-lint` | Ruff check + format (self-healing) |
| Python | `python-test` | pytest with coverage (437 tests) |
| Python | `python-security` | Bandit security scan |
| Frontend | `frontend-typecheck` | TypeScript strict checking |
| Frontend | `frontend-lint` | ESLint (self-healing) |
| Frontend | `frontend-test` | Vitest with coverage |
| Frontend | `frontend-build` | Next.js build (zero-warning) |
| Frontend | `frontend-security` | npm audit (high/critical only) |
| Infra | `docker-build` | Verify dev + prod images |
| Infra | `secret-scan` | GitLeaks secret detection |

**Phase 2: Verification Gate** (1 job, depends on all Phase 1)

- `verify-all`: Checks ALL Phase 1 jobs passed
- Generates comprehensive verification report
- Hard gate - if any job failed, merge is blocked

## Workflow Files

### `.github/workflows/ci.yml`

Main CI pipeline. Runs on:
- Push to `main` or `develop`
- Pull requests to `main` or `develop`

### `.github/workflows/post-deploy-verify.yml`

Post-deployment health checks. Manually triggered with:
```bash
gh workflow run post-deploy-verify.yml -f environment=production
```

Verifies:
- `/health` and `/ready` endpoints
- Frontend loads successfully
- API docs accessible
- SSL/TLS certificate validity
- Rate limiting works

## Git Hooks (Husky)

Pre-commit hooks run the same checks locally to catch issues before pushing.

### Setup

Hooks are automatically configured when you run:
```bash
cd frontend
npm install
```

### What Hooks Do

**`pre-commit`**:
- Runs `ruff format` and `ruff check --fix` on staged Python files
- Runs `eslint --fix` on staged frontend files
- Auto-stages fixed files

**`commit-msg`**:
- Validates commit message format using commitlint
- Enforces conventional commits (feat, fix, docs, etc.)

### Bypassing Hooks (Use Sparingly)

```bash
git commit --no-verify -m "emergency fix"
```

**Only use for**:
- Emergency hotfixes
- Documentation-only changes
- Non-code changes where hooks don't apply

## Developer Workflow

### Normal Flow

1. Create feature branch
2. Make changes
3. Git hooks auto-fix on commit
4. Push to GitHub
5. CI runs, may auto-fix remaining issues
6. Pull latest changes if CI committed fixes
7. Merge when verify-all passes

### When CI Commits Fixes

If you see a commit from `github-actions[bot]`:

```bash
git pull origin your-branch-name
```

Then continue your work. The fixes are automatically committed with `[skip ci]` to avoid infinite loops.

### When Auto-Fix Fails

If the CI can't auto-fix an issue:
1. Check the job logs to see what failed
2. Fix the issue locally
3. Push the fix
4. CI will re-run

## Common Issues

### "Type check failed with errors"

MyPy found type errors. Common fixes:
- Add type hints to function signatures
- Use `# type: ignore[error-code]` for unavoidable issues
- Import types in `TYPE_CHECKING` block

### "Build produced N warning(s)"

Next.js build warnings. Common sources:
- Unused variables in components
- Missing dependencies in useEffect
- Console.log statements left in code

### "npm audit found critical/high vulnerabilities"

Security vulnerabilities in dependencies. Fix:
```bash
cd frontend
npm audit fix
```

If automated fix doesn't work, check if there's a breaking change and update code accordingly.

## Metrics

**Current Test Coverage**:
- Python: 437 tests (all passing)
- Frontend: Unit + integration tests (Vitest)
- E2E: Playwright browser tests

**CI Performance**:
- Phase 1 jobs: ~3-5 minutes (parallel)
- Phase 2 gate: ~10 seconds
- Total: ~5-6 minutes per run

## Skipping CI

Use `[skip ci]` in commit messages to skip CI:

```bash
git commit -m "docs: update README [skip ci]"
```

**When to use**:
- Documentation-only changes (*.md files)
- README updates
- Comment-only changes
- CI configuration changes (after testing locally)

**Never use for**:
- Code changes
- Dependency updates
- Configuration changes affecting runtime
- Anything that could break production

## Self-Healing Commit Format

When CI auto-fixes issues, it commits with:

```
fix(python): auto-fix lint/format errors [skip ci]

Auto-fixed by self-healing CI pipeline.
See T-13 (Verify Before Done) and P-005 (Self-Healing CI/CD).
```

or

```
fix(frontend): auto-fix lint errors [skip ci]

Auto-fixed by self-healing CI pipeline.
See T-13 (Verify Before Done) and P-005 (Self-Healing CI/CD).
```

## Verification Report

The `verify-all` job generates a comprehensive report in the GitHub Actions summary:

```
## 🔍 Verification Report

**T-13 (Verify Before Done) & P-005 (Self-Healing CI/CD) Compliance**

| Check | Status |
|-------|--------|
| 🐍 python-typecheck | ✅ success |
| 🐍 python-lint | ✅ success |
| 🐍 python-test | ✅ success |
| 🐍 python-security | ✅ success |
| ⚛️ frontend-typecheck | ✅ success |
| ⚛️ frontend-lint | ✅ success |
| ⚛️ frontend-test | ✅ success |
| ⚛️ frontend-build | ✅ success |
| ⚛️ frontend-security | ✅ success |
| 🐳 docker-build | ✅ success |
| 🔒 secret-scan | ✅ success |

**Result: ✅ All checks passed. Merge-ready.**

**Self-Healing Actions Taken:** Check individual job logs for auto-fixes.
```

## References

- **T-13**: Verify Before Done - Never mark work complete without verification
- **P-005**: Self-Healing CI/CD - CI should fix common issues automatically
- **Conventional Commits**: https://www.conventionalcommits.org/
- **Aut-O Templates**: Original source of self-healing patterns

## Maintenance

### Updating CI

When modifying CI workflows:
1. Test changes on a feature branch first
2. Use `[skip ci]` judiciously to avoid wasted CI runs
3. Document any new checks or jobs in this file
4. Update CONTRIBUTING.md if developer workflow changes

### Adding New Checks

To add a new check:
1. Add job to Phase 1 in `.github/workflows/ci.yml`
2. Add job to `needs` list in `verify-all`
3. Add job to verification report in `verify-all` step
4. Document in this file
5. Update CONTRIBUTING.md

## Contact

Questions about CI? Open an issue with `type:chore` label.
