# `cy ci` — CI Orchestrator

The CI orchestrator runs the full PR enrichment flow as a single command. It lints your PySpark files, optionally enriches findings with cost data from the Cluster Yield server, and outputs structured JSON suitable for posting as PR comments and inline annotations.

`cy ci` owns everything between "engineer opens a PR" and "here is the structured output to post." It never calls the GitHub API, GitLab API, or any platform-specific API — the caller posts the results.

## Quick Start

```bash
# Lint only (free tier, no server needed)
cy ci --paths src/

# With enrichment (adds dollar amounts to findings)
cy ci --paths src/ --api-key $CLUSTER_YIELD_KEY --environment prod-analytics

# With change classification (detects what changed vs base branch)
cy ci --paths src/ --base-ref origin/main

# Full flow
cy ci \
  --paths src/ \
  --base-ref origin/main \
  --api-key $CLUSTER_YIELD_KEY \
  --environment prod-analytics \
  --format json \
  --output /tmp/cy-results.json
```

## Flags

| Flag | Required | Default | Description |
|------|----------|---------|-------------|
| `--paths` | Yes | — | Directories or files to lint |
| `--base-ref` | No | — | Git ref for base branch. Enables change classification and changed-line detection |
| `--api-key` | No | — | Cluster Yield API key. Enables server-side enrichment with cost data |
| `--environment` | No | — | Snapshot environment name. Required when `--api-key` is set |
| `--base-url` | No | `https://api.clusteryield.app` | API base URL (for self-hosted deployments) |
| `--format` | No | `json` | Output format: `json` or `github-comment` (markdown) |
| `--output` | No | stdout | Write output to a file instead of stdout |
| `--timeout` | No | `30` | Server request timeout in seconds |

### Environment Variables

These can be used instead of flags:

- `CLUSTER_YIELD_API_KEY` — fallback for `--api-key`
- `CLUSTER_YIELD_ENVIRONMENT` — fallback for `--environment`
- `CLUSTER_YIELD_BASE_URL` — fallback for `--base-url`

## Output Schema

```json
{
  "findings": [],
  "changeClassifications": [],
  "enrichedFindings": [],
  "changeFindings": [],
  "planFindings": [],
  "comment": {
    "markdown": "## Cluster Yield — 3 findings ($6,100/month impact)\n...",
    "annotations": [
      {
        "file": "pipelines/order_report.py",
        "line": 52,
        "level": "warning",
        "message": "CY001: .collect() called without filtering."
      }
    ],
    "stats": { "total": 3, "enriched": true }
  },
  "stats": {
    "filesScanned": 4,
    "filesWithFindings": 1,
    "linterFindings": 3,
    "changeClassifications": 0,
    "enriched": true,
    "matchStats": {},
    "errors": {}
  }
}
```

Without `--api-key`, `enrichedFindings`, `changeFindings`, and `planFindings` are empty arrays, `enriched` is `false`, and the comment shows pattern-only findings without dollar amounts.

## Graceful Degradation

`cy ci` produces useful output at every capability level and never fails the pipeline on enrichment issues.

| Capability | Available When | What It Adds |
|------------|---------------|--------------|
| Linter findings | Always | Pattern-only findings (no $) |
| Change classifications | `--base-ref` provided | Change type labels (filter_removed, etc.) |
| Notebook awareness | Databricks notebook detected | Correct line mapping for `.py` notebook exports |
| Server enrichment | `--api-key` + `--environment` + server reachable | Dollar amounts on findings |

| Failure | Behavior |
|---------|----------|
| Server unreachable | Warning to stderr. Unenriched findings still emitted |
| Invalid API key | Warning to stderr. Same as unreachable |
| Server timeout | Warning to stderr. Same as unreachable |
| Parse error in a file | Warning per file. Other files still processed |

## GitHub Actions

```yaml
name: Cluster Yield
on: pull_request

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # needed for git diff against base

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - run: pip install cylint

      - run: |
          cy ci \
            --paths src/ \
            --base-ref ${{ github.event.pull_request.base.sha }} \
            --format json \
            --output /tmp/cy-results.json
        env:
          CLUSTER_YIELD_API_KEY: ${{ secrets.CLUSTER_YIELD_KEY }}
          CLUSTER_YIELD_ENVIRONMENT: prod-analytics
```

> **Note:** Use `fetch-depth: 0` so that `git diff` can access the base branch commit.

## GitLab CI

```yaml
pyspark-lint:
  script:
    - pip install cylint
    - cy ci --paths src/ --base-ref $CI_MERGE_REQUEST_DIFF_BASE_SHA --format json
```

## Module Structure

| Module | Purpose |
|--------|---------|
| `orchestrator.py` | Main `CIOrchestrator` class — wires lint, cell maps, enrichment, and formatting |
| `cell_map.py` | Databricks notebook cell splitting, fingerprinting, and coordinate conversion |
| `enrich.py` | `/enrich` endpoint request assembly, HTTP client, and response processing |
| `comment.py` | PR comment formatting with confidence tiers and annotation generation |

## Programmatic Usage

```python
from cylint.ci import CIOrchestrator

orch = CIOrchestrator(
    paths=["src/pipelines/"],
    api_key="cy_key_...",
    environment="prod-analytics",
)
result = orch.run()

# Structured JSON
print(result.to_json())

# Just the PR comment markdown
print(result.comment.markdown)

# Annotations for inline PR comments
for ann in result.comment.annotations:
    print(f"{ann['file']}:{ann['line']} [{ann['level']}] {ann['message']}")
```

## Exit Codes

| Code | Meaning |
|------|---------|
| 0 | No linter findings |
| 1 | One or more linter findings |
