# CI S3 Asset Upload Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** CI builds and uploads dashboard frontend tarball + lambda zip to a shared S3 artifacts bucket, so deploy commands can pull pre-built assets instead of building locally.

**Architecture:** Add a `cogent-polis-ci-artifacts` S3 bucket to the polis core CDK stack. Grant the existing GitHub Actions OIDC role write access. Add build+upload steps to the existing dashboard and executor CI workflows. Update CLI deploy commands to accept `--sha` to pull pre-built assets.

**Tech Stack:** AWS CDK (S3, IAM), GitHub Actions, AWS CLI, Node.js, Python/uv

---

### Task 1: Add CI Artifacts S3 Bucket to Polis Core Stack

**Files:**
- Modify: `src/polis/cdk/stacks/core.py:433-476`

**Step 1: Add the S3 bucket and grant access**

In `src/polis/cdk/stacks/core.py`, add the S3 bucket after the GitHub OIDC role (line 457), grant read/write to the GitHub Actions role, and export the bucket name. Insert before the `# --- Outputs ---` comment:

```python
        # --- CI Artifacts Bucket ---
        self.ci_artifacts_bucket = s3.Bucket(
            self,
            "CIArtifactsBucket",
            bucket_name="cogent-polis-ci-artifacts",
            removal_policy=RemovalPolicy.RETAIN,
            auto_delete_objects=False,
            lifecycle_rules=[
                s3.LifecycleRule(
                    id="expire-old-artifacts",
                    expiration=Duration.days(90),
                ),
            ],
        )
        self.ci_artifacts_bucket.grant_read_write(self.github_actions_role)
```

Add to the Outputs section:

```python
        cdk.CfnOutput(self, "CIArtifactsBucket", value=self.ci_artifacts_bucket.bucket_name)
```

Also add the `aws_s3` import at the top of the file:

```python
from aws_cdk import aws_s3 as s3
```

**Step 2: Verify CDK synth**

Run: `cd /Users/daveey/code/cogents/cogents.1 && python -m pytest tests/ -k "polis" --no-header -q 2>/dev/null; echo "---"; cd src/polis && python -c "from polis.cdk.stacks.core import PolisStack; print('import ok')"`

Expected: Import succeeds without errors.

**Step 3: Commit**

```bash
git add src/polis/cdk/stacks/core.py
git commit -m "feat(ci): add CI artifacts S3 bucket to polis core stack"
```

---

### Task 2: Create S3 Upload Composite Action

**Files:**
- Create: `.github/actions/s3-upload/action.yml`

**Step 1: Create the composite action**

```yaml
name: "Upload CI Artifact to S3"
description: "Authenticate via OIDC and upload a file to the CI artifacts bucket"

inputs:
  aws_role:
    description: "IAM role ARN to assume via OIDC"
    required: true
  aws_region:
    description: "AWS region"
    required: true
    default: "us-east-1"
  source_path:
    description: "Local file path to upload"
    required: true
  s3_key:
    description: "S3 key (path within the bucket)"
    required: true
  bucket:
    description: "S3 bucket name"
    required: true
    default: "cogent-polis-ci-artifacts"

runs:
  using: "composite"
  steps:
    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: ${{ inputs.aws_role }}
        aws-region: ${{ inputs.aws_region }}

    - name: Upload to S3
      shell: bash
      run: |
        aws s3 cp "${{ inputs.source_path }}" "s3://${{ inputs.bucket }}/${{ inputs.s3_key }}"
        echo "Uploaded: s3://${{ inputs.bucket }}/${{ inputs.s3_key }}"
```

**Step 2: Commit**

```bash
git add .github/actions/s3-upload/action.yml
git commit -m "feat(ci): add s3-upload composite action"
```

---

### Task 3: Add Frontend Tarball Build+Upload to Dashboard Workflow

**Files:**
- Modify: `.github/workflows/docker-build-dashboard.yml`

**Step 1: Add asset build+upload steps after Docker build**

Add steps after the existing "Build and push dashboard image" step. The workflow already has OIDC credentials configured by the ecr-build action, but s3-upload will re-configure them (composite actions are self-contained).

```yaml
      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Build frontend tarball
        run: |
          cd dashboard/frontend
          npm ci
          npx next build
          mkdir -p /tmp/frontend-pkg
          cp -r .next/standalone/* /tmp/frontend-pkg/
          [ -d .next/static ] && mkdir -p /tmp/frontend-pkg/.next && cp -r .next/static /tmp/frontend-pkg/.next/static
          [ -d public ] && cp -r public /tmp/frontend-pkg/public
          tar czf /tmp/frontend.tar.gz -C /tmp/frontend-pkg .
          echo "SIZE=$(du -h /tmp/frontend.tar.gz | cut -f1)" >> "$GITHUB_ENV"

      - name: Upload frontend tarball to S3
        uses: ./.github/actions/s3-upload
        with:
          aws_role: ${{ vars.AWS_ROLE }}
          source_path: /tmp/frontend.tar.gz
          s3_key: dashboard/${{ github.sha }}/frontend.tar.gz
```

Update the Summary step to include the asset info:

```yaml
      - name: Summary
        run: |
          echo "### Dashboard Image Built" >> "$GITHUB_STEP_SUMMARY"
          echo "" >> "$GITHUB_STEP_SUMMARY"
          echo "**Docker Tags:**" >> "$GITHUB_STEP_SUMMARY"
          echo '```' >> "$GITHUB_STEP_SUMMARY"
          echo "${{ steps.build.outputs.image_uri }}" >> "$GITHUB_STEP_SUMMARY"
          echo '```' >> "$GITHUB_STEP_SUMMARY"
          echo "" >> "$GITHUB_STEP_SUMMARY"
          echo "**Frontend Tarball:** \`s3://cogent-polis-ci-artifacts/dashboard/${{ github.sha }}/frontend.tar.gz\` (${{ env.SIZE }})" >> "$GITHUB_STEP_SUMMARY"
          echo "" >> "$GITHUB_STEP_SUMMARY"
          echo "**Deploy:** \`cogent <name> dashboard deploy --sha ${{ steps.build.outputs.short_sha }}\`" >> "$GITHUB_STEP_SUMMARY"
```

**Step 2: Commit**

```bash
git add .github/workflows/docker-build-dashboard.yml
git commit -m "feat(ci): build and upload dashboard frontend tarball to S3"
```

---

### Task 4: Add Lambda Zip Build+Upload to Executor Workflow

**Files:**
- Modify: `.github/workflows/docker-build-executor.yml`

**Step 1: Add lambda packaging + upload steps after Docker build**

Add steps after the existing "Build and push executor image" step:

```yaml
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install uv
        uses: astral-sh/setup-uv@v4

      - name: Package Lambda zip
        run: |
          # Install runtime deps into temp dir
          DEPS_DIR=$(mktemp -d)
          uv pip install --target "$DEPS_DIR" --quiet --python 3.12 --python-platform linux \
            pydantic pydantic-settings pydantic-core annotated-types \
            Pillow google-genai anthropic asana

          # Create zip with src/ contents + deps
          cd src
          zip -r /tmp/lambda.zip . -x '*/__pycache__/*' '*.pyc' '*.egg-info/*'
          cd "$DEPS_DIR"
          zip -r /tmp/lambda.zip . -x '*/__pycache__/*' '*.pyc'

          echo "SIZE=$(du -h /tmp/lambda.zip | cut -f1)" >> "$GITHUB_ENV"

      - name: Upload Lambda zip to S3
        uses: ./.github/actions/s3-upload
        with:
          aws_role: ${{ vars.AWS_ROLE }}
          source_path: /tmp/lambda.zip
          s3_key: lambda/${{ github.sha }}/lambda.zip
```

Update the Summary step:

```yaml
      - name: Summary
        run: |
          echo "### Executor Image Built" >> "$GITHUB_STEP_SUMMARY"
          echo "" >> "$GITHUB_STEP_SUMMARY"
          echo "**Docker Tags:**" >> "$GITHUB_STEP_SUMMARY"
          echo '```' >> "$GITHUB_STEP_SUMMARY"
          echo "${{ steps.build.outputs.image_uri }}" >> "$GITHUB_STEP_SUMMARY"
          echo '```' >> "$GITHUB_STEP_SUMMARY"
          echo "" >> "$GITHUB_STEP_SUMMARY"
          echo "**Lambda Zip:** \`s3://cogent-polis-ci-artifacts/lambda/${{ github.sha }}/lambda.zip\` (${{ env.SIZE }})" >> "$GITHUB_STEP_SUMMARY"
          echo "" >> "$GITHUB_STEP_SUMMARY"
          echo "**Deploy:** \`cogent <name> cogtainer update ecs --tag executor-${{ steps.build.outputs.short_sha }}\`" >> "$GITHUB_STEP_SUMMARY"
```

**Step 2: Commit**

```bash
git add .github/workflows/docker-build-executor.yml
git commit -m "feat(ci): build and upload lambda zip to S3"
```

---

### Task 5: Update CLI to Pull Pre-Built Assets from CI Bucket

**Files:**
- Modify: `src/cogtainer/update_cli.py`

**Step 1: Add CI artifact download helpers**

Add after the `_upload_dashboard_tarball` function (~line 486):

```python
CI_ARTIFACTS_BUCKET = "cogent-polis-ci-artifacts"


def _resolve_commit_sha(sha: str) -> str:
    """Resolve a short or long SHA to the full SHA used in CI artifacts."""
    import subprocess

    try:
        return subprocess.check_output(
            ["git", "rev-parse", sha], text=True
        ).strip()
    except Exception:
        return sha


def _download_ci_artifact(session: boto3.Session, s3_key: str, dest_path: str) -> None:
    """Download a CI artifact from the shared bucket."""
    s3_client = session.client("s3", region_name=DEFAULT_REGION)
    s3_client.download_file(CI_ARTIFACTS_BUCKET, s3_key, dest_path)


def _check_ci_artifact_exists(session: boto3.Session, s3_key: str) -> bool:
    """Check if a CI artifact exists in the shared bucket."""
    s3_client = session.client("s3", region_name=DEFAULT_REGION)
    try:
        s3_client.head_object(Bucket=CI_ARTIFACTS_BUCKET, Key=s3_key)
        return True
    except Exception:
        return False
```

**Step 2: Add `--sha` to `update lambda` command**

Modify the `update_lambda` function to accept `--sha` and download from CI instead of packaging locally:

```python
@update.command("lambda")
@click.option("--profile", default=None, help=_PROFILE_HELP)
@click.option("--sha", default=None, help="Use pre-built lambda zip from CI (git SHA)")
@click.pass_context
def update_lambda(ctx: click.Context, profile: str | None, sha: str | None):
```

Add before `zip_bytes = _package_lambda_code()`:

```python
    if sha:
        full_sha = _resolve_commit_sha(sha)
        s3_key = f"lambda/{full_sha}/lambda.zip"
        click.echo(f"  Downloading pre-built lambda from CI ({sha[:8]})...")
        import tempfile
        with tempfile.NamedTemporaryFile(suffix=".zip", delete=False) as tmp:
            tmp_path = tmp.name
        try:
            _download_ci_artifact(session, s3_key, tmp_path)
            with open(tmp_path, "rb") as f:
                zip_bytes = f.read()
        except Exception as e:
            raise click.ClickException(
                f"CI artifact not found: s3://{CI_ARTIFACTS_BUCKET}/{s3_key}\n"
                f"Check CI: gh run list --repo Metta-AI/cogents-v1 --workflow docker-build-executor.yml"
            )
        finally:
            os.unlink(tmp_path)
    else:
        zip_bytes = _package_lambda_code()
```

**Step 3: Add `--sha` to `update dashboard` command**

Modify the `update_dashboard` function to accept `--sha`:

```python
@update.command("dashboard")
@click.option("--docker", is_flag=True, help="Force rebuild and push Docker image")
@click.option("--skip-health", is_flag=True, help="Skip waiting for service stability")
@click.option("--sha", default=None, help="Use pre-built frontend from CI (git SHA)")
@click.pass_context
def update_dashboard(ctx: click.Context, docker: bool, skip_health: bool, sha: str | None):
```

Modify `_build_and_upload_frontend` to accept an optional sha, and in the function body, if sha is provided, download from CI instead of building locally:

```python
def _build_and_upload_frontend(session, safe_name, project_root, sha: str | None = None):
    """Build Next.js frontend and upload tarball to S3."""
    if sha:
        full_sha = _resolve_commit_sha(sha)
        s3_key = f"dashboard/{full_sha}/frontend.tar.gz"
        click.echo(f"  Downloading pre-built frontend from CI ({sha[:8]})...")
        import tempfile
        with tempfile.NamedTemporaryFile(suffix=".tar.gz", delete=False) as tmp:
            tarball_path = tmp.name
        try:
            _download_ci_artifact(session, s3_key, tarball_path)
        except Exception:
            raise click.ClickException(
                f"CI artifact not found: s3://{CI_ARTIFACTS_BUCKET}/{s3_key}\n"
                f"Check CI: gh run list --repo Metta-AI/cogents-v1 --workflow docker-build-dashboard.yml"
            )
    else:
        tarball_path, _size_mb = _build_dashboard_tarball(project_root)

    # Upload to cogent's sessions bucket
    click.echo("  Uploading to S3...")
    t1 = time.monotonic()
    bucket = _get_sessions_bucket(session, safe_name)
    try:
        s3_uri = _upload_dashboard_tarball(session, bucket, tarball_path)
    finally:
        os.unlink(tarball_path)
```

Update the call site in `update_dashboard` to pass sha:

```python
    _build_and_upload_frontend(session, safe_name, project_root, sha=sha)
```

**Step 4: Commit**

```bash
git add src/cogtainer/update_cli.py
git commit -m "feat(ci): add --sha flag to deploy commands for pre-built CI assets"
```

---

### Task 6: Grant CI Artifacts Bucket Read to Polis Admin Role

**Files:**
- Modify: `src/polis/cdk/stacks/core.py`

**Step 1: Grant read access to the admin role**

The deploy CLI uses the polis admin role. Add after the `grant_read_write` for github_actions_role:

```python
        self.ci_artifacts_bucket.grant_read(self.admin_role)
```

This ensures `cogent update lambda --sha` and `cogent update dashboard --sha` can download artifacts.

**Step 2: Commit**

```bash
git add src/polis/cdk/stacks/core.py
git commit -m "feat(ci): grant polis admin role read access to CI artifacts bucket"
```

---

### Post-Deploy Steps

After merging:

1. **Deploy CDK:** `polis update` to create the S3 bucket and update IAM policies
2. **Set GitHub variable:** `gh variable set CI_ARTIFACTS_BUCKET --body "cogent-polis-ci-artifacts" --repo Metta-AI/cogents-v1` (optional, bucket name is hardcoded but useful for reference)
3. **Verify:** Push a change to `dashboard/` or `src/cogos/` and confirm CI uploads artifacts
4. **Test deploy:** `cogent <name> cogtainer update lambda --sha <sha>` and `cogent <name> dashboard deploy --sha <sha>`
