# GCP Deployment

## Scope
Run the production Uforia stack on GKE in the `polynomial-uforia` project with:
- Cloud SQL Postgres as the runtime store
- ArgoCD syncing from `Polynomial-Protocol/iac-config`
- GitHub Actions building and publishing images on each push to `main` / `master`

The current public production endpoint is:
- [https://uforia.polynomial.fi](https://uforia.polynomial.fi)

Current production runtime intentionally excludes:
- live microstructure deployment
- local Parquet/runtime file serving
- Mac Mini `launchd` as the primary production host

## Repos
- app repo: `/Users/mubarisnk/dev/personal/uforia`
- infra repo: `/Users/mubarisnk/dev/polynomial/iac-config`

Relevant production manifests live in:
- `/Users/mubarisnk/dev/polynomial/iac-config/services/uforia/prod`
- `/Users/mubarisnk/dev/polynomial/iac-config/argocd/apps/uforia.yaml`

## Production architecture
- GCP project: `polynomial-uforia`
- region: `europe-north1`
- cluster: `cluster`
- Artifact Registry:
  - `europe-north1-docker.pkg.dev/polynomial-uforia/uforia/web`
  - `europe-north1-docker.pkg.dev/polynomial-uforia/uforia/api`
  - `europe-north1-docker.pkg.dev/polynomial-uforia/uforia/python`
- ingress host: `uforia.polynomial.fi`
- DB: Cloud SQL Postgres
- runtime namespace: `uforia`

Production workloads:
- `uforia-web`
- `uforia-api`
- `uforia-options-archiver-btc`
- `uforia-options-archiver-eth`
- `uforia-refresh-hot-derived`
- `uforia-refresh-hot-options`
- `uforia-refresh-hourly-derived`
- `uforia-refresh-hourly-options`

## Required GitHub secrets
Set these in `Polynomial-Protocol/uforia`:

- `GCP_WORKLOAD_IDENTITY_PROVIDER`
  - `projects/162275958321/locations/global/workloadIdentityPools/github-actions/providers/uforia-github`
- `GCP_ARTIFACT_PUSH_SA`
  - `github-ci-repo-uforia-uforia@polynomial-uforia.iam.gserviceaccount.com`
- `GH_PAT`
  - org-level secret is fine
  - must have write access to `Polynomial-Protocol/iac-config`

Optional:
- `SLACK_DEPLOY_WEBHOOK`

## One-time infrastructure bootstrap
Authenticate locally:

```bash
gcloud auth application-default login
gcloud auth application-default set-quota-project polynomial-uforia
gcloud config set project polynomial-uforia
pulumi login gs://polynomial-pulumi-statefiles
```

Run Pulumi stacks:

```bash
cd /Users/mubarisnk/dev/polynomial/iac-config/pulumi/projects/polynomial-gcp-network-main
pulumi stack select organization/polynomial-gcp-network-main/uforia-prod
pulumi up

cd /Users/mubarisnk/dev/polynomial/iac-config/pulumi/projects/polynomial-gcp-cluster
pulumi stack select organization/polynomial-gcp-cluster/uforia-prod
pulumi up

cd /Users/mubarisnk/dev/polynomial/iac-config/pulumi/projects/polynomial-service-repos
pulumi stack select organization/polynomial-service-repos/uforia-prod
pulumi up

cd /Users/mubarisnk/dev/polynomial/iac-config/pulumi/projects/uforia-postgres
pulumi stack select organization/uforia-postgres/uforia-prod
pulumi up

cd /Users/mubarisnk/dev/polynomial/iac-config/pulumi/projects/polynomial-monitoring
pulumi stack select organization/polynomial-monitoring/uforia-prod
pulumi up
```

Get cluster credentials:

```bash
gcloud container clusters get-credentials cluster --region europe-north1 --project polynomial-uforia
```

Apply Argo app definitions if needed:

```bash
kubectl apply -k /Users/mubarisnk/dev/polynomial/iac-config/argocd
```

## Database bootstrap
The production runtime expects:
- `UFORIA_DB_ENABLED=true`
- `UFORIA_DB_READ_MODE=db`
- `UFORIA_DB_WRITE_MODE=db`
- `UFORIA_DB_DSN` in the `uforia-runtime` secret

Run the DB migration once against the production image:

```bash
JOB=uforia-db-migrate-$(date +%s)
cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: ${JOB}
  namespace: uforia
spec:
  ttlSecondsAfterFinished: 300
  template:
    spec:
      serviceAccountName: uforia-runtime
      restartPolicy: Never
      containers:
        - name: migrate
          image: europe-north1-docker.pkg.dev/polynomial-uforia/uforia/python:main-<commit-sha>
          args: ["db", "migrate"]
          envFrom:
            - configMapRef:
                name: uforia-runtime
            - secretRef:
                name: uforia-runtime
EOF
```

## Deploy flow
Every push to `main` or `master` in `Polynomial-Protocol/uforia` runs:
- test
- build and push `web`, `api`, and `python` images
- update image tags in `Polynomial-Protocol/iac-config`
- Argo sync through `app-uforia-prod`
- smoke check against production

Main workflow:
- `/Users/mubarisnk/dev/personal/uforia/.github/workflows/prod-deploy.yml`

PyPI release workflow for the terminal client is separate:
- `/Users/mubarisnk/dev/personal/uforia/.github/workflows/terminal-release.yml`

## Manual rollout
If you need to force a rollout before the next automated sync:

1. Build and push images from the app repo.
2. Update:
   - `/Users/mubarisnk/dev/polynomial/iac-config/services/uforia/prod/kustomization.yaml`
3. Push `iac-config`.
4. Force Argo refresh if needed:

```bash
kubectl -n argocd annotate application app-uforia-prod argocd.argoproj.io/refresh=hard --overwrite
```

5. If necessary, apply the kustomization directly:

```bash
kubectl apply -k /Users/mubarisnk/dev/polynomial/iac-config/services/uforia/prod
```

## Verification
Argo:

```bash
kubectl -n argocd get application app-uforia-prod
kubectl -n argocd get application app-uforia-prod -o jsonpath='{.status.sync.status}{" "}{.status.health.status}{" "}{.status.sync.revision}{"\n"}'
```

Runtime:

```bash
kubectl -n uforia get deploy
kubectl -n uforia get cronjob
kubectl -n uforia get pods
kubectl -n uforia get externalsecret,secret
```

Public endpoints:

```bash
curl -fsS https://uforia.polynomial.fi/api/v1/system/health | jq
curl -fsS https://uforia.polynomial.fi/api/v1/options/dashboards | jq '.data | length'
curl -fsS 'https://uforia.polynomial.fi/api/v1/prices/series?asset=BTC&venue_scope=all&timeframe=15m&max_points=16' | jq '.data.latest_available_at'
```

CLI:

```bash
uforia system show
uforia overview show
uforia options dashboards
```

## Known operational notes
- The refresh jobs are DB-serialized with advisory locks so hot/hourly jobs do not overlap by scope.
- If `prices` look stale while `/api/v1/prices` is fresh, check `/api/v1/prices/series` and the `all`-venue backfill path in `DefaultPriceProvider`.
- If `/options` loads an empty dashboard table, check `/api/v1/options/dashboards` latency first. The web client now renders an explicit error state on load failure instead of silently rendering an empty list.
- If qVVIX is missing only at the most recent point, inspect the latest `vvix_series` rows. Production refresh now targets the current hour and resolves qVVIX against the latest available vol-index point at or before the requested timestamp.
