# GCP Deployment

## Scope
Run the production Uforia stack on GKE in the `polynomial-uforia` project with:
- Cloud SQL Postgres as the runtime store
- ArgoCD syncing from `Polynomial-Protocol/iac-config`
- GitHub Actions building and publishing images on each push to `main` / `master`

The current public production endpoint is:
- [https://uforia.polynomial.fi](https://uforia.polynomial.fi)

Current production runtime intentionally excludes:
- local Parquet/runtime file serving
- Mac Mini `launchd` as the primary production host

Production still runs `uforia-microstructure` because the overview price series depend on
live `microstructure_bar` updates.

## Repos
- app repo: `/Users/mubarisnk/dev/personal/uforia`
- infra repo: `/Users/mubarisnk/dev/polynomial/iac-config`

Relevant production manifests live in:
- `/Users/mubarisnk/dev/polynomial/iac-config/services/uforia/prod`
- `/Users/mubarisnk/dev/polynomial/iac-config/argocd/apps/uforia.yaml`

## Production architecture
- GCP project: `polynomial-uforia`
- region: `europe-north1`
- cluster: `cluster`
- Artifact Registry:
  - `europe-north1-docker.pkg.dev/polynomial-uforia/uforia/web`
  - `europe-north1-docker.pkg.dev/polynomial-uforia/uforia/api`
  - `europe-north1-docker.pkg.dev/polynomial-uforia/uforia/python`
- ingress host: `uforia.polynomial.fi`
- DB: Cloud SQL Postgres
- runtime namespace: `uforia`

Production workloads:
- `uforia-web`
- `uforia-api`
- `uforia-microstructure`
- `uforia-options-archiver-btc`
- `uforia-options-archiver-eth`
- `uforia-refresh-hot-derived`
- `uforia-refresh-hot`
- `uforia-refresh-hourly-derived`
- `uforia-refresh-hourly`

Current refresh cadence in production:
- options ingestion: continuous daemon, polled every `5s`
- options dashboards/signals: hourly snapshots, with hot refresh orchestration keeping the latest hour current
- perps/composite derived signals: hot refresh every `5m` plus hourly catch-up
- overview prices: live microstructure bars plus API-side backfill for degraded/missing venue data

## Required GitHub secrets
Set these in `Polynomial-Protocol/uforia`:

- `GCP_WORKLOAD_IDENTITY_PROVIDER`
  - `projects/162275958321/locations/global/workloadIdentityPools/github-actions/providers/uforia-github`
- `GCP_ARTIFACT_PUSH_SA`
  - `github-ci-repo-uforia-uforia@polynomial-uforia.iam.gserviceaccount.com`
- `GH_PAT`
  - org-level secret is fine
  - must have write access to `Polynomial-Protocol/iac-config`

Optional:
- `SLACK_DEPLOY_WEBHOOK`

## One-time infrastructure bootstrap
Authenticate locally:

```bash
gcloud auth application-default login
gcloud auth application-default set-quota-project polynomial-uforia
gcloud config set project polynomial-uforia
pulumi login gs://polynomial-pulumi-statefiles
```

Run Pulumi stacks:

```bash
cd /Users/mubarisnk/dev/polynomial/iac-config/pulumi/projects/polynomial-gcp-network-main
pulumi stack select organization/polynomial-gcp-network-main/uforia-prod
pulumi up

cd /Users/mubarisnk/dev/polynomial/iac-config/pulumi/projects/polynomial-gcp-cluster
pulumi stack select organization/polynomial-gcp-cluster/uforia-prod
pulumi up

cd /Users/mubarisnk/dev/polynomial/iac-config/pulumi/projects/polynomial-service-repos
pulumi stack select organization/polynomial-service-repos/uforia-prod
pulumi up

cd /Users/mubarisnk/dev/polynomial/iac-config/pulumi/projects/uforia-postgres
pulumi stack select organization/uforia-postgres/uforia-prod
pulumi up

cd /Users/mubarisnk/dev/polynomial/iac-config/pulumi/projects/polynomial-monitoring
pulumi stack select organization/polynomial-monitoring/uforia-prod
pulumi up
```

Get cluster credentials:

```bash
gcloud container clusters get-credentials cluster --region europe-north1 --project polynomial-uforia
```

Apply Argo app definitions if needed:

```bash
kubectl apply -k /Users/mubarisnk/dev/polynomial/iac-config/argocd
```

## Database bootstrap
The production runtime expects:
- `UFORIA_DB_ENABLED=true`
- `UFORIA_DB_READ_MODE=db`
- `UFORIA_DB_WRITE_MODE=db`
- `UFORIA_DB_DSN` in the `uforia-runtime` secret

Run the DB migration once against the production image:

```bash
JOB=uforia-db-migrate-$(date +%s)
cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: ${JOB}
  namespace: uforia
spec:
  ttlSecondsAfterFinished: 300
  template:
    spec:
      serviceAccountName: uforia-runtime
      restartPolicy: Never
      containers:
        - name: migrate
          image: europe-north1-docker.pkg.dev/polynomial-uforia/uforia/python:main-<commit-sha>
          args: ["db", "migrate"]
          envFrom:
            - configMapRef:
                name: uforia-runtime
            - secretRef:
                name: uforia-runtime
EOF
```

## Deploy flow
Every push to `main` or `master` in `Polynomial-Protocol/uforia` runs:
- test
- build and push `web`, `api`, and `python` images
- update image tags in `Polynomial-Protocol/iac-config`
- Argo sync through `app-uforia-prod`
- smoke check against production

Main workflow:
- `/Users/mubarisnk/dev/personal/uforia/.github/workflows/prod-deploy.yml`

Image publishing notes:
- production images are published as immutable tags of the form `main-<full-commit-sha>`
- the publish workflow is idempotent on reruns and skips pushes when the immutable tag already exists
- production images are built for `linux/amd64`

PyPI release workflow for the terminal client is separate:
- `/Users/mubarisnk/dev/personal/uforia/.github/workflows/terminal-release.yml`

## Manual rollout
If you need to force a rollout before the next automated sync:

1. Build and push images from the app repo.
2. Update:
   - `/Users/mubarisnk/dev/polynomial/iac-config/services/uforia/prod/kustomization.yaml`
3. Push `iac-config`.
4. Force Argo refresh if needed:

```bash
kubectl -n argocd annotate application app-uforia-prod argocd.argoproj.io/refresh=hard --overwrite
```

5. If necessary, apply the kustomization directly:

```bash
kubectl apply -k /Users/mubarisnk/dev/polynomial/iac-config/services/uforia/prod
```

## Verification
Argo:

```bash
kubectl -n argocd get application app-uforia-prod
kubectl -n argocd get application app-uforia-prod -o jsonpath='{.status.sync.status}{" "}{.status.health.status}{" "}{.status.sync.revision}{"\n"}'
```

Runtime:

```bash
kubectl -n uforia get deploy
kubectl -n uforia get cronjob
kubectl -n uforia get pods
kubectl -n uforia top pods
kubectl -n uforia get externalsecret,secret
```

Public endpoints:

```bash
curl -fsS https://uforia.polynomial.fi/api/v1/system/health | jq
curl -fsS https://uforia.polynomial.fi/api/v1/options/dashboards | jq '.data | length'
curl -fsS 'https://uforia.polynomial.fi/api/v1/prices/series?asset=BTC&venue_scope=all&timeframe=15m&max_points=16' | jq '.data.points[-1].ts'
```

CLI:

```bash
uforia system show
uforia overview show
uforia options dashboards
```

## Known operational notes
- The refresh jobs are DB-serialized with advisory locks so hot/hourly jobs do not overlap by scope.
- Current advisory lock keys:
  - options-only: `72001`
  - derived-only: `72002`
  - all-scope: `72003`
- The refresh scripts invoke the installed venv-managed admin binary directly:
  - `/app/.venv/bin/uforia-admin`
  - do not swap this back to `uv run uforia-admin`; that adds avoidable install/build overhead inside the worker pods
- If hot refresh jobs complete instantly with `Hot refresh already running.`, check the local `.runtime/refresh-hot.lock` behavior alongside the DB advisory lock wrapper. The lock must be acquired after the advisory-lock re-exec path, otherwise the child process self-skips before doing real work.
- If `prices` look stale while `/api/v1/prices` is fresh, check `/api/v1/prices/series` and the `all`-venue backfill path in `DefaultPriceProvider`.
- If the overview chart shows a visible gap or flat bridge:
  1. compare `venue_scope=binance`, `bybit`, and `all`
  2. inspect `/api/v1/prices/series?...&max_points=240`
  3. confirm the live `uforia-microstructure` bars sidecar is healthy
  4. confirm the API is backfilling degraded trailing bars rather than only internal holes
- If `/options` loads a long shell state or an empty dashboard table:
  1. check `/api/v1/options/dashboards`
  2. check that same endpoint from inside the `uforia-web` pod to measure in-cluster latency
  3. confirm the web client timeout and `/proxy` route timeout are longer than that latency
  4. if the endpoint is slow but returns data, inspect `DefaultOptionsDashboardProvider.get_latest()` before changing UI timeouts
  The web client now renders an explicit fallback instead of silently showing an empty page, but slow dashboard reads will still present as a long shell if the API is near the timeout budget.
- qVVIX should keep updating even when `rvvix_14d` / `rvvix_30d` are null.
  - `qvvix_30d` only needs short recent hourly vol-index history
  - `rvvix_14d` / `rvvix_30d` require much longer history windows and can remain null while qVVIX is live
- If qVVIX stalls while the underlying vol-index is current, treat it as a refresh/runtime issue first.
- If perps/composite stay stale while options are current:
  1. check `uforia-refresh-hot-derived` and `uforia-refresh-hourly-derived`
  2. inspect advisory lock `72002` in Postgres and identify the current holder
  3. confirm the running derived pod is actually doing work and not just holding the lock idle
  4. use `kubectl -n uforia top pods` to confirm the derived pod has enough memory to finish catch-up work
  5. perps signal construction should only use the recent hot context window anchored to the requested `end`; if catch-up is loading multi-day raw microstructure windows again, inspect `build_perps_signal_series()`
- Argo UI can return `permission denied` for `app-uforia-prod` if the UI request carries the wrong project filter. The direct app URL should use `project=this-prod`.
