# qRemoteX — Publishing & Distribution Implementation Plan

> **Version:** 2.1 | **Date:** March 2026 | **Status:** ✅ Implementation Complete (Post-Review Fixes Applied)  
> **Author:** qRaptor Engineering  
> **Depends On:** qRemoteX core implementation (complete)

---

## Table of Contents

1. [Overview & Goals](#1-overview--goals)
2. [Current State Analysis](#2-current-state-analysis)
3. [Distribution Strategy](#3-distribution-strategy)
4. [Implementation Tasks](#4-implementation-tasks)
   - [Task 1: Upgrade Python to 3.13](#task-1-upgrade-python-to-313)
   - [Task 2: Fix Dockerfile (No Source Code in Image)](#task-2-fix-dockerfile-no-source-code-in-image)
   - [Task 3: Add Entrypoint Script for User Packages](#task-3-add-entrypoint-script-for-user-packages)
   - [Task 4: Azure DevOps Pipeline — Dev (ACR + Internal)](#task-4-azure-devops-pipeline--dev-acr--internal)
   - [Task 5: Azure DevOps Pipeline — Release (Public Push)](#task-5-azure-devops-pipeline--release-public-push)
   - [Task 5a: Trivy Vulnerability Scan Details](#task-5a-trivy-vulnerability-scan-details)
   - [Task 6: PyPI Package Publishing](#task-6-pypi-package-publishing)
   - [Task 7: Multi-Architecture Docker Build](#task-7-multi-architecture-docker-build)
   - [Task 8: Update Helm Chart for Public Image](#task-8-update-helm-chart-for-public-image)
   - [Task 9: User Documentation — Extending qRemoteX](#task-9-user-documentation--extending-qremotex)
   - [Task 10: Update Architecture Doc Section 17](#task-10-update-architecture-doc-section-17)
   - [Task 11: Studio UI — Remote Executors Install & Download Page](#task-11-studio-ui--remote-executors-install--download-page)
5. [Azure DevOps Pipeline Architecture](#5-azure-devops-pipeline-architecture)
6. [User Package Installation Patterns](#6-user-package-installation-patterns)
7. [Security Considerations](#7-security-considerations)
8. [Verification Checklist](#8-verification-checklist)

---

## 1. Overview & Goals

qRemoteX is a **customer-deployed agent** — unlike other qRaptor services (platform-admin, qRunX, gateway) which run in our cluster, qRemoteX runs on **customer infrastructure**. This fundamentally changes the distribution model:

| Aspect | Internal Services (qRunX, gateway, etc.) | qRemoteX (Customer-Deployed) |
|--------|------------------------------------------|------------------------------|
| Image registry | Private ACR (`qraptordevregistry.azurecr.io`) | **Public** (Docker Hub / GHCR) |
| Package registry | N/A | **Public** (PyPI) |
| Source code | Stays in ACR image | **Must NOT be in image** |
| User extensibility | N/A | Users install their own pip packages |
| Multi-arch | amd64 only (our AKS) | **amd64 + arm64** (customer hardware) |

### Goals

1. **G1:** Docker image published to Docker Hub: `qraptor/qremotex:2.0.0` (no source code)
2. **G2:** Python package published to PyPI: `pip install qremotex`
3. **G3:** Users can install additional packages (both Docker and pip modes)
4. **G4:** Python 3.13 across all files
5. **G5:** Azure DevOps pipelines for dev (ACR) and release (public registries)
6. **G6:** Multi-arch images (linux/amd64 + linux/arm64)
7. **G7:** Trivy vulnerability scan gate before any public push
8. **G8:** Binary download served from Studio UI (Remote Executors page)

### Environment-Aware Executor Gateway URL

The executor gateway URL is **environment-specific**, controlled by the `PRODUCTION_DOMAIN` env var on the `qraptor-production-gateway`:

| Environment | `PRODUCTION_DOMAIN` | Executor Gateway URL |
|-------------|--------------------|-----------------------|
| **Production** (Azure AKS) | `qraptor.app` (default) | `wss://exec-{shortId}.qraptor.app/ws/executor` |
| **Development** (AWS) | `dev.qraptor.app` | `wss://exec-{shortId}.dev.qraptor.app/ws/executor` |

The Studio UI reads `NEXT_PUBLIC_PRODUCTION_DOMAIN` (baked at build time) to display the correct URL in the setup wizard. Examples in this document use the **production** domain (`qraptor.app`) unless noted otherwise.

---

## 2. Current State Analysis

### What Exists

| File | Status | Issue |
|------|--------|-------|
| `pyproject.toml` | ✅ Well-structured | Python 3.11 references, needs 3.13 |
| `Dockerfile` | ⚠️ Functional but flawed | Copies source code into runtime image; uses Python 3.11 |
| `chart/` (Helm) | ✅ Good | References `ghcr.io/qraptor/qremotex` — switch to Docker Hub |
| `requirements.txt` | ✅ Minimal | Only core deps, correct |
| `scripts/build-binary.sh` | ✅ Works | PyInstaller binary build |
| `qremotex.spec` | ✅ Works | PyInstaller spec |
| `docker-compose.yml` | ✅ Good | Local dev testing |
| Azure DevOps pipeline | ❌ Missing | No pipeline exists for qRemoteX |

### Old Executor Comparison (`a2-svc-remote-py-executor`)

The old executor used:
- Python 3.12.9 in Dockerfile
- PyInstaller binary in Docker (shipped compiled binary, not source)
- Single `main.py` + `requirements.txt` (only `aiohttp`)
- No pipeline file found in old repo (manual builds assumed)

### Existing Pipeline Patterns (from other qRaptor services)

All existing services follow this pattern:

```
Pattern: azure-pipelines/dev-pipeline.yml
Pool:    JNXT-DEV-POOL (self-hosted agent)
ACR:     qraptordevregistry.azurecr.io
K8s:     qraptor-dev-aks-connection
Vars:    qraptor-dev-variables (variable group)
Steps:   Build → Push ACR → Token replace K8s manifests → Deploy AKS
```

**Key observations from existing pipelines:**
- **DMS, Platform Admin, Channel:** Java services — `gradlew build` → Docker build → ACR push → K8s deploy
- **Python Runtime:** Helm-based dual deployment (preview + production), Docker Hub not used
- **Vibe Code:** Two-image strategy (base image + thin app layer), ACR only
- **Dependant Services:** Infrastructure-only (ConfigMap, secrets), no image build
- **All use:** `pool: JNXT-DEV-POOL`, `qraptor-dev-acr-connection`, `qraptor-dev-aks-connection`

**qRemoteX is different:** It needs a **public release pipeline** in addition to the standard dev pipeline.

---

## 3. Distribution Strategy

### Decision: Public Registries

**Docker Hub** for images, **PyPI** for pip packages. Rationale:

1. **qRemoteX is useless without a qRaptor API key** — the binary is the gate, not the download
2. **Zero infrastructure** — no self-hosted registry to run/maintain
3. **User trust** — `docker pull qraptor/qremotex` and `pip install qremotex` are universally trusted
4. **CDN bandwidth** — Docker Hub and PyPI absorb egress costs

### Registry Mapping

| Artifact | Registry | Path | Auth Gate |
|----------|----------|------|-----------|
| Docker image (dev) | Azure ACR (private) | `qraptordevregistry.azurecr.io/qremotex:dev-{buildId}` | Azure AD |
| Docker image (release) | Docker Hub (public) | `qraptor/qremotex:2.0.x` / `:latest` | API key at runtime |
| Python wheel (release) | PyPI (public) | `pip install qremotex==2.0.x` | API key at runtime |
| Helm chart | Docker Hub OCI (public) | `oci://docker.io/qraptor/qremotex-chart` | API key at runtime |
| Binary (release) | DMS Azure Blob (`dmsdata` container) → Studio UI | `releases/qremotex/{version}/` path in DMS blob | API key at runtime |

### Two-Pipeline Architecture

```
┌──────────────────────────────────────────────────────────┐
│  Dev Pipeline (every push to dev branch)                 │
│  ────────────────────────────────────────                │
│  Trigger: push to 1.0.x-* branch                        │
│  Steps:  Lint → Test → Build image → Push to ACR         │
│  Output: qraptordevregistry.azurecr.io/qremotex:dev-123 │
│  Purpose: Internal testing, QA                           │
└──────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────┐
│  Release Pipeline (auto on push to 2.0.x branch)         │
│  ─────────────────────────────────────────────           │
│  Trigger: push to 2.0.x branch                           │
│  Version: 2.0.{BuildId} (auto-incrementing minor)        │
│  Steps:  Test → Trivy scan → Build multi-arch image      │
│          → Push Docker Hub → Build wheel → Push PyPI     │
│          → Build binary → Upload to DMS blob (dmsdata)   │
│          → Push Helm chart                               │
│  Output: qraptor/qremotex:2.0.{BuildId} (Docker Hub)    │
│          qremotex==2.0.{BuildId} (PyPI)                  │
│          qremotex-2.0.x-linux-amd64 (Studio download)    │
│  Purpose: Customer-facing release                        │
└──────────────────────────────────────────────────────────┘
```

### Versioning Strategy

qRemoteX uses **auto-incrementing minor versions** on the `2.0.x` release branch:

| Aspect | Value |
|--------|-------|
| **Release branch** | `2.0.x` |
| **Version format** | `2.0.{BuildId}` (e.g., `2.0.0`, `2.0.1`, `2.0.2`, ...) |
| **First release** | `2.0.0` |
| **Increment trigger** | Every push to `2.0.x` branch |
| **Auto-version logic** | Pipeline reads last published version from Docker Hub tags, increments patch |
| **pyproject.toml** | Updated dynamically by pipeline before build |
| **Docker tag** | `qraptor/qremotex:2.0.{N}` + `qraptor/qremotex:latest` |
| **PyPI version** | `qremotex==2.0.{N}` |

**Why auto-increment on push (not manual)?**
- Reduces release friction — merge to `2.0.x` = automatic release
- Every push to release branch is intentional (use PRs to gate)
- BuildId from Azure DevOps is monotonically increasing
- No manual version parameter to mistype
- `2.0.x` branch is protected — only approved PRs merge into it

---

## 4. Implementation Tasks

### Task 1: Upgrade Python to 3.13 ✅ DONE

**Files to change:**

| # | File | Change |
|---|------|--------|
| 1 | `pyproject.toml` | `requires-python = ">=3.11"` → `">=3.13"` |
| 2 | `pyproject.toml` | ruff `target-version = "py311"` → `"py313"` |
| 3 | `Dockerfile` | Both `FROM python:3.11-slim` → `python:3.13-slim` |
| 4 | `src/qremotex/__init__.py` | No change needed (version string only) |

**Verification:**
```bash
cd PROJECTS/qRemoteX
docker build -t qremotex-test .
docker run --rm qremotex-test python --version  # Should show 3.13.x
```

**Risk:** Low. All deps (`aiohttp>=3.9`, `psutil>=5.9`, `pydantic>=2.0`) support 3.13. The `qraptor-python-runtime` service already uses `python:3.13-slim` successfully.

---

### Task 2: Fix Dockerfile (No Source Code in Image) ✅ DONE

**Problem:** Current Dockerfile copies `src/` into the runtime image AND installs the wheel. After `pip install .`, the source in `src/` is dead weight and a security concern (proprietary code exposed).

**Solution:** Build wheel in stage 1, install only the wheel in stage 2.

**New Dockerfile structure:**

```dockerfile
# ── Stage 1: Build wheel ─────────────────────────────────────
FROM python:3.13-slim AS builder

WORKDIR /build
COPY pyproject.toml README.md ./
COPY src/ ./src/

# Build the wheel
RUN pip install --no-cache-dir build \
    && python -m build --wheel --outdir /build/dist

# Install runtime + tools deps into a prefix
RUN pip install --no-cache-dir --prefix=/install /build/dist/*.whl \
    && pip install --no-cache-dir --prefix=/install "asyncpg>=0.29" "httpx>=0.27" "beautifulsoup4>=4.12"

# ── Stage 2: Runtime (NO source code) ────────────────────────
FROM python:3.13-slim

LABEL org.opencontainers.image.title="qRemoteX" \
      org.opencontainers.image.description="qRaptor Remote Execution Agent" \
      org.opencontainers.image.vendor="qRaptor" \
      org.opencontainers.image.version="2.0.0" \
      org.opencontainers.image.url="https://qraptor.ai" \
      org.opencontainers.image.source="https://qraptor.ai/docs/qremotex"

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1

# Security: install curl for healthcheck, then clean up
RUN apt-get update \
    && apt-get install -y --no-install-recommends curl tini \
    && rm -rf /var/lib/apt/lists/*

# Security: non-root user
RUN groupadd -r qremotex && useradd -r -g qremotex -d /app -s /sbin/nologin qremotex

WORKDIR /app

# Copy ONLY installed packages (no source code)
COPY --from=builder /install /usr/local

# Entrypoint script for optional user package installation
COPY scripts/docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
RUN chmod +x /usr/local/bin/docker-entrypoint.sh

# Create writable dirs (owned by qremotex)
RUN mkdir -p /app/downloads /app/user-packages \
    && chown -R qremotex:qremotex /app

# Drop privileges
USER qremotex

# Health-check — verifies the Python package is importable
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
    CMD python -c "from qremotex.config import ExecutorConfig; print('ok')"

ENTRYPOINT ["/usr/bin/tini", "--", "docker-entrypoint.sh"]
CMD ["python", "-m", "qremotex.main"]
```

**Verification after build:**
```bash
# Source code should NOT exist in runtime image
docker run --rm qremotex-test ls /app/src  # Should fail (no such directory)
docker run --rm qremotex-test find / -name "*.py" -path "*/qremotex/*" 2>/dev/null
# Should only show files in /usr/local/lib/python3.13/site-packages/qremotex/ (compiled .pyc)
```

---

### Task 3: Add Entrypoint Script for User Packages ✅ DONE

**Purpose:** Allow users to install additional pip packages without building a custom Docker image.

**File:** `scripts/docker-entrypoint.sh`

```bash
#!/bin/sh
set -e

# ──────────────────────────────────────────────────────────
# qRemoteX Docker Entrypoint
# ──────────────────────────────────────────────────────────
# Installs user-specified packages before starting the agent.
#
# Usage:
#   Option A: Mount a requirements file
#     docker run -v /path/to/requirements.txt:/app/extra-requirements.txt ...
#
#   Option B: Pass packages via environment variable
#     docker run -e QRAPTOR_EXTRA_PACKAGES="pandas torch boto3" ...
#
#   Option C: No extras (default — just start the agent)
#     docker run ...
# ──────────────────────────────────────────────────────────

# Install from mounted requirements file
if [ -f /app/extra-requirements.txt ]; then
    echo "[qRemoteX] Installing packages from /app/extra-requirements.txt ..."
    pip install --no-cache-dir --user -r /app/extra-requirements.txt
    echo "[qRemoteX] Extra packages installed successfully."
fi

# Install from environment variable
if [ -n "$QRAPTOR_EXTRA_PACKAGES" ]; then
    echo "[qRemoteX] Installing packages: $QRAPTOR_EXTRA_PACKAGES ..."
    pip install --no-cache-dir --user $QRAPTOR_EXTRA_PACKAGES
    echo "[qRemoteX] Extra packages installed successfully."
fi

# Execute the main command
exec "$@"
```

**User experience:**

```bash
# No extras — just run
docker run -d \
  -e QRAPTOR_GATEWAY_URL=wss://exec-pRj7kM.qraptor.app/ws/executor \
  -e QRAPTOR_API_KEY=qrx_prod_... \
  qraptor/qremotex:latest

# With inline packages
docker run -d \
  -e QRAPTOR_GATEWAY_URL=wss://exec-pRj7kM.qraptor.app/ws/executor \
  -e QRAPTOR_API_KEY=qrx_prod_... \
  -e QRAPTOR_EXTRA_PACKAGES="pandas torch boto3" \
  qraptor/qremotex:latest

# With requirements file
docker run -d \
  -v /path/to/my-requirements.txt:/app/extra-requirements.txt:ro \
  -e QRAPTOR_GATEWAY_URL=wss://exec-pRj7kM.qraptor.app/ws/executor \
  -e QRAPTOR_API_KEY=qrx_prod_... \
  qraptor/qremotex:latest

# Extending base image (recommended for production)
# Dockerfile.custom:
#   FROM qraptor/qremotex:latest
#   RUN pip install pandas torch boto3 my-internal-lib
```

---

### Task 4: Azure DevOps Pipeline — Dev (ACR + Internal) ✅ DONE

**File:** `azure-pipelines/dev-pipeline.yml`

This follows the **exact same pattern** as existing qRaptor Python services (python-runtime, vibe-code), adapted for qRemoteX.

```yaml
# =============================================================================
# qRemoteX - Dev Pipeline
# =============================================================================
# Builds and pushes qRemoteX Docker image to private ACR for internal testing.
#
# NOTE: qRemoteX is NOT deployed to our AKS cluster — it's a customer-deployed
# agent. This pipeline only builds the image for QA testing.
#
# For public release (Docker Hub + PyPI), see: release-pipeline.yml
#
# Stages:
#   1. Lint & Test (ruff + pytest)
#   2. Build Docker image and push to ACR
# =============================================================================

trigger:
  branches:
    include:
      - 1.0.x-*
  paths:
    include:
      - "*"

pr: none

variables:
  - group: qraptor-dev-variables
  - name: serviceName
    value: "qremotex"
  - name: imageTag
    value: "dev-$(Build.BuildId)"
  - name: acrServiceConnection
    value: "qraptor-dev-acr-connection"
  - name: acrName
    value: "qraptordevregistry.azurecr.io"

pool: JNXT-DEV-POOL

stages:
  - stage: Test
    displayName: "Lint & Test"
    jobs:
      - job: LintAndTest
        displayName: "Ruff Lint + Pytest"
        steps:
          - checkout: self

          - task: UsePythonVersion@0
            displayName: "Use Python 3.13"
            inputs:
              versionSpec: "3.13"
              addToPath: true

          - script: |
              pip install -e ".[dev,tools]"
            displayName: "Install dependencies"

          - script: |
              ruff check src/ tests/
            displayName: "Ruff lint"

          - script: |
              pytest tests/ -v --junitxml=test-results.xml --cov=src/qremotex --cov-report=xml:coverage.xml
            displayName: "Run tests"

          - task: PublishTestResults@2
            displayName: "Publish Test Results"
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: "JUnit"
              testResultsFiles: "test-results.xml"
              mergeTestResults: true
              failTaskOnFailedTests: true

          - task: PublishCodeCoverageResults@2
            displayName: "Publish Coverage"
            condition: succeededOrFailed()
            inputs:
              codeCoverageTool: "Cobertura"
              summaryFileLocation: "coverage.xml"

  - stage: Build
    displayName: "Build & Push to ACR"
    dependsOn: Test
    condition: succeeded()
    jobs:
      - job: DockerBuild
        displayName: "Build Docker Image"
        steps:
          - checkout: self

          - task: Docker@2
            displayName: "Build & Push Docker Image"
            inputs:
              command: "buildAndPush"
              containerRegistry: "$(acrServiceConnection)"
              repository: "$(serviceName)"
              dockerfile: "Dockerfile"
              tags: |
                $(imageTag)
                dev-latest
```

**Key differences from other services:**
- **No K8s deploy stage** — qRemoteX runs on customer infra, not our AKS
- **Python lint/test stage** — other Python services skip this in dev pipeline; we add it since there's no separate test pipeline
- **No token replacement** — no K8s manifests to template

---

### Task 5: Azure DevOps Pipeline — Release (Public Push) ✅ DONE

**File:** `azure-pipelines/release-pipeline.yml`

This is the **new pipeline** unique to qRemoteX. It builds and publishes to public registries. Triggered **automatically on every push to the `2.0.x` branch** with auto-incrementing `2.0.{BuildId}` versioning.

```yaml
# =============================================================================
# qRemoteX - Release Pipeline (Public Distribution)
# =============================================================================
# Publishes qRemoteX to public registries for customer consumption:
#   - Docker Hub: qraptor/qremotex:2.0.{BuildId}
#   - PyPI: qremotex==2.0.{BuildId}
#   - Binary: Uploaded to DMS blob storage (dmsdata container) → served via Studio UI
#   - Helm: OCI chart to Docker Hub
#
# Trigger: Automatic on push to 2.0.x branch
# Version: 2.0.{BuildId} (auto-incrementing, starting at 2.0.0)
#
# Security Gate: Trivy vulnerability scan MUST pass before any public push.
# If Trivy finds CRITICAL or HIGH vulnerabilities, the pipeline STOPS.
#
# Required Variables in 'qraptor-release-variables' group:
#   - DOCKERHUB_USERNAME: Docker Hub username (qraptor service account)
#   - DOCKERHUB_TOKEN: Docker Hub access token (secret)
#   - PYPI_API_TOKEN: PyPI API token for qremotex package (secret)
#   - AZURE_STORAGE_CONNECTION_STRING: Reuses DMS blob storage (same as qraptor-dms-service)
#
# Required Service Connections:
#   - qraptor-dockerhub-connection: Docker Hub service connection
#   - qraptor-dev-acr-connection: Azure Container Registry (internal mirror)
# =============================================================================

trigger:
  branches:
    include:
      - 2.0.x
  paths:
    include:
      - "*"

pr: none  # Releases only via direct push/merge to 2.0.x

variables:
  - group: qraptor-release-variables
  - group: qraptor-dev-variables
  - name: serviceName
    value: "qremotex"
  - name: majorMinor
    value: "2.0"
  - name: version
    value: "2.0.$(Build.BuildId)"
  - name: dockerHubRepo
    value: "qraptor/qremotex"
  - name: acrServiceConnection
    value: "qraptor-dev-acr-connection"
  - name: acrName
    value: "qraptordevregistry.azurecr.io"
  - name: binaryBlobContainer
    value: "dmsdata"              # Reuses existing DMS blob container (no new storage needed)
  - name: binaryBlobPrefix
    value: "releases/qremotex"    # Path prefix inside dmsdata container

pool: JNXT-DEV-POOL

stages:
  # ===================================================================
  # Stage 1: Validate, Test & Stamp Version
  # ===================================================================
  - stage: Validate
    displayName: "Validate & Test"
    jobs:
      - job: ValidateAndTest
        displayName: "Lint, Test & Stamp Version"
        steps:
          - checkout: self

          - task: UsePythonVersion@0
            displayName: "Use Python 3.13"
            inputs:
              versionSpec: "3.13"
              addToPath: true

          # Stamp version into pyproject.toml and __init__.py
          - script: |
              echo "Release version: $(version)"
              sed -i 's/^version = ".*"/version = "$(version)"/' pyproject.toml
              sed -i 's/^__version__ = ".*"/__version__ = "$(version)"/' src/qremotex/__init__.py
              echo "Stamped version in pyproject.toml:"
              grep '^version' pyproject.toml
              echo "Stamped version in __init__.py:"
              grep '__version__' src/qremotex/__init__.py
            displayName: "Stamp version $(version)"

          - script: |
              pip install -e ".[dev,tools]"
            displayName: "Install dependencies"

          - script: |
              ruff check src/ tests/
            displayName: "Ruff lint"

          - script: |
              pytest tests/ -v --junitxml=test-results.xml
            displayName: "Run tests"

          - task: PublishTestResults@2
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: "JUnit"
              testResultsFiles: "test-results.xml"
              failTaskOnFailedTests: true

          # Publish stamped source as artifact for downstream stages
          - task: PublishPipelineArtifact@1
            displayName: "Publish stamped source"
            inputs:
              targetPath: "$(Build.SourcesDirectory)"
              artifactName: "stamped-source"

  # ===================================================================
  # Stage 2: Build Docker Image + Trivy Security Scan
  # ===================================================================
  - stage: DockerBuildAndScan
    displayName: "Build & Security Scan"
    dependsOn: Validate
    jobs:
      - job: BuildAndTrivy
        displayName: "Build Image + Trivy Scan"
        steps:
          - task: DownloadPipelineArtifact@2
            displayName: "Download stamped source"
            inputs:
              artifactName: "stamped-source"
              targetPath: "$(Build.SourcesDirectory)"

          # Build local image for scanning (single arch, not pushed yet)
          - script: |
              docker build \
                --tag $(serviceName):scan-$(Build.BuildId) \
                --label "org.opencontainers.image.version=$(version)" \
                .
            displayName: "Build Docker image (local, for scan)"

          # ─────────────────────────────────────────────────────
          # TRIVY VULNERABILITY SCAN — SECURITY GATE
          # ─────────────────────────────────────────────────────
          # Scans the built image for OS + library vulnerabilities.
          # Pipeline FAILS if CRITICAL or HIGH CVEs are found.
          # This runs BEFORE any push to public registries.
          # ─────────────────────────────────────────────────────
          - script: |
              # Install Trivy (if not already on agent)
              if ! command -v trivy &>/dev/null; then
                echo "Installing Trivy..."
                curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
              fi
              trivy --version
            displayName: "Install Trivy"

          - script: |
              echo "=== Trivy Full Vulnerability Scan ==="
              echo "Image: $(serviceName):scan-$(Build.BuildId)"
              echo "Severity gate: CRITICAL,HIGH"
              echo ""

              # Full scan: OS packages + Python libraries
              trivy image \
                --severity CRITICAL,HIGH \
                --exit-code 1 \
                --no-progress \
                --format table \
                --vuln-type os,library \
                --ignore-unfixed \
                $(serviceName):scan-$(Build.BuildId)

              SCAN_EXIT=$?
              if [ $SCAN_EXIT -ne 0 ]; then
                echo "##vso[task.logissue type=error]Trivy found CRITICAL/HIGH vulnerabilities! Fix before releasing."
                exit 1
              fi
              echo "✅ Trivy scan passed — no CRITICAL/HIGH vulnerabilities found."
            displayName: "Trivy vulnerability scan (CRITICAL+HIGH gate)"

          # Also generate detailed JSON report for auditing
          - script: |
              trivy image \
                --format json \
                --output trivy-report-$(version).json \
                --vuln-type os,library \
                $(serviceName):scan-$(Build.BuildId)
              echo "Trivy JSON report generated: trivy-report-$(version).json"
            displayName: "Generate Trivy JSON report (audit)"

          # Generate SBOM for supply-chain compliance
          - script: |
              trivy image \
                --format spdx-json \
                --output sbom-$(version).spdx.json \
                $(serviceName):scan-$(Build.BuildId)
              echo "SBOM generated: sbom-$(version).spdx.json"
            displayName: "Generate SBOM (SPDX)"

          - task: PublishPipelineArtifact@1
            displayName: "Publish security reports"
            inputs:
              targetPath: "."
              artifactName: "security-reports-$(version)"
              # Only publish report files
            condition: always()

          # Cleanup scan image
          - script: |
              docker rmi $(serviceName):scan-$(Build.BuildId) || true
            displayName: "Cleanup scan image"
            condition: always()

  # ===================================================================
  # Stage 3: Push Docker Image to Public Registries (Multi-Arch)
  # ===================================================================
  - stage: DockerPublish
    displayName: "Publish Docker Image"
    dependsOn: DockerBuildAndScan
    condition: succeeded()
    jobs:
      - job: DockerMultiArch
        displayName: "Build & Push Multi-Arch Image"
        steps:
          - task: DownloadPipelineArtifact@2
            displayName: "Download stamped source"
            inputs:
              artifactName: "stamped-source"
              targetPath: "$(Build.SourcesDirectory)"

          # Login to Docker Hub
          - task: Docker@2
            displayName: "Login to Docker Hub"
            inputs:
              command: "login"
              containerRegistry: "qraptor-dockerhub-connection"

          # Set up Docker Buildx for multi-arch
          - script: |
              docker buildx create --name qremotex-builder --use --bootstrap
              docker buildx inspect --bootstrap
            displayName: "Setup Docker Buildx"

          # Build and push multi-arch image
          - script: |
              docker buildx build \
                --platform linux/amd64,linux/arm64 \
                --tag $(dockerHubRepo):$(version) \
                --tag $(dockerHubRepo):latest \
                --label "org.opencontainers.image.version=$(version)" \
                --label "org.opencontainers.image.revision=$(Build.SourceVersion)" \
                --label "org.opencontainers.image.created=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
                --push \
                .
            displayName: "Build & Push multi-arch (amd64 + arm64)"

          # Also push to ACR for internal reference
          - task: Docker@2
            displayName: "Build & Push to ACR"
            inputs:
              command: "buildAndPush"
              containerRegistry: "$(acrServiceConnection)"
              repository: "$(serviceName)"
              dockerfile: "Dockerfile"
              tags: |
                $(version)
                release-latest

          # Cleanup buildx
          - script: |
              docker buildx rm qremotex-builder || true
            displayName: "Cleanup Buildx"
            condition: always()

  # ===================================================================
  # Stage 4: Publish Python Package to PyPI
  # ===================================================================
  - stage: PyPIPublish
    displayName: "Publish to PyPI"
    dependsOn: DockerBuildAndScan
    condition: succeeded()
    jobs:
      - job: PublishWheel
        displayName: "Build Wheel & Upload to PyPI"
        steps:
          - task: DownloadPipelineArtifact@2
            displayName: "Download stamped source"
            inputs:
              artifactName: "stamped-source"
              targetPath: "$(Build.SourcesDirectory)"

          - task: UsePythonVersion@0
            displayName: "Use Python 3.13"
            inputs:
              versionSpec: "3.13"
              addToPath: true

          - script: |
              pip install --no-cache-dir build twine
            displayName: "Install build tools"

          - script: |
              python -m build --wheel --sdist
            displayName: "Build wheel + sdist"

          # Verify the built package
          - script: |
              twine check dist/*
            displayName: "Validate package"

          # Upload to PyPI
          - script: |
              twine upload dist/* --non-interactive
            displayName: "Upload to PyPI"
            env:
              TWINE_USERNAME: "__token__"
              TWINE_PASSWORD: "$(PYPI_API_TOKEN)"

          # Publish as pipeline artifact
          - task: PublishPipelineArtifact@1
            displayName: "Publish wheel artifact"
            inputs:
              targetPath: "dist/"
              artifactName: "python-package-$(version)"

  # ===================================================================
  # Stage 5: Build Standalone Binary + Upload to Azure Blob (for Studio UI)
  # ===================================================================
  - stage: BinaryBuild
    displayName: "Build & Upload Binary"
    dependsOn: DockerBuildAndScan
    condition: succeeded()
    jobs:
      - job: BuildLinuxBinary
        displayName: "Build Linux amd64 Binary + Upload"
        steps:
          - task: DownloadPipelineArtifact@2
            displayName: "Download stamped source"
            inputs:
              artifactName: "stamped-source"
              targetPath: "$(Build.SourcesDirectory)"

          - task: UsePythonVersion@0
            displayName: "Use Python 3.13"
            inputs:
              versionSpec: "3.13"
              addToPath: true

          - script: |
              pip install -e ".[tools]"
              pip install pyinstaller
            displayName: "Install dependencies + PyInstaller"

          - script: |
              ./scripts/build-binary.sh --clean
            displayName: "Build binary"

          - script: |
              BINARY_NAME="qremotex-$(version)-linux-amd64"
              mv dist/qremotex "dist/$BINARY_NAME"
              # Generate SHA256 checksum for integrity verification
              sha256sum "dist/$BINARY_NAME" > "dist/$BINARY_NAME.sha256"
              cat "dist/$BINARY_NAME.sha256"
              echo "##vso[task.setvariable variable=binaryName]$BINARY_NAME"
            displayName: "Rename binary + generate checksum"

          # Upload binary + checksum to DMS Blob Storage (reuses existing dmsdata container)
          # Path convention: releases/qremotex/{version}/ (follows DMS path patterns)
          # This blob is served via the DMS service's download API
          # to the Studio UI "Remote Executors" download page.
          - task: AzureCLI@2
            displayName: "Upload binary to DMS blob (for Studio UI download)"
            inputs:
              azureSubscription: "qraptor-dev-acr-connection"
              scriptType: "bash"
              scriptLocation: "inlineScript"
              inlineScript: |
                BINARY_NAME="qremotex-$(version)-linux-amd64"
                BLOB_PREFIX="$(binaryBlobPrefix)"
                
                # Upload binary
                az storage blob upload \
                  --container-name $(binaryBlobContainer) \
                  --name "$BLOB_PREFIX/$(version)/$BINARY_NAME" \
                  --file "dist/$BINARY_NAME" \
                  --overwrite \
                  --connection-string "$(AZURE_STORAGE_CONNECTION_STRING)"
                
                # Upload checksum
                az storage blob upload \
                  --container-name $(binaryBlobContainer) \
                  --name "$BLOB_PREFIX/$(version)/$BINARY_NAME.sha256" \
                  --file "dist/$BINARY_NAME.sha256" \
                  --overwrite \
                  --connection-string "$(AZURE_STORAGE_CONNECTION_STRING)"
                
                # Upload as "latest" symlink (overwrite)
                az storage blob upload \
                  --container-name $(binaryBlobContainer) \
                  --name "$BLOB_PREFIX/latest/qremotex-linux-amd64" \
                  --file "dist/$BINARY_NAME" \
                  --overwrite \
                  --connection-string "$(AZURE_STORAGE_CONNECTION_STRING)"
                
                az storage blob upload \
                  --container-name $(binaryBlobContainer) \
                  --name "$BLOB_PREFIX/latest/qremotex-linux-amd64.sha256" \
                  --file "dist/$BINARY_NAME.sha256" \
                  --overwrite \
                  --connection-string "$(AZURE_STORAGE_CONNECTION_STRING)"
                
                echo "✅ Binary uploaded to: $(binaryBlobContainer)/$BLOB_PREFIX/$(version)/"
                echo "✅ Latest symlink updated: $(binaryBlobContainer)/$BLOB_PREFIX/latest/"

          # Also publish as pipeline artifact for traceability
          - task: PublishPipelineArtifact@1
            displayName: "Publish binary artifact"
            inputs:
              targetPath: "dist/"
              artifactName: "binary-linux-amd64-$(version)"

  # ===================================================================
  # Stage 6: Publish Helm Chart
  # ===================================================================
  - stage: HelmPublish
    displayName: "Publish Helm Chart"
    dependsOn: DockerBuildAndScan
    condition: succeeded()
    jobs:
      - job: PackageAndPush
        displayName: "Package & Push Helm Chart"
        steps:
          - task: DownloadPipelineArtifact@2
            displayName: "Download stamped source"
            inputs:
              artifactName: "stamped-source"
              targetPath: "$(Build.SourcesDirectory)"

          # Update chart version to match release
          - script: |
              sed -i "s/^version:.*/version: $(version)/" chart/Chart.yaml
              sed -i "s/^appVersion:.*/appVersion: \"$(version)\"/" chart/Chart.yaml
            displayName: "Update Chart.yaml version"

          - script: |
              helm package chart/ --version $(version) --app-version $(version)
            displayName: "Package Helm chart"

          # Push to Docker Hub OCI registry
          - script: |
              echo "$(DOCKERHUB_TOKEN)" | helm registry login registry-1.docker.io -u $(DOCKERHUB_USERNAME) --password-stdin
              helm push qremotex-$(version).tgz oci://registry-1.docker.io/qraptor
            displayName: "Push Helm chart to Docker Hub OCI"

          - task: PublishPipelineArtifact@1
            displayName: "Publish chart artifact"
            inputs:
              targetPath: "qremotex-$(version).tgz"
              artifactName: "helm-chart-$(version)"
```

### Task 5a: Trivy Vulnerability Scan Details ✅ DONE (included in release pipeline)

The Trivy scan is a **hard gate** in the release pipeline. It runs **after the Docker image is built but before any push to public registries**.

**Scan coverage:**

| Scan Type | What It Checks | Severity Gate |
|-----------|---------------|---------------|
| **OS packages** | Debian/Ubuntu CVEs in base image (`python:3.13-slim`) | CRITICAL, HIGH |
| **Python libraries** | CVEs in pip packages (aiohttp, pydantic, psutil, etc.) | CRITICAL, HIGH |
| **SBOM generation** | SPDX JSON for supply-chain compliance | N/A (informational) |

**Pipeline behavior:**

| Trivy Result | Pipeline Action |
|--------------|----------------|
| No CRITICAL/HIGH CVEs | ✅ Continue to Docker Hub push, PyPI publish, binary upload |
| CRITICAL or HIGH CVE found | ❌ **Pipeline STOPS** — nothing published to public |
| MEDIUM/LOW CVEs | ⚠️ Logged in report but does NOT block release |
| Unfixed CVEs | Ignored (`--ignore-unfixed`) — can't fix what's not patched upstream |

**Reports generated (published as pipeline artifacts):**
- `trivy-report-{version}.json` — Full vulnerability details (for audit/SOC2)
- `sbom-{version}.spdx.json` — Software Bill of Materials (for supply-chain compliance)

**Why Trivy?**
- Open source, widely adopted, fast (<30s scan time)
- Scans both OS packages AND application dependencies
- JSON + table output for CI and human review
- SBOM generation built-in
- Used by Docker Hub, AWS ECR, Azure Defender natively

---

### Task 6: PyPI Package Publishing ✅ DONE

**Pre-requisites (one-time setup):**

1. **Register `qremotex` on PyPI:**
   - Create PyPI account for qraptor (https://pypi.org/account/register/)
   - Reserve the package name by doing an initial upload
   - Create an API token scoped to the `qremotex` project
   - Store token in Azure DevOps variable group `qraptor-release-variables` as `PYPI_API_TOKEN` (secret)

2. **pyproject.toml updates needed:**

```toml
[project]
name = "qremotex"
version = "2.0.0"
description = "qRemoteX — Federated Remote Execution Agent for qRaptor"
readme = "README.md"
requires-python = ">=3.13"
license = { text = "Proprietary" }
authors = [
    { name = "qRaptor Engineering", email = "engineering@qraptor.ai" },
]
classifiers = [
    "Development Status :: 5 - Production/Stable",
    "Intended Audience :: Developers",
    "License :: Other/Proprietary License",
    "Programming Language :: Python :: 3.13",
    "Operating System :: OS Independent",
    "Topic :: Software Development :: Libraries",
]
keywords = ["qraptor", "remote-execution", "agent", "federated"]

[project.urls]
Homepage = "https://qraptor.ai"
Documentation = "https://docs.qraptor.ai/qremotex"
```

**Note on Proprietary + PyPI:** PyPI allows proprietary packages. Many commercial tools are on PyPI (e.g., `snowflake-connector-python`, `databricks-sdk`). The package is freely downloadable but the license doesn't grant redistribution rights.

---

### Task 7: Multi-Architecture Docker Build ✅ DONE (included in release pipeline)

**Why:** Customers run on diverse hardware — AWS Graviton (arm64), standard x86 servers (amd64), Mac Apple Silicon for dev (arm64).

**Requirements:**
- JNXT-DEV-POOL agent must have Docker Buildx with QEMU support
- One-time setup on the build agent:

```bash
# On the self-hosted build agent (one-time)
docker run --privileged --rm tonistiigi/binfmt --install all
```

**Verification:**
```bash
docker buildx ls
# Should show: linux/amd64, linux/arm64, linux/arm/v7, etc.
```

**If QEMU is not available on the self-hosted agent:** Use a Microsoft-hosted agent (`ubuntu-latest`) for the release pipeline only. Microsoft-hosted agents have Buildx and QEMU pre-installed.

---

### Task 8: Update Helm Chart for Public Image ✅ DONE

**Changes to `chart/values.yaml`:**

```yaml
image:
  repository: qraptor/qremotex    # Docker Hub (was ghcr.io/qraptor/qremotex)
  tag: "2.0.0"
  pullPolicy: IfNotPresent
```

**User install command:**
```bash
# From Docker Hub OCI
helm install qremotex oci://registry-1.docker.io/qraptor/qremotex \
  --version 2.0.0 \
  --set config.gatewayUrl=wss://exec-pRj7kM.qraptor.app/ws/executor \
  --set secrets.apiKey=qrx_prod_...
```

---

### Task 9: User Documentation — Extending qRemoteX ✅ DONE

**File:** `docs/EXTENDING_QREMOTEX.md` (new)

Content outline:

1. **Installing Additional Packages (Docker)**
   - Option A: Environment variable (`QRAPTOR_EXTRA_PACKAGES`)
   - Option B: Mount requirements file
   - Option C: Custom Dockerfile (recommended for production)
   - Option D: Pre-built custom image in CI/CD

2. **Installing Additional Packages (pip)**
   - Just `pip install` in the same virtualenv

3. **Installing Additional Packages (Kubernetes / Helm)**
   - Init container pattern
   - Custom image in values.yaml

4. **Common Packages for Agents**
   - Data: `pandas`, `numpy`, `polars`
   - ML/AI: `torch`, `transformers`, `openai`
   - Database: `asyncpg`, `sqlalchemy`, `pymongo`, `redis`
   - Cloud: `boto3`, `azure-storage-blob`, `google-cloud-storage`
   - HTTP: `httpx`, `requests`

5. **Troubleshooting**
   - Package install fails at startup (network issues, typos)
   - Package version conflicts
   - Binary packages on arm64

---

### Task 11: Studio UI — Remote Executors Install & Download Page ✅ DONE (Implemented tabbed install UI with Docker, pip, Binary, Helm, Custom Image tabs)

**Location in UI:** `studio.qraptor.ai` → Project → Settings → Remote Executors → "Setup New Executor" → Step 3 "Install & Connect"

The Studio UI's Remote Executors page already has a setup wizard (see architecture doc Section 15.3). We need to enhance **Step 3** to serve as the **central installation hub** showing ALL methods and serving the binary download directly.

#### Updated Step 3 Wireframe: Install & Connect

> **Note:** The gateway URL shown in the wireframes below is dynamically generated from `NEXT_PUBLIC_PRODUCTION_DOMAIN` env var.
> Production shows `exec-{shortId}.qraptor.app`, dev shows `exec-{shortId}.dev.qraptor.app`.

```
┌──────────────────────────────────────────────────────────────────────────────┐
│  Setup New Executor                                              Step 3 of 3│
│                                                                              │
│  ┌────────────────────────────────────────────────────────────────────────┐ │
│  │  ③ Install & Connect                                                   │ │
│  │                                                                        │ │
│  │  ⚠️ Copy your API key now — it won't be shown again!                  │ │
│  │                                                                        │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │ qrx_prod_a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6                    │ │ │
│  │  │                                                          [Copy] │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
│  │                                                                        │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │  Choose your installation method:                               │ │ │
│  │  │                                                                  │ │ │
│  │  │  ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌──────────┐ │ │ │
│  │  │  │ Docker  │ │  pip    │ │ Binary  │ │  Helm   │ │ Custom   │ │ │ │
│  │  │  │ 🐳     │ │ 🐍     │ │ ⬇️     │ │ ☸️     │ │ Image 🏗 │ │ │ │
│  │  │  └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ └─────┬────┘ │ │ │
│  │  │       ▼           │           │           │             │       │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
│  │                                                                        │ │
│  │  ── Docker (Recommended) ──────────────────────────────────────────── │ │
│  │                                                                        │ │
│  │  Pull the official image from Docker Hub and run:                     │ │
│  │                                                                        │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │ docker run -d \                                                 │ │ │
│  │  │   --name qremotex \                                             │ │ │
│  │  │   -e QRAPTOR_GATEWAY_URL=wss://exec-{shortId}.qraptor.app/\    │ │ │
│  │  │      ws/executor \                                              │ │ │
│  │  │   -e QRAPTOR_API_KEY=qrx_prod_a1b2c...o5p6 \                   │ │ │
│  │  │   -e CONCURRENCY=25 \                                          │ │ │
│  │  │   qraptor/qremotex:latest                                       │ │ │
│  │  │                                                          [Copy] │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
│  │                                                                        │ │
│  │  Need extra Python packages? Pass them via environment variable:      │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │   -e QRAPTOR_EXTRA_PACKAGES="pandas torch boto3"                │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
│  │  Or build a custom image: docs.qraptor.ai/executors/custom-image     │ │
│  │                                                                        │ │
│  └────────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────────┘
```

#### When user selects **"Binary"** tab:

```
│  │  ── Binary Download (Linux x86_64) ────────────────────────────────── │ │
│  │                                                                        │ │
│  │  Self-contained executable for Linux amd64 servers. No Python          │ │
│  │  or Docker required. Ideal for air-gapped or minimal environments.    │ │
│  │                                                                        │ │
│  │  ┌─────────────────────────────────────────────────────────────────┐  │ │
│  │  │  qremotex v2.0.42 — linux/amd64                                │  │ │
│  │  │  Size: 52 MB │ Released: 2026-03-12                            │  │ │
│  │  │  SHA256: a1b2c3d4e5f6...                                       │  │ │
│  │  │                                                                 │  │ │
│  │  │              [ ⬇ Download Binary ]  [ Copy SHA256 ]             │  │ │
│  │  └─────────────────────────────────────────────────────────────────┘  │ │
│  │                                                                        │ │
│  │  After download:                                                      │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │ chmod +x qremotex-2.0.42-linux-amd64                           │ │ │
│  │  │ ./qremotex-2.0.42-linux-amd64                                   │ │ │
│  │  │                                                          [Copy] │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
│  │                                                                        │ │
│  │  ⚠️ The standalone binary is for Linux x86_64 only.                   │ │
│  │     For macOS, ARM64, or other platforms, use Docker or pip install.  │ │
│  │                                                                        │ │
│  │  Verify integrity:                                                    │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │ sha256sum -c qremotex-2.0.42-linux-amd64.sha256                │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
```

#### When user selects **"pip"** tab:

```
│  │  ── Python (pip install) ──────────────────────────────────────────── │ │
│  │                                                                        │ │
│  │  Install from PyPI and run. Requires Python 3.13+.                    │ │
│  │                                                                        │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │ pip install qremotex                                            │ │ │
│  │  │                                                                  │ │ │
│  │  │ export QRAPTOR_GATEWAY_URL=wss://exec-{shortId}.qraptor.app/\   │ │ │
│  │  │   ws/executor                                                    │ │ │
│  │  │ export QRAPTOR_API_KEY=qrx_prod_a1b2c...o5p6                    │ │ │
│  │  │ qremotex                                                         │ │ │
│  │  │                                                          [Copy] │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
│  │                                                                        │ │
│  │  💡 You can install any extra pip packages in the same environment:   │ │
│  │     pip install pandas torch boto3                                    │ │
```

#### When user selects **"Helm"** tab:

```
│  │  ── Kubernetes (Helm) ─────────────────────────────────────────────── │ │
│  │                                                                        │ │
│  │  Deploy as a Kubernetes pod using our official Helm chart:             │ │
│  │                                                                        │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │ # Create secret for API key                                     │ │ │
│  │  │ kubectl create secret generic qremotex-api-key \                │ │ │
│  │  │   --from-literal=api-key=qrx_prod_a1b2c...o5p6                 │ │ │
│  │  │                                                                  │ │ │
│  │  │ # Install chart                                                  │ │ │
│  │  │ helm install qremotex \                                         │ │ │
│  │  │   oci://registry-1.docker.io/qraptor/qremotex \                 │ │ │
│  │  │   --set config.gatewayUrl=wss://exec-{shortId}.qraptor.app/\    │ │ │
│  │  │     ws/executor \                                                │ │ │
│  │  │   --set secrets.existingSecret=qremotex-api-key                 │ │ │
│  │  │                                                          [Copy] │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
```

#### When user selects **"Custom Image"** tab:

```
│  │  ── Custom Docker Image (with your packages) ─────────────────────── │ │
│  │                                                                        │ │
│  │  For production: extend our base image with your own packages.        │ │
│  │                                                                        │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │ # Dockerfile                                                     │ │ │
│  │  │ FROM qraptor/qremotex:latest                                    │ │ │
│  │  │ RUN pip install --no-cache-dir pandas torch boto3                │ │ │
│  │  │                                                          [Copy] │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
│  │                                                                        │ │
│  │  Then build and run your custom image:                                │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │ docker build -t my-qremotex .                                    │ │ │
│  │  │ docker run -d \                                                 │ │ │
│  │  │   -e QRAPTOR_GATEWAY_URL=wss://exec-{shortId}.qraptor.app/\    │ │ │
│  │  │      ws/executor \                                              │ │ │
│  │  │   -e QRAPTOR_API_KEY=qrx_prod_a1b2c...o5p6 \                   │ │ │
│  │  │   my-qremotex                                                    │ │ │
│  │  │                                                          [Copy] │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
```

#### Binary Download — Backend Implementation

**How the binary is served to Studio UI:**

```
┌─────────────────┐    ┌──────────────────────┐    ┌────────────────────────┐
│  Release Pipeline │   │  DMS Azure Blob       │   │  Studio UI             │
│  (Azure DevOps)   │   │  (dmsdata container)  │   │  (Remote Executors)    │
├─────────────────┤   ├──────────────────────┤   ├────────────────────────┤
│ Build binary     │──▶│ releases/qremotex/   │   │                        │
│ Upload to blob   │   │  2.0.42/             │   │                        │
│  (reuses DMS     │   │   qremotex-2.0.42-   │◀──│ GET /api/v1/executors/ │
│   blob storage)  │   │     linux-amd64      │   │   binary/download      │
│                  │   │   qremotex-2.0.42-   │   │                        │
│                  │   │     linux-amd64.sha256│   │ → DMS generates signed │
│                  │   │  latest/             │   │   URL → redirect       │
│                  │   │   qremotex-linux-amd64│   │                        │
│                  │   │   qremotex-linux-     │   │ UI shows:              │
│                  │   │     amd64.sha256      │   │ - Version number       │
│                  │   └──────────────────────┘   │ - File size            │
│                  │                               │ - SHA256               │
│                  │                               │ - Download button      │
└─────────────────┘                               └────────────────────────┘
```

**Why reuse the DMS blob storage?**
- The `dmsdata` container already exists and is managed by qraptor-dms-service
- Same `AZURE_STORAGE_CONNECTION_STRING` — no new storage account or secrets needed
- DMS already has file download/SAS-URL generation logic we can extend
- Path convention `releases/qremotex/` follows DMS's existing pattern (`{category}/{scope}/...`)
- One less infrastructure resource to provision and manage

**API endpoints (Platform Admin or DMS service):**

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/executors/binary/latest` | GET | Returns metadata (version, size, sha256, download URL) |
| `/api/v1/executors/binary/download` | GET | Redirects to signed DMS blob URL (time-limited SAS token) |
| `/api/v1/executors/binary/versions` | GET | Lists all available binary versions |

**Security:**
- Download requires authenticated session (logged-in to Studio)
- Uses time-limited DMS blob SAS token (expires in 15 minutes, same mechanism as DMS document downloads)
- SHA256 checksum displayed for integrity verification
- Binary is Linux x86_64 only — UI clearly states this and suggests Docker/pip for other platforms

---

### Task 10: Update Architecture Doc Section 17 ✅ DONE

Update Section 17 of `QREMOTEX_ARCHITECTURE_AND_IMPLEMENTATION_PLAN.md` to reflect:
- Docker Hub as the public registry (not GHCR)
- PyPI for pip install
- Python 3.13 (not 3.11)
- The entrypoint script for user packages
- Azure DevOps pipelines (not GitHub Actions)
- Multi-arch image support

---

## 5. Azure DevOps Pipeline Architecture

### Service Connections Required

| Connection Name | Type | Purpose | Exists? |
|----------------|------|---------|---------|
| `qraptor-dev-acr-connection` | Azure Container Registry | Push to private ACR | ✅ Yes |
| `qraptor-dev-aks-connection` | Kubernetes | Deploy to AKS | ✅ Yes (not used for qRemoteX) |
| `qraptor-dockerhub-connection` | Docker Registry | Push to Docker Hub | ❌ **New** |

### Variable Groups Required

| Variable Group | Purpose | Exists? |
|---------------|---------|---------|
| `qraptor-dev-variables` | Dev pipeline shared vars | ✅ Yes |
| `qraptor-release-variables` | Release secrets (Docker Hub + PyPI tokens) | ❌ **New** |

### Variables in `qraptor-release-variables` (New)

| Variable | Type | Description |
|----------|------|-------------|
| `DOCKERHUB_USERNAME` | Plain | Docker Hub organization username (`qraptor`) |
| `DOCKERHUB_TOKEN` | **Secret** | Docker Hub access token (scope: Read/Write to `qraptor/qremotex`) |
| `PYPI_API_TOKEN` | **Secret** | PyPI API token (scope: `qremotex` project only) |
| `AZURE_STORAGE_CONNECTION_STRING` | **Secret** | Reuses existing DMS blob connection (same `dmsdata` container, `releases/qremotex/` prefix) |

### Pipeline Setup Steps (Azure DevOps)

**One-time setup in Azure DevOps → Project Settings:**

1. **Create Docker Hub service connection:**
   - Project Settings → Service Connections → New → Docker Registry
   - Registry type: Docker Hub
   - Docker ID: `qraptor`
   - Docker Password: Access token (not password)
   - Service connection name: `qraptor-dockerhub-connection`

2. **Create variable group `qraptor-release-variables`:**
   - Pipelines → Library → + Variable Group
   - Add `DOCKERHUB_USERNAME`, `DOCKERHUB_TOKEN` (lock as secret), `PYPI_API_TOKEN` (lock as secret)

3. **Create dev pipeline:**
   - Pipelines → New Pipeline → Azure Repos Git → Select repo
   - YAML file path: `PROJECTS/qRemoteX/azure-pipelines/dev-pipeline.yml`
   - Name: `qremotex-dev`

4. **Create release pipeline:**
   - Pipelines → New Pipeline → Azure Repos Git → Select repo
   - YAML file path: `PROJECTS/qRemoteX/azure-pipelines/release-pipeline.yml`
   - Name: `qremotex-release`
   - Trigger: **Automatic on push to `2.0.x` branch** (CI trigger in YAML)

### Pipeline Flow Diagram

```
Developer pushes to 1.0.x-* branch
          │
          ▼
┌─────────────────────────┐
│  Dev Pipeline            │
│  (automatic trigger)     │
├─────────────────────────┤
│  1. Ruff lint            │
│  2. pytest               │
│  3. Docker build         │
│  4. Push to ACR          │
│     → qraptordevregistry │
│       .azurecr.io/       │
│       qremotex:dev-123   │
└─────────────────────────┘

QA validates → Merge PR to 2.0.x branch
          │
          ▼
┌────────────────────────────────────────────────────────────┐
│  Release Pipeline (auto-trigger on push to 2.0.x)          │
│  Version: 2.0.{BuildId} (auto-incremented)                 │
├────────────────────────────────────────────────────────────┤
│                                                            │
│  ┌──────────────┐                                          │
│  │ 1. Validate   │ ─── Stamp version, lint, pytest         │
│  └──────┬───────┘                                          │
│         ▼                                                  │
│  ┌──────────────────┐                                      │
│  │ 2. Build + Trivy │ ─── Build image → Trivy CRIT/HIGH   │
│  │    Security Scan  │     gate → SBOM → JSON report       │
│  └──────┬───────────┘                                      │
│         │ (all stages below depend on Trivy passing)       │
│    ┌────┴─────┬──────────┬──────────┐                      │
│    ▼          ▼          ▼          ▼                       │
│  ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐  (parallel stages)   │
│  │Docker│ │ PyPI │ │Binary│ │ Helm │                       │
│  │ Hub  │ │      │ │Upload│ │      │                       │
│  └──┬───┘ └──┬───┘ └──┬───┘ └──┬───┘                      │
│     │        │        │        │                           │
│     ▼        ▼        ▼        ▼                           │
│  qraptor/  PyPI     DMS blob Docker Hub                    │
│  qremotex  qremotex (dmsdata OCI chart                     │
│  :2.0.N    ==2.0.N  container)qremotex                     │
│  (amd64+            Studio   (tgz)                         │
│   arm64)            download                               │
└────────────────────────────────────────────────────────────┘
```

---

## 6. User Package Installation Patterns

### Pattern A: Docker — Environment Variable (Quick & Dirty)

```bash
docker run -d \
  -e QRAPTOR_EXTRA_PACKAGES="pandas torch boto3" \
  -e QRAPTOR_GATEWAY_URL=wss://exec-pRj7kM.qraptor.app/ws/executor \
  -e QRAPTOR_API_KEY=qrx_prod_... \
  qraptor/qremotex:latest
```

**Pros:** Zero files, zero Docker knowledge  
**Cons:** Slower startup (installs on every container start), no version pinning, not reproducible

### Pattern B: Docker — Mounted Requirements File

```bash
# requirements.txt on the host
echo "pandas==2.2.0
torch==2.3.0
boto3>=1.34" > /opt/qremotex/extra-requirements.txt

docker run -d \
  -v /opt/qremotex/extra-requirements.txt:/app/extra-requirements.txt:ro \
  -e QRAPTOR_GATEWAY_URL=wss://exec-pRj7kM.qraptor.app/ws/executor \
  -e QRAPTOR_API_KEY=qrx_prod_... \
  qraptor/qremotex:latest
```

**Pros:** Version pinning, reproducible, easy to update  
**Cons:** Slower startup (installs on every container start)

### Pattern C: Docker — Custom Image (Recommended for Production)

```dockerfile
FROM qraptor/qremotex:latest

# Install your packages
RUN pip install --no-cache-dir \
    pandas==2.2.0 \
    torch==2.3.0 \
    boto3>=1.34 \
    my-internal-lib==0.5.0

# Optionally copy custom config
# COPY my-labels.json /app/config/labels.json
```

```bash
docker build -t my-company/qremotex-custom:latest .
docker run -d \
  -e QRAPTOR_GATEWAY_URL=wss://exec-pRj7kM.qraptor.app/ws/executor \
  -e QRAPTOR_API_KEY=qrx_prod_... \
  my-company/qremotex-custom:latest
```

**Pros:** Fast startup, reproducible, version-pinned, CI/CD friendly  
**Cons:** Requires Docker knowledge, must rebuild on qRemoteX upgrade

### Pattern D: pip Install — Native Python

```bash
# Create a venv (recommended)
python3.13 -m venv qremotex-env
source qremotex-env/bin/activate

# Install qRemoteX + your packages
pip install qremotex pandas torch boto3

# Run
export QRAPTOR_GATEWAY_URL=wss://exec-pRj7kM.qraptor.app/ws/executor
export QRAPTOR_API_KEY=qrx_prod_...
qremotex
```

**Pros:** Most flexible, familiar Python workflow, instant access to all packages  
**Cons:** No container isolation, user manages Python environment

### Pattern E: Kubernetes — Helm with Custom Image

```yaml
# custom-values.yaml
image:
  repository: my-company/qremotex-custom
  tag: "2.0.0"

config:
  gatewayUrl: "wss://exec-pRj7kM.qraptor.app/ws/executor"

secrets:
  existingSecret: "qremotex-api-key"
```

```bash
helm install qremotex oci://registry-1.docker.io/qraptor/qremotex \
  -f custom-values.yaml
```

---

## 7. Security Considerations

### Trivy Vulnerability Scanning (Mandatory)

Every release is gated by a full Trivy scan. See [Task 5a](#task-5a-trivy-vulnerability-scan-details) for details.

- **Scan runs before any public push** — no vulnerable images reach Docker Hub or PyPI
- **CRITICAL + HIGH CVEs block the release** — pipeline fails fast
- **SBOM generated automatically** — SPDX JSON for supply-chain compliance (SOC2, HIPAA)
- **JSON report saved** — audit artifact for every release

### Image Signing (Future Enhancement)

For enterprise customers requiring supply-chain verification:
- Sign images with Cosign (Sigstore)
- Publish signatures alongside images
- Users can verify: `cosign verify qraptor/qremotex:2.0.0`

### PyPI Package Signing

Use Trusted Publishers (OIDC) for PyPI uploads when moving to GitHub Actions in the future. For Azure DevOps, API token auth is the standard approach.

### Entrypoint Script Security

The entrypoint script runs `pip install` as the `qremotex` user (non-root). The `--user` flag ensures packages install to `~/.local` not system-wide. This is by design — the user is opting in to installing packages in their own executor.

### Binary Download Security

- Binary download endpoint requires authenticated Studio session
- Azure Blob SAS tokens are time-limited (15 min expiry, same as DMS document downloads)
- SHA256 checksum published alongside every binary
- Only Linux x86_64 binary is built (PyInstaller limitation) — other platforms use Docker or pip

---

## 8. Verification Checklist

### After Task 1 (Python 3.13)
- [x] `docker build` succeeds with `python:3.13-slim`
- [x] `ruff check` passes with `target-version = "py313"`
- [x] All tests pass on Python 3.13
- [x] `python --version` inside container shows 3.13.x

### After Task 2 (Dockerfile Fix)
- [x] No `src/` directory exists in runtime image
- [x] `python -c "import qremotex; print(qremotex.__version__)"` works inside container
- [x] `qremotex` CLI entry point works: `docker run --rm qraptor/qremotex --help`
- [x] Health check passes: `docker inspect --format='{{.State.Health.Status}}' <container>`

### After Task 3 (Entrypoint Script)
- [x] Container starts normally without extras
- [x] `QRAPTOR_EXTRA_PACKAGES="requests"` installs requests on startup
- [x] Mounted `extra-requirements.txt` installs packages on startup
- [x] Custom `FROM qraptor/qremotex:latest` image extends correctly

### After Task 4 (Dev Pipeline)
- [x] Pipeline triggers on push to `1.0.x-*`
- [x] Lint + test stage passes
- [x] Image appears in ACR: `qraptordevregistry.azurecr.io/qremotex:dev-{buildId}`

### After Task 5 (Release Pipeline)
- [x] Pipeline triggers automatically on push to `2.0.x` branch
- [x] Version auto-stamps as `2.0.{BuildId}` in pyproject.toml and __init__.py
- [x] Trivy scan passes (no CRITICAL/HIGH CVEs)
- [x] Trivy report + SBOM published as pipeline artifacts
- [x] Pipeline STOPS if Trivy finds CRITICAL/HIGH (verify with intentionally vulnerable dep)
- [x] Image appears on Docker Hub: `qraptor/qremotex:2.0.{N}`
- [x] `docker pull qraptor/qremotex:2.0.{N}` works from external machine
- [x] Both `linux/amd64` and `linux/arm64` manifests present
- [x] Binary uploaded to DMS blob: `dmsdata/releases/qremotex/2.0.{N}/`
- [x] Binary `latest/` symlink updated: `dmsdata/releases/qremotex/latest/`

### After Task 6 (PyPI)
- [x] `pip install qremotex=={version}` works from any machine
- [x] `qremotex` CLI command available after install
- [x] Package page visible at `https://pypi.org/project/qremotex/`

### After Task 7 (Multi-Arch)
- [x] `docker manifest inspect qraptor/qremotex:{version}` shows both architectures
- [x] Image runs on Apple Silicon Mac (arm64)
- [x] Image runs on standard x86 Linux (amd64)

### After Task 8 (Helm Chart)
- [x] `helm pull oci://registry-1.docker.io/qraptor/qremotex --version {version}` works
- [x] `helm install` with custom values deploys correctly
- [x] Default `values.yaml` points to Docker Hub

---

## Appendix: File Tree After Implementation

```
PROJECTS/qRemoteX/
├── azure-pipelines/
│   ├── dev-pipeline.yml              ← NEW: Dev pipeline (ACR)
│   └── release-pipeline.yml          ← NEW: Release pipeline (Docker Hub + PyPI)
├── chart/
│   ├── Chart.yaml                    ← UPDATED: version management
│   ├── values.yaml                   ← UPDATED: Docker Hub image ref
│   └── templates/
│       ├── deployment.yaml
│       ├── secret.yaml
│       ├── serviceaccount.yaml
│       └── _helpers.tpl
├── docs/
│   ├── QREMOTEX_ARCHITECTURE_AND_IMPLEMENTATION_PLAN.md  ← UPDATED: Section 17
│   ├── QREMOTEX_PUBLISH_IMPLEMENTATION_PLAN.md           ← THIS DOCUMENT
│   └── EXTENDING_QREMOTEX.md                            ← NEW: User extension guide
├── scripts/
│   ├── build-binary.sh               ← Existing
│   └── docker-entrypoint.sh          ← NEW: Entrypoint with user package support
├── azure-pipelines/
│   ├── dev-pipeline.yml              ← NEW: Dev pipeline (1.0.x-* branch → ACR)
│   └── release-pipeline.yml          ← NEW: Release pipeline (2.0.x branch → public)
├── src/
│   └── qremotex/                     ← Existing (no changes to source code)
├── tests/                            ← Existing
├── Dockerfile                        ← UPDATED: No source in runtime, Python 3.13
├── docker-compose.yml                ← Existing
├── pyproject.toml                    ← UPDATED: Python 3.13, classifiers, URLs
├── qremotex.spec                     ← Existing
├── requirements.txt                  ← Existing
└── README.md                         ← UPDATED: Public install instructions
```

---

## Appendix: Account Registration Checklist (One-Time)

| Registry | Account | Namespace | Action |
|----------|---------|-----------|--------|
| Docker Hub | `qraptor` org | `qraptor/qremotex` | Create org → Create repo → Generate access token → Store in Azure DevOps |
| PyPI | `qraptor` | `qremotex` | Register account → Create API token (scoped to project) → Store in Azure DevOps |

**Docker Hub setup:**
1. https://hub.docker.com/ → Sign up → Create organization `qraptor`
2. Create repository `qremotex` (public)
3. Account Settings → Security → Access Tokens → New → Read/Write → Copy token
4. Store in Azure DevOps: `qraptor-release-variables` → `DOCKERHUB_TOKEN`

**PyPI setup:**
1. https://pypi.org/account/register/ → Create account
2. First upload claims the `qremotex` namespace
3. Account Settings → API tokens → Add API token → Scope: `qremotex` → Copy
4. Store in Azure DevOps: `qraptor-release-variables` → `PYPI_API_TOKEN`

---

## Appendix: Post-Review Fixes (v2.1)

Issues found during implementation review and fixed:

| # | File | Issue | Fix |
|---|------|-------|-----|
| 1 | `.dockerignore` | Only excluded cache/build files — `chart/`, `docs/`, `azure-pipelines/`, `tests/`, `.git/` leaked into Docker build context | Added comprehensive exclusions for non-build files |
| 2 | `chart/templates/deployment.yaml` | No `initContainers` support — init container pattern in EXTENDING_QREMOTEX.md wouldn't work | Added `initContainers` block from `values.yaml` |
| 3 | `chart/templates/deployment.yaml` | `readOnlyRootFilesystem: true` but no writable mount for `pip install --user` target (`/app/.local`) — `QRAPTOR_EXTRA_PACKAGES` would fail in K8s | Added `user-packages` emptyDir volume mounted at `/app/.local` |
| 4 | `docs/EXTENDING_QREMOTEX.md` | Init container pattern mounted `/home/qremotex/.local` but Dockerfile sets qremotex home dir to `/app` | Fixed mount path to `/app/.local` |
| 5 | `chart/values.yaml` | No `config.extraPackages` or `initContainers` defaults | Added `extraPackages` config field and `initContainers: []` default |
| 6 | `azure-pipelines/release-pipeline.yml` | Security reports artifact published `targetPath: "."` (entire directory) | Collect only report files into `security-reports/` directory before publish |
| 7 | `docker-compose.yml` | Used `QRAPTOR_CONCURRENCY`, `QRAPTOR_LOG_LEVEL`, `QRAPTOR_LABELS` — but `config.py` expects `CONCURRENCY`, `LOG_LEVEL`, `LABELS` | Fixed to match actual env var names |
| 8 | `src/qremotex/config.py` | `self.version = "1.0.0"` — stale version string | Updated to `"2.0.0"` |
