# HLA-Compass Module Development — Complete Reference

> This document is the single source of truth for building modules on the
> HLA-Compass platform. An LLM or developer reading only this file should be
> able to construct a correct, production-ready module.

## Guiding Principles

| Priority | Principle |
|----------|-----------|
| **P0** | **UI / UX / DX** — Developer and user experience comes first. |
| **P1** | **Security** — Every data path is scoped, signed, and auditable. |
| **P2** | **Performance** — Efficient by default; never block needlessly. |
| **P3** | **Elegance / Maintainability** — Clean, readable, minimal surface area. |

**GUIDANCE:** No workarounds, no placeholders, no speculative abstractions, no
over-engineering. If you do not know, ask.

---

## Quick Start

```bash
# 1. Install the SDK
pip install hla-compass

# 2. Authenticate
hla-compass auth login

# 3. Scaffold a new module
hla-compass init my-module            # no-ui (default)
hla-compass init my-module --template ui  # with-ui (React frontend)

# 4. Develop locally
cd my-module
hla-compass dev                       # runs execute() with examples/sample_input.json

# 5. Validate
hla-compass validate                  # 7-layer manifest + code validation

# 6. Publish
hla-compass keys init                 # one-time: generate signing key pair
hla-compass publish --env dev --scope org
```

---

## Module Contract

### Base Class

Every module extends `hla_compass.Module` and implements `execute()`:

```python
from __future__ import annotations

from typing import Any, Dict

from pydantic import BaseModel, Field
from hla_compass import Module


class Input(BaseModel):
    """Strongly-typed input schema. Validated automatically before execute()."""
    sequence: str = Field(..., description="Peptide sequence to analyse")
    allele: str = Field("HLA-A*02:01", description="Target HLA allele")


class MyModule(Module):
    Input = Input  # Binds the Pydantic model for auto-validation

    def execute(self, input_data: Input, context: Dict[str, Any]) -> Dict[str, Any]:
        # --- your logic here ---
        peptides = self.api.get_peptides(
            filters={"sequence": input_data.sequence}, limit=10
        )

        return self.success(
            results={"peptides": peptides, "allele": input_data.allele},
            summary={"found": len(peptides)},
        )
```

### Lifecycle

```
Module.__init__(manifest_path)
    │
    ▼
Module.run(input_data, context)
    ├── validate_inputs(input_data)   # Pydantic or JSON Schema
    ├── _initialize_helpers(context)  # sets up self.api, self.data, self.storage, self.db
    └── execute(input_data, context)  # YOUR CODE
         └── return self.success(results=..., summary=...)
              or raise self.error("message", details={...})
```

### Key Methods

| Method | Purpose |
|--------|---------|
| `execute(input_data, context)` | **Abstract.** Your module logic. Must return `self.success(...)`. |
| `validate_inputs(input_data)` | Auto-called. Uses `Input` Pydantic model or manifest JSON Schema. |
| `success(results, summary=None)` | Return structured success output. |
| `error(message, details=None)` | Raise `ModuleError` with optional detail dict. |
| `bootstrap()` | Inspect available helpers (data, storage, db). |
| `check()` | Override to verify prerequisites. |
| `sync_manifest(path=None)` | Write Pydantic `Input` schema into `manifest.json`. |

### Properties Available Inside `execute()`

| Property | Type | Description |
|----------|------|-------------|
| `self.api` | `APIClient` | REST API client (see below). |
| `self.data` | `DataClient` | SQL + storage access to the Data Catalog. |
| `self.storage` | `Storage` | File I/O for module run artifacts. |
| `self.db` | `ScientificQuery` | Direct database access via RDS Data API. |
| `self.context` | `RuntimeContext` | Typed execution context. |
| `self.run_id` | `str` | Current run UUID. |
| `self.module_id` | `str` | Module identifier. |
| `self.module_version` | `str` | Semantic version string. |
| `self.organization_id` | `str` | Tenant org id. |
| `self.user_id` | `str` | Initiating user. |
| `self.environment` | `str` | `dev` / `staging` / `prod`. |
| `self.credit` | `CreditReservation \| None` | Credit reservation metadata. |
| `self.logger` | `Logger` | Pre-configured logger with run context. |

---

## Available APIs Inside Modules

### `self.api` — APIClient

The REST API client for the HLA-Compass platform.

#### Entity Queries

```python
# Peptides
peptides = self.api.get_peptides(filters={"sequence": "SIINFEKL"}, limit=100, offset=0)
peptide  = self.api.get_peptide("SIINFEKL")        # by sequence or numeric ID
peptide  = self.api.get_peptide("42")               # by numeric ID

# Proteins
proteins = self.api.get_proteins(filters={"accession": "P12345"}, limit=100, offset=0)
protein  = self.api.get_protein("P12345")           # by accession or numeric ID

# Samples
samples  = self.api.get_samples(filters={"tissue": "lung"}, limit=100, offset=0)
sample   = self.api.get_sample("1")                 # by numeric ID

# HLA Alleles (access via SQL — see Data Access Patterns)
# self.api.get_hla_alleles() now works through scientific-query capable auth paths
# (module DB access or API-key scientific-query access). It is not a plain JWT-only
# REST endpoint, so self.data.sql.query() remains the explicit fallback for custom SQL.
```

#### Pagination Iterators

```python
# Yields all results across pages automatically
for peptide in self.api.iter_peptides(filters={"length_min": 8}, page_size=100, max_results=1000):
    process(peptide)

for protein in self.api.iter_proteins(page_size=50):
    process(protein)

for sample in self.api.iter_samples(page_size=50):
    process(sample)
```

#### Relationship Helpers

```python
# Peptide relationships
samples_for_peptide  = self.api.get_peptide_samples("42", limit=100)
proteins_for_peptide = self.api.get_peptide_proteins("42", limit=100)

# Protein relationships
peptides_for_protein = self.api.get_protein_peptides("99", limit=100)

# Sample relationships
peptides_in_sample   = self.api.get_sample_peptides("7", limit=100)
```

#### Module Operations

```python
# List / get modules
modules = self.api.list_modules(category="analysis", limit=100)
module  = self.api.get_module("module-uuid")

# Start a module run
run = self.api.start_module_run(
    "module-uuid",
    parameters={"sequence": "SIINFEKL"},
    mode="interactive",         # or "workflow"
    compute_profile="standard", # optional
    version="1.2.0",           # optional
)

# Monitor
status = self.api.get_module_run(run["runId"])
logs   = self.api.get_module_run_logs(run["runId"], limit=50)
result = self.api.get_module_run_result(run["runId"])

# Cancel
self.api.cancel_module_run(run["runId"])

# Heartbeat (keep alive during long runs)
self.api.heartbeat_module_run(run["runId"], refresh_urls=True)

# Complete a UI-only run
self.api.complete_module_run(run["runId"], status="completed", summary={"rows": 42})

# List runs
all_runs = self.api.list_module_runs(module_id="mod-id", status="completed", page=1, limit=20)
```

#### Convenience Facades

```python
# ModulesFacade — self.api.modules
self.api.modules.list(category="analysis")
self.api.modules.get("module-uuid")
self.api.modules.run("module-uuid", params={"sequence": "SIINFEKL"})

# RunsFacade — self.api.runs
self.api.runs.list(module_id="mod-id", status="running")
self.api.runs.get("run-uuid")
self.api.runs.logs("run-uuid", limit=100)
self.api.runs.results("run-uuid")
self.api.runs.cancel("run-uuid")
self.api.runs.heartbeat("run-uuid")
self.api.runs.complete("run-uuid", status="completed", summary={...})
self.api.runs.revoke("run-uuid", reason="admin cleanup")
```

#### Dataset Versioning & Lineage

```python
# DatasetVersionsFacade — self.api.dataset_versions
version = self.api.dataset_versions.create(
    name="peptide-cohort-2024-q1",
    sources=[
        {"source_table": "peptides", "source_schema": "scientific", "filter_json": {"length_min": 8}},
    ],
    description="Q1 peptide analysis cohort",
    label="v1.0",
    total_row_count=50000,
)

versions = self.api.dataset_versions.list(status="active", limit=20, page=1)
single   = self.api.dataset_versions.get("version-uuid")
self.api.dataset_versions.archive("version-uuid")

# Lineage queries
lineage     = self.api.dataset_versions.get_run_lineage("run-uuid")
downstream  = self.api.dataset_versions.get_downstream_runs("version-uuid", limit=20)
artifact    = self.api.dataset_versions.get_artifact_lineage("s3://bucket/org/runs/run-id/outputs/result.parquet")
```

#### Publishing

```python
# Register container module via intake pipeline
result = self.api.register_container_module(payload={...}, idempotency_key="unique-key")
config = self.api.get_publish_config()
status = self.api.get_module_publish_status("publish-job-uuid")
```

---

### `self.data` — DataClient

Scoped access to the Data Catalog (SQL + Object Storage).

#### SQL Queries

```python
# Parameterized SQL with Row-Level Security
result = self.data.sql.query(
    "SELECT sequence, mass FROM peptides WHERE length >= %s LIMIT %s",
    params=[8, 100],
)
# Returns: {"columns": [...], "data": [...], "count": N}

# HLA allele queries (use SQL since REST endpoint is not deployed)
alleles = self.data.sql.query(
    "SELECT allele_name, locus FROM hlaalleles WHERE locus = %s",
    params=["A"],
)
```

#### Object Storage

```python
# Get temporary AWS STS credentials scoped to the catalog bucket
creds = self.data.storage.get_credentials(mode="read")  # or "write"
# Returns: {"accessKeyId", "secretAccessKey", "sessionToken", "region", "bucket", "prefix"}

# Get a pre-configured boto3 S3 client
s3 = self.data.storage.get_s3_client(mode="read")

# Read parquet files directly
df = self.data.storage.read_parquet("runs/123.parquet", engine="polars")  # or "pandas"
```

---

### `self.storage` — Storage

File I/O for module run artifacts. All files are scoped under `files/{run_id}/`.

#### Save Methods

```python
# JSON
url = self.storage.save_json("results.json", {"peptides": [...]})
url = self.storage.save_json("results.json.gz", data, compress=True)

# CSV (pandas DataFrame)
url = self.storage.save_csv("output.csv", dataframe, index=False)
url = self.storage.save_csv("output.csv.gz", dataframe, compress=True)

# Excel (requires hla-compass[data])
url = self.storage.save_excel("report.xlsx", dataframe)
url = self.storage.save_excel("multi.xlsx", {"Sheet1": df1, "Sheet2": df2})

# Plain text
url = self.storage.save_text("log.txt", "Processing complete.")

# HTML
url = self.storage.save_html("report.html", "<h1>Results</h1>...")

# Matplotlib figure
url = self.storage.save_figure("plot.png", fig, format="png", dpi=150)

# Raw file (bytes, string, or file-like)
url = self.storage.save_file("data.bin", raw_bytes, content_type="application/octet-stream")

# Atomic multi-file save (rollback on failure)
urls = self.storage.save_files_atomic([
    {"filename": "a.json", "content": '{"ok": true}', "content_type": "application/json"},
    {"filename": "b.csv", "content": csv_string},
])

# Workflow-specific artifacts
url = self.storage.save_workflow_file(
    "step_output.json", json_str,
    workflow_id="wf-1", workflow_run_id="wr-1", step_id="step-1",
)
```

#### Read Methods

```python
raw_bytes = self.storage.load("results.json")
data      = self.storage.load_json("results.json")
text      = self.storage.load_text("log.txt")
files     = self.storage.list_files(prefix="results/")
url       = self.storage.create_download_url("results.json", expires_in=3600)
```

---

### `self.db` — ScientificQuery

Direct database access via AWS RDS Data API with Row-Level Security.
Only available when `DB_CLUSTER_ARN` and `DB_APP_SECRET_ARN` environment variables are set
(automatically configured in Fargate/Batch execution).

```python
# Execute predefined PostgreSQL functions
results = self.db.execute_function(
    "search_peptides_by_sequence",
    {"pattern": "SIINFEKL", "max_results": 100},
)

results = self.db.execute_function(
    "search_peptides_by_hla",
    {"allele_name": "HLA-A*02:01", "min_intensity": 0.5, "max_results": 50},
)
```

**Available functions** (defined in `scientific` schema):
- `search_peptides_by_sequence(pattern, max_results)` — search peptides by sequence pattern
- `search_peptides_by_hla(allele_name, min_intensity, max_results)` — find peptides by HLA allele

---

### `self.context` — RuntimeContext

Typed, immutable execution context. Implements `Mapping[str, Any]` for backward compatibility.

| Field | Type | Description |
|-------|------|-------------|
| `run_id` | `str` | Unique run UUID |
| `module_id` | `str` | Module identifier |
| `module_version` | `str` | Semantic version |
| `organization_id` | `str` | Tenant organization |
| `user_id` | `str` | Initiating user |
| `environment` | `str` | `dev` / `staging` / `prod` |
| `correlation_id` | `str` | Request correlation ID |
| `requested_at` | `str` | ISO 8601 timestamp |
| `roles` | `tuple[str, ...]` | User roles |
| `mode` | `str \| None` | `interactive` or `workflow` |
| `tier` | `str \| None` | Pricing tier |
| `runtime_profile` | `str \| None` | Compute profile |
| `state_machine_execution_arn` | `str \| None` | Step Functions ARN |
| `credit` | `CreditReservation \| None` | `.reservation_id`, `.estimated_act` |
| `workflow` | `WorkflowMetadata` | `.workflow_id`, `.workflow_run_id`, `.workflow_step_id`, `.has_workflow` |

---

## Manifest Reference

`manifest.json` is the module's deployment descriptor. Required fields:

```json
{
  "schemaVersion": "1.0",
  "name": "my-module",
  "version": "1.0.0",
  "type": "no-ui",
  "computeType": "fargate",
  "description": "What this module does",
  "execution": {
    "entrypoint": "backend.main:MyModule",
    "supports": ["interactive", "workflow"],
    "defaultMode": "interactive"
  },
  "author": {
    "name": "Your Name",
    "email": "you@example.com",
    "organization": "Your Org"
  },
  "inputs": {
    "type": "object",
    "title": "Module Inputs",
    "required": ["sequence"],
    "properties": {
      "sequence": {
        "type": "string",
        "description": "Peptide sequence"
      },
      "options": {
        "type": "object",
        "properties": {
          "limit": {
            "type": "integer",
            "default": 100,
            "description": "Max results"
          }
        }
      }
    }
  },
  "outputs": {
    "results": {
      "type": "object",
      "description": "Analysis results"
    }
  },
  "resources": {
    "memory": 512,
    "vcpus": 0.25,
    "maxRuntimeSeconds": 3600,
    "environment": {
      "LOG_LEVEL": "INFO"
    }
  },
  "permissions": {
    "database": ["read"],
    "storage": true,
    "network": []
  },
  "computeType": "fargate",
  "pricing": {
    "tiers": [
      {
        "tier": "developer",
        "model": "usage-based",
        "amountAct": 0.25
      }
    ]
  },
  "metadata": {
    "exposedParameters": [],
    "auditLog": true
  }
}
```

### Compute Types

| Type | Use Case | Timeout |
|------|----------|---------|
| `lambda` | Fast, lightweight (< 15 min) | 900s |
| `fargate` | Standard modules (default) | 14400s (4h) |
| `batch` | GPU / long-running workloads | 86400s (24h) |

### UI Module Extensions

For `"type": "with-ui"` modules, add:

```json
{
  "ui": {
    "framework": "react",
    "buildCommand": "npm run build",
    "distPath": "frontend/dist"
  }
}
```

---

## Data Access Patterns

### Pattern 1: REST API (simple entity queries)

```python
def execute(self, input_data, context):
    peptides = self.api.get_peptides(
        filters={"sequence": input_data.sequence},
        limit=50,
    )
    return self.success(results={"peptides": peptides})
```

### Pattern 2: SQL Query (complex filtering, joins, aggregations)

```python
def execute(self, input_data, context):
    result = self.data.sql.query(
        """
        SELECT p.sequence, p.mass, s.name AS sample_name
        FROM peptides p
        JOIN peptide_samples ps ON ps.peptide_id = p.peptide_id
        JOIN samples s ON s.sample_id = ps.sample_id
        WHERE p.length >= %s
        ORDER BY p.mass DESC
        LIMIT %s
        """,
        params=[input_data.min_length, input_data.limit],
    )
    return self.success(results=result)
```

### Pattern 3: Direct Database Function

```python
def execute(self, input_data, context):
    if self.db:
        results = self.db.execute_function(
            "search_peptides_by_sequence",
            {"pattern": input_data.sequence, "max_results": input_data.limit},
        )
        return self.success(results={"peptides": results})

    # Fallback to REST API
    peptides = self.api.get_peptides(
        filters={"sequence": input_data.sequence},
        limit=input_data.limit,
    )
    return self.success(results={"peptides": peptides})
```

### Pattern 4: S3 Storage (save and retrieve artifacts)

```python
def execute(self, input_data, context):
    # Process data...
    results = {"analysis": "..."}

    # Save results to S3
    url = self.storage.save_json("analysis_results.json", results)

    # Read parquet from data catalog
    df = self.data.storage.read_parquet("datasets/peptides.parquet", engine="polars")

    return self.success(
        results={"artifact_url": url, "row_count": len(df)},
        summary={"saved": True},
    )
```

### Pattern 5: Pagination with Iterators

```python
def execute(self, input_data, context):
    all_peptides = []
    for peptide in self.api.iter_peptides(
        filters={"length_min": 8},
        page_size=100,
        max_results=5000,
    ):
        all_peptides.append(peptide)

    return self.success(
        results={"peptides": all_peptides},
        summary={"total": len(all_peptides)},
    )
```

---

## MCP Endpoint Integration (Forward-Looking)

> **Status:** Planned architecture. This pattern is not yet enforced by the
> platform but describes the target design for exposing modules as
> MCP-compatible tool servers for AI client consumption.

Modules can expose their functionality as MCP (Model Context Protocol) tools by
adding a thin FastAPI server alongside the Module class:

```python
"""mcp_server.py — MCP-compatible tool server wrapping the module."""

from __future__ import annotations

from typing import Any, Dict

from fastapi import FastAPI
from pydantic import BaseModel, Field

from hla_compass import Module
from hla_compass.client import APIClient

app = FastAPI(title="My Module MCP Server")


class AnalyseInput(BaseModel):
    """JSON Schema input for the MCP tool."""
    sequence: str = Field(..., description="Peptide sequence to analyse")
    allele: str = Field("HLA-A*02:01", description="Target HLA allele")


@app.post("/tools/analyse-peptide")
async def analyse_peptide(params: AnalyseInput) -> Dict[str, Any]:
    """MCP tool: Analyse a peptide against an HLA allele."""
    api = APIClient()
    peptides = api.get_peptides(filters={"sequence": params.sequence}, limit=10)
    return {
        "results": peptides,
        "allele": params.allele,
        "count": len(peptides),
    }


@app.get("/tools")
async def list_tools():
    """MCP tool discovery endpoint."""
    return {
        "tools": [
            {
                "name": "analyse-peptide",
                "description": "Search and analyse a peptide against an HLA allele",
                "inputSchema": AnalyseInput.model_json_schema(),
            }
        ]
    }
```

To declare MCP support in the manifest (planned):

```json
{
  "mcp": {
    "tools": [
      {
        "name": "analyse-peptide",
        "description": "Search and analyse a peptide against an HLA allele",
        "entrypoint": "mcp_server:app"
      }
    ]
  }
}
```

---

## Testing & Validation

```bash
# Validate manifest + code (7-layer check)
hla-compass validate

# Run locally with sample input
hla-compass dev                          # uses examples/sample_input.json
hla-compass dev --input custom.json      # custom input file

# Run module test suite
hla-compass test

# Local UI preview (with-ui modules only)
hla-compass serve
```

### Writing Tests

```python
"""tests/test_module.py"""

import pytest
from unittest.mock import MagicMock, patch
from backend.main import MyModule


@pytest.fixture
def module():
    return MyModule(manifest_path="manifest.json")


@pytest.fixture
def mock_context():
    return {
        "run_id": "test-run-1",
        "module_id": "test-module",
        "module_version": "1.0.0",
        "organization_id": "org-test",
        "user_id": "user-test",
        "environment": "dev",
        "correlation_id": "corr-1",
        "requested_at": "2024-01-01T00:00:00Z",
    }


def test_execute_returns_success(module, mock_context):
    input_data = {"sequence": "SIINFEKL", "allele": "HLA-A*02:01"}
    result = module.run(input_data, mock_context)
    assert result["status"] == "success"
    assert "results" in result
```

---

## Publishing

```bash
# Generate signing keys (one-time)
hla-compass keys init

# Publish to dev (org-scoped)
hla-compass publish --env dev --scope org

# Publish to prod (platform-wide, requires approval)
hla-compass publish --env prod --scope public

# Check publish job status
hla-compass publish-status <publish-job-id>
```

---

## Common Recipes

### Filter Peptides by HLA Allele

```python
def execute(self, input_data, context):
    if self.db:
        peptides = self.db.execute_function(
            "search_peptides_by_hla",
            {"allele_name": input_data.allele, "min_intensity": 0.5, "max_results": 100},
        )
    else:
        peptides = self.data.sql.query(
            """
            SELECT p.peptide_id AS id, p.sequence
            FROM scientific.peptides p
            JOIN scientific.samplepeptidebinders spb ON spb.peptide_id = p.peptide_id
            WHERE spb.allele_name = %s
            LIMIT 100
            """,
            params=[input_data.allele],
        )
        peptides = peptides["data"]
    return self.success(results={"peptides": peptides}, summary={"count": len(peptides)})
```

### Atomic Multi-File Save

```python
def execute(self, input_data, context):
    import json

    results = {"analysis": "..."}
    metadata = {"run_id": self.run_id, "timestamp": "2024-01-01"}

    urls = self.storage.save_files_atomic([
        {"filename": "results.json", "content": json.dumps(results)},
        {"filename": "metadata.json", "content": json.dumps(metadata)},
    ])
    return self.success(results={"artifacts": urls})
```

### Large Dataset with Pagination

```python
def execute(self, input_data, context):
    batch = []
    for peptide in self.api.iter_peptides(page_size=200, max_results=10000):
        batch.append(peptide)
        if len(batch) >= 1000:
            self._process_batch(batch)
            batch = []
    if batch:
        self._process_batch(batch)

    return self.success(results={"status": "processed"})
```

### Workflow-Mode Module with Step Artifacts

```python
def execute(self, input_data, context):
    workflow = self.context.workflow
    if workflow.has_workflow:
        self.logger.info(f"Running as workflow step: {workflow.workflow_step_id}")

    # Save step-specific artifact
    if self.storage and workflow.has_workflow:
        self.storage.save_workflow_file(
            "step_result.json",
            json.dumps({"step": "complete"}),
        )

    return self.success(results={"workflow": workflow.has_workflow})
```

### Credit-Aware Execution

```python
def execute(self, input_data, context):
    if self.credit:
        self.logger.info(
            f"Credit reservation: {self.credit.reservation_id}, "
            f"estimated ACT: {self.credit.estimated_act}"
        )

    # Proceed with analysis
    return self.success(results={"credit_reserved": self.credit is not None})
```

---

## Error Handling

```python
from hla_compass.module import Module, ModuleError, ValidationError

class MyModule(Module):
    def execute(self, input_data, context):
        if not input_data.get("sequence"):
            # Raises ValidationError caught by the framework
            raise ValidationError("Sequence is required")

        try:
            result = self.api.get_peptide(input_data["sequence"])
        except Exception as e:
            # Raises ModuleError with details
            self.error(f"Failed to fetch peptide: {e}", details={"sequence": input_data["sequence"]})

        return self.success(results=result)
```

Error types handled by the framework:
- `ValidationError` → `"validation_error"` status
- `ExecutionTimeoutError` → `"timeout_error"` status
- `ModuleError` → `"module_error"` status
- `Exception` (unhandled) → `"internal_error"` status

---

## Import Reference

```python
# Core
from hla_compass import Module
from hla_compass.module import Module, ModuleError, ValidationError, ExecutionTimeoutError

# API client (standalone usage outside modules)
from hla_compass.client import APIClient, APIError

# Data access
from hla_compass.data import DataClient, SQLClient, StorageClient, PeptideData

# Storage
from hla_compass.storage import Storage, StorageError

# Context
from hla_compass.context import RuntimeContext, CreditReservation, WorkflowMetadata

# Types
from hla_compass.types import (
    ExecutionContext, Peptide, Protein, Sample,
    JobStatus, ComputeType, ModuleType,
)

# Auth
from hla_compass.auth import Auth, AuthError

# Config
from hla_compass.config import Config
```
