# Phase 1.5: Launch Packaging Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Get mlx-triage repository into a polished, flywheel-optimized state so that visitors from blog posts, social media, and networking encounters arrive at an impressive, ready-to-use project.

**Architecture:** Documentation-heavy phase. No new features — polish existing work for public consumption. README is the single most important artifact (it's what visitors see first). All docs reference existing evidence in `docs/evidence/` and validation data in `docs/validation-results.md`. CI validates the existing 102-test suite across Python 3.11-3.13 on macOS.

**Tech Stack:** Markdown, GitHub Actions YAML, pyproject.toml (TOML), existing Python codebase unchanged.

**Preconditions already met this session:**
- LICENSE (MIT) — committed
- Stub CLI commands removed — committed
- 15 bugs fixed — committed
- 102 tests passing
- Check 0.4 false negative resolved (importlib.metadata)
- Shared model loading implemented (Bug #10)

---

## Task 1: README.md Overhaul

The #1 priority from the flywheel audit. This is what every visitor sees first.

**Files:**
- Modify: `README.md`

**Step 1: Write the full README**

Replace the current placeholder README with a complete, compelling project page.

```markdown
# mlx-triage

**Diagnose MLX model quality issues on Apple Silicon.**

Getting bad outputs from your MLX model? Before blaming the model, run `mlx-triage` to check whether the problem is the model weights, the MLX infrastructure, or your configuration.

```bash
pip install mlx-triage
mlx-triage check ./my-model
```

## What It Checks

### Tier 0 — Sanity Checks (no MLX needed, < 30 seconds)

| Check | What it catches |
|-------|----------------|
| **Dtype Compatibility** | BF16→FP16 precision loss, training/storage dtype mismatches |
| **Tokenizer & EOS Config** | Missing EOS tokens, chat template issues, Llama 3 dual-stop-token edge cases |
| **Weight File Integrity** | NaN/Inf values, all-zero layers, corrupt safetensors headers |
| **MLX Version & Known Bugs** | Outdated MLX with documented bugs affecting your model architecture |

### Tier 1 — Statistical Smoke Tests (MLX required)

| Check | What it catches |
|-------|----------------|
| **Determinism** | Non-reproducible outputs at temp=0 (infrastructure issue, not model) |
| **Reference Divergence** | MLX output diverging from PyTorch/Transformers reference |
| **Quantization Quality** | Excessive perplexity indicating broken quantization |

## Install

Requires Python 3.11+ and macOS on Apple Silicon (M1–M4).

```bash
# From PyPI
pip install mlx-triage

# With MLX for Tier 1 checks
pip install "mlx-triage[mlx]"

# With reference comparison (Tier 1, Test 1.2)
pip install "mlx-triage[reference]"

# Development
git clone https://github.com/swaylenhayes/mlx-triage.git
cd mlx-triage
uv sync --extra dev
```

## Usage

```bash
# Tier 0 only (default — no MLX needed)
mlx-triage check /path/to/model

# Tier 0 + Tier 1
mlx-triage check /path/to/model --tier 1

# JSON output
mlx-triage check /path/to/model --format json

# Save report to file
mlx-triage check /path/to/model --tier 1 --output report.json
```

Tier 0 runs in under 30 seconds on any model. Tier 1 requires MLX and takes 5–15 minutes depending on model size.

## How It Works

mlx-triage uses a tiered diagnostic protocol — each tier increases in depth and cost:

1. **Tier 0** reads model files directly (safetensors headers, config JSON, tokenizer config) without loading the model into memory. This catches the most common issues instantly.

2. **Tier 1** loads the model via MLX and runs statistical tests — determinism checks (10 runs at temp=0), perplexity measurement against a fixed eval corpus, and optional comparison against a PyTorch reference backend.

3. **Tiers 2–3** (planned) will add isolation tests (batch invariance, memory pressure, context length stress) and deep diagnostics (layer-wise activation comparison, cross-runtime analysis).

If Tier 0 finds critical issues, Tier 1 is skipped — fix the fundamentals first.

## Known Bugs Database

mlx-triage ships with a curated database of documented MLX bugs ([`known_bugs.yaml`](src/mlx_triage/data/known_bugs.yaml)), sourced from MLX GitHub issues and community postmortems. The version check cross-references your installed MLX version and model architecture against known issues.

Contributing a bug report to the database is the easiest way to help — see [CONTRIBUTING.md](CONTRIBUTING.md).

## Validation

Validated against 15 models across 5 families (Qwen, Llama, Phi, LiquidAI, Nanbeige), 4 quantization levels (bf16, 8-bit, 4-bit, 4-bit DWQ), and sizes from 0.6B to 30B parameters. Zero false negatives. See [full results](docs/validation-results.md).

## Requirements

- **macOS** on Apple Silicon (M1, M2, M3, M4)
- **Python 3.11+**
- Tier 0 works without MLX installed
- Tier 1 requires `mlx` and `mlx-lm` (`pip install "mlx-triage[mlx]"`)
- Reference divergence check (Test 1.2) requires `transformers` and `torch` (`pip install "mlx-triage[reference]"`)

## Research Basis

The diagnostic protocol is grounded in systematic analysis of MLX infrastructure defects and model quality issues. See [METHODOLOGY.md](METHODOLOGY.md) for the evidence basis and research citations.

## Contributing

Contributions welcome — especially to the known bugs database. See [CONTRIBUTING.md](CONTRIBUTING.md).

## License

[MIT](LICENSE)
```

**Step 2: Verify README renders correctly**

Run: `python -c "import pathlib; r = pathlib.Path('README.md').read_text(); print(f'README: {len(r)} chars, {len(r.splitlines())} lines')"`
Expected: ~3000+ chars, ~100+ lines. Visually review for Markdown correctness.

**Step 3: Commit**

```bash
git add README.md
git commit -m "docs: overhaul README for public launch"
```

---

## Task 2: pyproject.toml Polish

Add PyPI classifiers, project URLs, and author metadata.

**Files:**
- Modify: `pyproject.toml`

**Step 1: Add classifiers, URLs, and author**

Add after `license = {text = "MIT"}`:

```toml
authors = [
    {name = "Swaylen Hayes"},
]
classifiers = [
    "Development Status :: 4 - Beta",
    "Environment :: Console",
    "Intended Audience :: Developers",
    "Intended Audience :: Science/Research",
    "License :: OSI Approved :: MIT License",
    "Operating System :: MacOS",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
    "Programming Language :: Python :: 3.13",
    "Topic :: Scientific/Engineering :: Artificial Intelligence",
    "Topic :: Software Development :: Quality Assurance",
    "Typing :: Typed",
]
```

Add a new `[project.urls]` section after `[project.scripts]`:

```toml
[project.urls]
Homepage = "https://github.com/swaylenhayes/mlx-triage"
Repository = "https://github.com/swaylenhayes/mlx-triage"
"Bug Tracker" = "https://github.com/swaylenhayes/mlx-triage/issues"
Documentation = "https://github.com/swaylenhayes/mlx-triage#readme"
```

**Step 2: Verify pyproject.toml is valid**

Run: `uv run python -c "import tomllib; tomllib.load(open('pyproject.toml', 'rb')); print('Valid TOML')"`
Expected: `Valid TOML`

**Step 3: Verify build still works**

Run: `uv build --no-cache 2>&1 | tail -3`
Expected: Successfully built wheel and sdist.

**Step 4: Commit**

```bash
git add pyproject.toml
git commit -m "chore: add PyPI classifiers, URLs, and author to pyproject.toml"
```

---

## Task 3: CONTRIBUTING.md

Lead with the lowest-barrier contribution (known bugs YAML), then cover check development.

**Files:**
- Create: `CONTRIBUTING.md`

**Step 1: Write CONTRIBUTING.md**

```markdown
# Contributing to mlx-triage

Thanks for your interest in contributing. Here's how to get involved.

## Report a Known MLX Bug

The easiest and highest-impact contribution is adding to the known bugs database.

**File:** [`src/mlx_triage/data/known_bugs.yaml`](src/mlx_triage/data/known_bugs.yaml)

Each entry follows this format:

```yaml
- id: MLX-NNN
  title: "Short description of the bug"
  mlx_issue: 1234          # GitHub issue number, or null
  source: "optional source" # If no GitHub issue
  affected_versions: ["< 0.24.0"]  # or ["all"]
  severity: critical        # critical | high | warning
  detection: "tier0.version_check"  # Which check catches this
  symptom: "What the user sees when this bug bites them"
  architecture: ["llama", "mistral"]  # or ["all"]
  remediation: "What the user should do"
```

Open a PR with your addition. Include a link to the source (GitHub issue, postmortem, etc.).

## Add a Diagnostic Check

All checks follow the same pattern:

1. A check function in `src/mlx_triage/tierN/` that returns `DiagnosticResult`
2. Tests in `tests/test_tierN_*.py`
3. Wire into the tier runner (`src/mlx_triage/tierN/__init__.py`)

### Development Setup

```bash
git clone https://github.com/swaylenhayes/mlx-triage.git
cd mlx-triage
uv sync --extra dev
uv run pytest -v  # Should see 102 tests pass
```

### TDD Workflow

This project follows test-driven development:

1. Write a failing test in `tests/`
2. Run it to confirm it fails: `uv run pytest tests/test_file.py::test_name -v`
3. Write the minimal implementation to make it pass
4. Run the full suite: `uv run pytest -v`
5. Commit with a conventional commit message (`feat:`, `fix:`, `test:`, `docs:`)

### Check Function Contract

Every check function must:

- Return `DiagnosticResult` (never raise exceptions for expected conditions)
- Use `CheckStatus` severity levels: PASS < INFO < SKIP < WARNING < FAIL < CRITICAL
- Include `remediation` text for any status above PASS
- Include a `check_id` (e.g., `"0.1"`) and human-readable `name`

```python
from mlx_triage.models import CheckStatus, DiagnosticResult

def check_something(model_path: str) -> DiagnosticResult:
    # ... inspection logic ...
    return DiagnosticResult(
        check_id="0.5",
        name="Something Check",
        status=CheckStatus.PASS,
        detail="Everything looks good.",
    )
```

## Code Style

- Python 3.11+ with type hints
- No external linter enforced yet — keep it clean and consistent with existing code
- Conventional commits: `feat:`, `fix:`, `docs:`, `test:`, `chore:`

## Questions?

Open an issue. We're friendly.
```

**Step 2: Verify file exists and renders**

Run: `wc -l CONTRIBUTING.md`
Expected: ~80-90 lines

**Step 3: Commit**

```bash
git add CONTRIBUTING.md
git commit -m "docs: add CONTRIBUTING.md with bug database and check development guides"
```

---

## Task 4: METHODOLOGY.md

Document the evidence basis and tiered protocol rationale. This is what establishes credibility with researchers and advanced practitioners.

**Files:**
- Create: `METHODOLOGY.md`

**Step 1: Write METHODOLOGY.md**

```markdown
# Methodology

mlx-triage's diagnostic protocol is grounded in systematic analysis of MLX inference quality issues across multiple model architectures and quantization levels.

## The Problem

When an MLX-served model produces poor output, practitioners face a diagnostic ambiguity: is the problem in the model weights, the quantization, the inference infrastructure, or the configuration? Without systematic tooling, debugging this requires significant expertise and time.

## Evidence Basis

The diagnostic protocol was designed from analysis of:

- **Infrastructure defect taxonomy** — categorization of MLX bugs by type, severity, and affected components ([analysis](docs/evidence/c1-infrastructure-defects-analysis.md), [taxonomy](docs/evidence/c1-infrastructure-defects-taxonomy.md))
- **First-party experiments** — controlled experiments isolating infrastructure vs. model quality effects ([analysis](docs/evidence/first-party-experiment-analysis.md))
- **Cross-model synthesis** — patterns observed across multiple model families and quantization levels ([C1 synthesis](docs/context/c1-cross-model-synthesis.md), [C2 synthesis](docs/context/c2-cross-model-synthesis.md))
- **Systematic review** — structured review of MLX infrastructure constraints and model quality factors ([infrastructure constraints](docs/research/c2-mlx-infrastructure-constraints.md), [model quality](docs/research/c2-model-quality-dominance.md))

## Tiered Protocol

The protocol uses escalating tiers, each more expensive than the last:

### Tier 0 — Sanity Checks

**Cost:** < 30 seconds, no MLX required.
**Method:** Direct file inspection (safetensors headers, JSON configs).

Catches the most common issues without loading the model:
- Dtype mismatches between training and storage formats (evidence: Gemma 3 BF16→FP16 failure pattern)
- Missing or misconfigured tokenizer settings (evidence: Llama 3 dual-stop-token edge case)
- Corrupt or numerically invalid weight files (evidence: safetensors BF16 silent corruption)
- Known MLX bugs affecting the installed version and model architecture

### Tier 1 — Statistical Smoke Tests

**Cost:** 5–15 minutes, MLX required.
**Method:** Model inference with statistical analysis.

Runs the model and measures output quality:
- **Determinism** — 10 identical runs at temp=0 across 5 diagnostic prompts. 100% agreement expected; divergence indicates infrastructure non-determinism (evidence: MLX issue #488, float16 error accumulation)
- **Reference divergence** — compare MLX output against PyTorch/Transformers on the same prompts. Token agreement thresholds derived from empirical validation across 15 models
- **Quantization quality** — perplexity measured on a fixed 800-token eval corpus. Thresholds calibrated from validation data: <15 (PASS), <25 (INFO), <50 (WARNING), ≥50 (FAIL)

### Tier 2–3 — Planned

Isolation tests (batch invariance, memory pressure, context length stress) and deep diagnostics (layer-wise activation comparison, cross-runtime analysis). See [roadmap](docs/roadmap.md).

## Known Bugs Database

The tool ships with a curated database of documented MLX issues (`src/mlx_triage/data/known_bugs.yaml`). Each entry includes:
- Source (MLX GitHub issue or community postmortem)
- Affected versions and architectures
- Severity classification
- Detection tier and remediation steps

This database is the tool's institutional knowledge. Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md).

## Limitations

- **Apple Silicon only** — MLX runs exclusively on Apple Silicon. Tier 0 file checks work anywhere, but Tier 1+ requires an Apple Silicon Mac.
- **Local models only** — v0.1 requires a local model directory path, not a HuggingFace model ID.
- **Reference backend precision** — the PyTorch reference runs at FP32 on CPU, which may show small numerical differences from the model's native precision.
- **Perplexity thresholds** — calibrated on 15 models. Edge cases (very large or very small models) may need threshold adjustment in future versions.

## Validation

The diagnostic protocol has been validated against 15 models across 5 families, 4 quantization levels, and sizes from 0.6B to 30B parameters. See [validation results](docs/validation-results.md) for the full data.
```

**Step 2: Verify file exists**

Run: `wc -l METHODOLOGY.md`
Expected: ~70-80 lines

**Step 3: Commit**

```bash
git add METHODOLOGY.md
git commit -m "docs: add METHODOLOGY.md with evidence basis and protocol rationale"
```

---

## Task 5: GitHub Actions CI

Minimal CI that runs tests on push and PRs. macOS runner with Python matrix.

**Files:**
- Create: `.github/workflows/ci.yml`

**Step 1: Create the CI workflow**

```yaml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: macos-latest
    strategy:
      matrix:
        python-version: ["3.11", "3.12", "3.13"]

    steps:
      - uses: actions/checkout@v4

      - name: Install uv
        uses: astral-sh/setup-uv@v4

      - name: Set up Python ${{ matrix.python-version }}
        run: uv python install ${{ matrix.python-version }}

      - name: Install dependencies
        run: uv sync --extra dev

      - name: Run tests
        run: uv run pytest -v --tb=short
```

**Step 2: Validate YAML syntax**

Run: `python -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml')); print('Valid YAML')"`
Expected: `Valid YAML`

**Step 3: Commit**

```bash
git add .github/workflows/ci.yml
git commit -m "ci: add GitHub Actions test matrix for Python 3.11-3.13 on macOS"
```

---

## Task 6: Update current-state.md

Reflect all work done in sessions 5 (bug fixes + flywheel audit + this packaging phase).

**Files:**
- Modify: `_private/current-state.md`

**Step 1: Update the file**

Update the following sections:
- **Where We Are**: Phase 1.5 IN PROGRESS (packaging)
- **Pickup Context / First Action**: Next step is Phase 1.5b (publish to GitHub + PyPI)
- **Completed This Session**: Bug audit (15 fixes), flywheel audit, launch packaging docs
- **Open Questions / Blockers**: Remove Check 0.4 (fixed) and shared model loading (fixed). Keep "no remote configured"
- **Test count**: 102
- **Branch State**: Update commit hash and test count

**Step 2: Verify**

Run: `head -10 _private/current-state.md`
Expected: Updated phase status

**Step 3: Commit**

```bash
git add _private/current-state.md
git commit -m "docs: update current-state for Phase 1.5 progress"
```

---

## Task 7: Blog Post Outline

Saved for later execution by the user (human outreach work), but the outline itself is agent work.

**Files:**
- Create: `docs/plans/blog-post-outline.md`

**Step 1: Write the outline**

```markdown
# Blog Post Outline — Introducing mlx-triage

> For authoring by Swaylen Hayes. This outline provides structure; voice and details are yours.

## Title Options

- "Your MLX Model Isn't Broken — Your Infrastructure Might Be"
- "I Built a Diagnostic Tool for MLX Model Quality (Here's What I Found)"
- "Stop Guessing Why Your MLX Model Outputs Garbage"

## Structure

### Hook (2-3 sentences)

The problem: you download an MLX model, run it, get bad output. Is it the model? The quantization? An MLX bug? You have no way to know without hours of debugging.

### The Diagnostic Ambiguity Problem (1-2 paragraphs)

- MLX is great but young — real bugs exist (link to known_bugs.yaml examples)
- Model quality and infrastructure quality are confounded
- Practitioners waste time blaming the wrong thing

### What mlx-triage Does (2-3 paragraphs + terminal screenshot)

- Tiered protocol: cheap checks first, expensive checks only if needed
- Tier 0 in 30 seconds without even loading the model
- Tier 1 for statistical validation (determinism, reference comparison, perplexity)
- Include terminal screenshot showing a real check

### What I Found Testing 15 Models (2-3 paragraphs)

- 100% determinism at temp=0 across all tested models
- Perplexity range 1.8–4.8 (all healthy)
- Correctly caught an incomplete download and an incompatible model type
- Key takeaway: most issues are configuration, not model quality

### The Known Bugs Database (1 paragraph)

- Curated from MLX GitHub issues and community postmortems
- Community-contributable via YAML — no code changes needed
- Link to CONTRIBUTING.md

### Try It (call to action)

```bash
pip install mlx-triage
mlx-triage check ./your-model
```

- Link to GitHub repo
- Link to issues for bug reports
- Mention Tier 2–3 roadmap

## Target Venues

1. dev.to (primary)
2. Personal blog / site
3. Cross-post to MLX GitHub Discussions
4. Hacker News (Show HN)
5. Apple Silicon ML communities (Discord, Reddit r/LocalLLaMA)

## Assets Needed

- Terminal screenshot of a Tier 0 + Tier 1 run (capture manually)
- GitHub repo link (after push)
- PyPI link (after publish)
```

**Step 2: Verify**

Run: `wc -l docs/plans/blog-post-outline.md`
Expected: ~70-80 lines

**Step 3: Commit**

```bash
git add docs/plans/blog-post-outline.md
git commit -m "docs: add blog post outline for launch announcement"
```

---

## Execution Order & Dependencies

```
Task 1 (README)          ← #1 priority, do first
Task 2 (pyproject.toml)  ← independent, can parallel with Task 1
Task 3 (CONTRIBUTING.md) ← README links to it, do after Task 1
Task 4 (METHODOLOGY.md)  ← README links to it, do after Task 1
Task 5 (GitHub Actions)  ← independent, can parallel with Tasks 3-4
Task 6 (current-state)   ← do last, reflects all other work
Task 7 (blog outline)    ← independent, can parallel with anything
```

Tasks 1+2 can run in parallel. Tasks 3+4+5+7 can all run in parallel after Task 1. Task 6 is last.

---

## What's NOT in This Plan (Phase 1.5b — Publish)

These require a GitHub remote and are a separate plan:

- Create GitHub repository
- Push all code
- Verify CI passes
- `uv build` + PyPI publish
- Git tag `v0.1.0`
- Capture terminal screenshot for README
- Blog post (human work, not agent work)

---

## Verification After All Tasks

```bash
# Tests still pass
uv run pytest -v
# Expected: 102 tests, all passing

# Build still works
uv build --no-cache
# Expected: wheel + sdist created

# README renders (check char count)
python -c "print(len(open('README.md').read()), 'chars')"
# Expected: 3000+ chars

# All new files exist
ls README.md CONTRIBUTING.md METHODOLOGY.md LICENSE .github/workflows/ci.yml
# Expected: all present

# pyproject.toml has classifiers
grep -c "classifiers" pyproject.toml
# Expected: 1 (the classifiers key)
```
