# The model is almost always the problem, not the runtime

**When LLM output quality disappoints, the fix is overwhelmingly a better model — not a better quantization scheme, memory manager, or serving stack.** Across 500,000+ evaluations, practitioner case studies, and community benchmarks from 2023–2025, the evidence converges: modern 4-bit quantization methods introduce 1–6% quality loss while cross-generation model improvements deliver 30–50% gains. The gap between the best and worst 4-bit quantization method on any benchmark is dwarfed by the gap between a mediocre and excellent base model. This makes the hypothesis that inference defects meaningfully mimic model-quality defects a low-probability scenario in most practical settings, and decoupled infrastructure-vs-weights testing an overhead that rarely pays for itself.

That said, the evidence is nuanced. No rigorous cross-framework quality comparison between MLX and CUDA exists, and edge cases in specific architectures do arise. Here is the full disconfirming case.

## All modern 4-bit methods have converged within a 1–3% quality band

The strongest evidence against infrastructure-focused debugging comes from quantization benchmarks. The oobabooga community benchmark on Llama-2-13B measured perplexity across every major 4-bit method — EXL2, AWQ, GPTQ, GGUF Q4_K_M, and BitsandBytes load_in_4bit — and found **the entire spread spans just 0.069 perplexity points** (4.308 to 4.377), a ~1.6% range. A JarvisLabs benchmark on Qwen2.5-32B-Instruct running on an H200 found that AWQ, GGUF Q4_K_M, BitsandBytes NF4, and Marlin-AWQ all achieved identical **51.8% HumanEval Pass@1** scores. The perplexity spread across all methods stayed within 6% of the FP16 baseline.

The academic literature confirms this convergence. The landmark "Give Me BF16 or Give Me Death?" paper by Kurtic et al. (ACL 2025) conducted over **500,000 individual evaluations** across the Llama-3.1 family at 8B, 70B, and 405B scales. FP8 quantization proved "effectively lossless." INT8 incurred only 1–3% degradation. Even INT4 weight-only quantization recovered >99% accuracy on average, with text similarity analysis showing quantized outputs closely resembling full-precision baselines. The llama.cpp project's own perplexity tables show Q4_K_M adding only **+1.20%** perplexity over FP16 on 70B models, while Q5_K_M adds a mere +0.40%.

What this means practically: switching from GPTQ to AWQ at 4-bit on the same model gains roughly **0.26 percentage points on MMLU** according to LessWrong benchmarking — an improvement so small it falls within measurement noise. If your model's outputs are unsatisfactory, changing the quantization scheme is essentially rearranging deck chairs.

## A quantized big model beats an unquantized small model every time

The most decisive evidence for "upgrade the model, not the infrastructure" comes from scale-comparison studies. Lee et al. (2024) explicitly demonstrated that **"quantizing a larger LLM to a similar size as a smaller FP16 LLM generally performs better across most benchmarks"** across 7B to 405B models tested with GPTQ, AWQ, SmoothQuant, and FP8. Jin et al. (2024) independently confirmed: "Quantized LLMs with larger parameter scales can outperform smaller LLMs."

The arithmetic is stark. Cross-generation quality improvements — Llama 2 to Llama 3, or Qwen 1.5 to Qwen 2.5 — deliver **30–50% benchmark gains**. Quantization from FP16 to 4-bit costs 1–6%. The generational leap is 5–10× larger than the quantization penalty. A 70B model at 4-bit (~35 GB) routinely outperforms a 13B model at full precision (~26 GB) on every benchmark tested. As one practitioner synthesis put it: "Use the largest model that fits your hardware at moderate quantization, not a small model at high precision."

This hierarchy has been confirmed in code generation specifically. Giagnorio et al. (March 2025) found that **"the new frontier for LLM quantization is 4-bit precision, resulting in an average memory footprint reduction of 70% without observing any significant decrease in performance."** Dettmers et al. (QLoRA) concluded that "4-bit quantization is almost universally optimal to reduce the model bits and maximize zero-shot accuracy."

## Practitioners who blamed infrastructure usually found model limitations

Concrete case studies reinforce the quantitative picture. The vllm-project/semantic-router issue #647 is a textbook example: a bug was filed against the PII enforcement infrastructure code, describing what appeared to be a routing/serving failure. Investigation revealed the enforcement logic worked perfectly. The actual problem was that the PII classification model produced confidence scores too low to trigger enforcement — SSNs scored **0.6478** against a 0.7 threshold, credit cards scored 0.3679, and the model misclassified credit card numbers as phone numbers. The fix was model-level, not infrastructure-level.

The MLX-VLM comprehensive testing report (GitHub issue #682) tested 38 vision-language models on Apple Silicon and found **12 models (32%) exhibited quality issues despite successful infrastructure execution**. Microsoft's Phi-3.5-vision-instruct fabricated entirely fictional content — inventing student enrollment data from 2010 to 2032 when shown images of English buildings. PaliGemma-2-3B at 4-bit produced minimal responses ignoring context. All models ran without infrastructure errors. Every quality problem traced to model capability, not serving defects.

The Llama 3 quantization sensitivity saga provides another instructive example. When users on r/LocalLLaMA and the llama.cpp GitHub found Llama 3 8B producing poor outputs at various quantization levels, they exhaustively tested Q4_0, Q4_K_M, IQ4_XS, and Q8_0. No quantization scheme switch resolved the issues. The root cause was architectural: Llama 3's training on **15 trillion tokens** (versus Llama 2's 2 trillion) created denser weight utilization that made it inherently more quantization-sensitive. The solutions were using a different model architecture (Solar or Mistral 7B quantized better), using full precision, or moving to a larger model. Infrastructure tuning was a dead end.

## MLX quantization quality matches the industry baseline — but no cross-framework comparison exists

The MLX-LM project's official BENCHMARKS.md provides the most authoritative MLX-ecosystem quality data. On Qwen3-4B-Instruct, 8-bit quantization loses only **0.20 MMLU Pro points** (0.31% degradation from bf16). Even aggressive 4-bit quantization shows 3.33 points of degradation (5.2%) — numbers consistent with every other quantization framework's results at the same precision. On the larger Qwen3-30B MoE model, 4-bit degradation drops to **1.91 points (2.6%)**, confirming that larger models absorb quantization more gracefully regardless of runtime.

MLX offers sophisticated quality-preservation tools: DWQ (distillation-based weight quantization using the full-precision model as teacher), AWQ, dynamic per-layer sensitivity quantization, GPTQ, and cascaded methods combining multiple approaches. Mixed-precision quantization via `--quant-predicate` allows keeping attention layers at higher precision while aggressively quantizing feedforward layers — a technique that proved essential for DeepSeek-V3's MLA architecture, where standard quantization of attention tensors caused severe degradation.

However, a critical gap exists: **no rigorous study compares MLX bf16 inference output quality to CUDA/PyTorch bf16 inference on the same model weights.** Every academic paper benchmarking MLX — including "Benchmarking On-Device ML on Apple Silicon" (arXiv:2510.18921), "Production-Grade Local LLM Inference on Apple Silicon" (arXiv:2511.05502), and Apple's own M5 showcase — measures only latency and throughput, never output fidelity. Apple makes no claims about numerical equivalence to CUDA. One Towards Data Science practitioner comparison noted MLX accuracy was "consistently slightly below" PyTorch on CNN tasks, without further investigation.

The absence of evidence for cross-framework equivalence is not evidence of absence — but it means the claim that "MLX and CUDA produce statistically indistinguishable outputs" remains an **untested assumption**. MLX issue #963 found quantized matrix operation errors were "negligible" versus unquantized versions, but the developer explicitly noted they "haven't run any vigorous perplexity measurements." The mlx_lm.evaluate tool itself has known bugs (issue #255) producing anomalous results on certain models, making cross-platform comparison through different evaluation harnesses unreliable.

## The diagnostic overhead of decoupled testing frequently exceeds its value

The Anthropic infrastructure crisis of August–September 2025 provides the most documented case of infrastructure-vs-model diagnostic confusion, though it cuts in the opposite direction from the infrastructure-innocence thesis. Three overlapping infrastructure bugs — context window misrouting affecting **16% of Sonnet 4 requests** at peak, TPU output corruption inserting Thai characters into English responses, and an XLA compiler bug eliminating highest-probability tokens — took **six weeks** to fully diagnose and resolve. Anthropic's own automated evaluations "simply didn't capture the degradation users were reporting," and privacy constraints prevented engineers from reproducing user-facing symptoms.

The lesson cuts both ways. Users assumed the model had been deliberately degraded ("lobotomized") when the problem was purely infrastructure. But teams consuming Claude's API who experienced degradation and began debugging their own infrastructure stacks — prompt engineering, RAG pipelines, quantization settings — were chasing ghosts. The emerging practitioner wisdom, articulated in debugging guides like paddo.dev's framework, now includes **"try a different model" as an early diagnostic step** rather than a last resort.

The "intelligence drift" pattern — documented across GPT-4, Claude 3.5 Sonnet, and GPT-4o between 2023 and 2025 — shows how the infrastructure-vs-model question recurs constantly. Jon Stokes investigated perceived Claude degradation and found his own behavior had changed: "I had gotten comfortable, lazy, and overconfident in the model's capabilities." Charlie Guo's analysis cataloged five competing theories for quality drops (cost-cutting, infrastructure bugs, context rot, user drift, post-training changes) and concluded that "we often hand-wave away inference as essentially a solved problem" when in reality "serving LLMs at scale is unprecedented in its complexity."

No published cost-benefit analysis of decoupled infrastructure-vs-model testing exists — this is a genuine gap in the practitioner literature. But the implicit calculus is clear from community practice: trying a better model takes minutes and costs nothing, while debugging infrastructure can consume weeks. The r/LocalLLaMA community has converged on a practical heuristic — "a smaller model at moderate quantization often beats an enormous model at harsh quantization" — that bypasses the infrastructure debugging question entirely by treating model selection and quantization level as a single joint decision.

## Conclusion: model selection dominates, but edge cases remain real

The disconfirming case against infrastructure-focused debugging is strong but not absolute. Five key takeaways emerge from this evidence:

**First, quantization method choice is nearly irrelevant at 4-bit and above.** All modern methods (AWQ, GPTQ, GGUF k-quants, BitsandBytes NF4) converge within a 1–3% quality band — far smaller than the difference between model generations or size classes. Switching quantization schemes will almost never solve a meaningful quality problem.

**Second, model architecture determines quantization tolerance, not the runtime.** Llama 3 quantizes worse than Gemma 2 or Qwen 2 at the same bit-width due to training decisions, not serving infrastructure. No infrastructure optimization can compensate for an architecture that uses its precision budget densely.

**Third, the MLX runtime introduces no known quality degradation beyond what quantization itself causes.** MLX's 8-bit quantization loses <0.3% MMLU Pro; its 4-bit numbers match industry baselines. But the absence of a direct MLX-vs-CUDA quality comparison means this conclusion rests on indirect evidence rather than rigorous testing.

**Fourth, "try a better model first" is the highest-leverage debugging action.** Cross-generation improvements are 5–10× larger than quantization losses. When 32% of vision-language models tested on MLX showed quality issues with working infrastructure, model selection was the only fix available.

**Fifth, decoupled infrastructure-vs-model testing may still be justified for production systems with strict quality requirements**, particularly on novel architectures (like DeepSeek-V3's MLA) or at extreme quantization levels (3-bit and below). The Anthropic case proves infrastructure bugs can be real and severe. But for the common case of a practitioner running a well-established model at 4-bit quantization on MLX, the infrastructure layer is overwhelmingly likely to be innocent — and the first diagnostic step should be trying a better model, not instrumenting the runtime.