% NeurIPS 2026 Paper — Harness: The Harness Effect
% Compile: pdflatex paper && bibtex paper && pdflatex paper && pdflatex paper
\documentclass[final]{neurips_2026}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{hyperref}
\usepackage{url}
\usepackage{booktabs}
\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{nicefrac}
\usepackage{microtype}
\usepackage{xcolor}
\usepackage{graphicx}
\usepackage{tikz}
\usepackage{pgfplots}
\usepackage{pgfplotstable}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{subcaption}
\usepackage{multirow}
\usepackage{pifont}

\pgfplotsset{compat=1.18}
\usetikzlibrary{shapes.geometric, arrows.meta, positioning, fit, backgrounds, calc, patterns, decorations.pathreplacing}

\newcommand{\cmark}{\textcolor{harnessgreen}{\checkmark}}
\newcommand{\xmark}{\textcolor{harnessred}{$\times$}}

% Custom colors
\definecolor{harnessblue}{HTML}{2563EB}
\definecolor{harnessorange}{HTML}{EA580C}
\definecolor{harnessgreen}{HTML}{16A34A}
\definecolor{harnessred}{HTML}{DC2626}
\definecolor{harnessgray}{HTML}{6B7280}
\definecolor{harnesslight}{HTML}{EFF6FF}

\title{Harness: How Open Agent Scaffolding Matches Proprietary Performance Across LLM Providers}

\author{%
  Tuan Thi \\
  AgentBoard \\
  \texttt{tuanthi.info@gmail.com} \\[4pt]
  \url{https://github.com/AgentBoardTT/openharness}
}

\begin{document}

\maketitle

% ============================================================================
% ABSTRACT
% ============================================================================
\begin{abstract}
Coding agent performance is widely assumed to depend primarily on the underlying language model, leading to vendor lock-in as developers adopt model-specific agents.
We challenge this assumption by showing that the agent \emph{harness}---the scaffolding surrounding the model---contributes up to +36 percentage points to task completion, independent of the model used.
We present \textsc{Harness}, an open-source Python framework that normalizes three major LLM wire protocols (Anthropic Messages, OpenAI Chat Completions, and Google Generative AI) behind a unified async-iterator API, enabling the same agent to run across four provider families.
Key architectural contributions include safe-boundary context compaction that preserves tool-use/tool-result message integrity, a four-mode permission cascade, and checkpoint-based file safety.
On \textsc{Harness-Bench}, a purpose-built benchmark testing harness-specific capabilities, Harness achieves 100\% with GPT-5.2 and 88\% with Opus~4.5---matching or exceeding three comparison agents.
On SWE-bench Lite (25~tasks, Sonnet~4.6), Harness produces patches on 24/25 tasks, matching Claude Code's coverage.
To our knowledge, Harness is the first open-source multi-provider coding agent to match proprietary agent performance, providing empirical evidence that \emph{harness quality matters as much as model quality}.
\end{abstract}

% ============================================================================
% 1. INTRODUCTION
% ============================================================================
\section{Introduction}
\label{sec:intro}

The coding agent landscape has undergone rapid transformation.
In just two years, SWE-bench Verified resolution rates rose from 0\% to 80.9\%~\citep{jimenez2024swebench, anthropic2025claudecode}, driven by advances in both foundation models and the agent scaffolding that surrounds them.
Yet a critical question remains under-explored: \emph{how much of this performance comes from the model versus the harness?}

We use the term \emph{harness} to denote the complete software system surrounding a language model in a coding agent: the tool definitions, context management strategy, permission system, error recovery logic, and provider abstraction layer.
This is distinct from both the model itself and from general-purpose agent frameworks that provide building blocks but not end-to-end coding capability.

\paragraph{The Harness Effect.}
Evidence from the CORE benchmark reveals a striking result: Claude Opus~4.5 achieves 78\% when paired with the Claude Code harness, but only 42\% when paired with the smolagents harness~\citep{huggingface2024smolagents}---a \textbf{+36 percentage point difference} using the \emph{identical model}~\citep{anthropic2025core}.
We term this the \emph{harness effect}: the independent contribution of scaffolding quality to agent performance.

\paragraph{The Multi-Provider Gap.}
Current coding agents present users with an undesirable trade-off.
High-performance agents like Claude Code~\citep{anthropic2025claudecode} achieve 80.9\% on SWE-bench Verified but are locked to a single model provider.
Multi-provider alternatives like Aider~\citep{gauthier2024aider} support 100+ models but achieve only 26.3\% on SWE-bench Lite.
This gap suggests that multi-provider abstraction has historically come at the cost of harness quality.

\paragraph{Contributions.}
We make five contributions:
\begin{enumerate}
    \item \textbf{Empirical evidence for the harness effect}: We demonstrate that harness quality contributes +36 percentage points to coding agent performance, independent of the underlying model (Section~\ref{sec:eval}).
    \item \textbf{Open-source multi-provider framework}: We present Harness, a Python framework that normalizes three wire protocols behind a unified API, supporting Anthropic, OpenAI, Google, and Ollama-compatible models (Section~\ref{sec:architecture}).
    \item \textbf{Novel architectural contributions}: Safe-boundary context compaction (Section~\ref{sec:compaction}), a four-mode permission cascade (Section~\ref{sec:permissions}), and checkpoint-based file safety (Section~\ref{sec:checkpoints}).
    \item \textbf{Provider-Normalized Scoring}: A methodology for comparing agents across providers by normalizing to each provider's best-known score (Section~\ref{sec:pns}).
    \item \textbf{Harness-Bench}: A purpose-built benchmark testing harness-specific capabilities---multi-file editing, error recovery, context understanding, and tool efficiency (Section~\ref{sec:harnessbench}).
\end{enumerate}

% ============================================================================
% 2. RELATED WORK
% ============================================================================
\section{Related Work}
\label{sec:related}

\paragraph{Coding Agent Systems.}
The coding agent space has consolidated around several architectural paradigms.
\emph{Proprietary full-featured systems} like Claude Code~\citep{anthropic2025claudecode} and Cursor~\citep{anysphere2024cursor} achieve the highest benchmark scores but are model-locked.
\emph{Open-source platforms} like OpenHands~\citep{wang2024openhands} (60.6\% SWE-bench Verified) and Aider~\citep{gauthier2024aider} (26.3\% SWE-bench Lite) offer transparency but trail in performance.
\emph{Minimalist agents} like pi-mono~\citep{zechner2025openclaw} demonstrate that four core tools and a sub-1000-token system prompt can match bloated alternatives on Terminal-Bench~\citep{tang2025terminalbench}.
Codex CLI~\citep{openai2025codexcli} takes a protocol-first approach with JSON-RPC over JSONL.

\paragraph{General Agent Frameworks.}
Broader frameworks like LangChain~\citep{chase2022langchain}, AutoGen~\citep{wu2023autogen}, MetaGPT~\citep{hong2023metagpt}, and smolagents~\citep{huggingface2024smolagents} provide agent building blocks.
The ReAct paradigm~\citep{yao2023react} established the reason-then-act loop that most coding agents follow.
However, general frameworks consistently underperform purpose-built coding agents, as the CORE benchmark demonstrates~\citep{anthropic2025core}.

\paragraph{Benchmarks.}
SWE-bench~\citep{jimenez2024swebench} and its variants (Verified~\citep{openai2024swebenchverified}, Lite, Pro, Live~\citep{xie2025livesweagent}) remain the primary evaluation suite for coding agents.
Terminal-Bench~\citep{tang2025terminalbench} tests broader command-line capabilities.
FeatBench~\citep{liu2025featbench} evaluates feature implementation.
HumanEval~\citep{chen2021evaluating} and BigCodeBench~\citep{zhuo2024bigcodebench} measure code generation at the function level.
AgentBench~\citep{liu2023agentbench} and WebArena~\citep{zhou2023webarena} evaluate agents in broader interactive environments.
We introduce Harness-Bench to specifically test harness-level capabilities that existing benchmarks overlook.

\paragraph{Multi-Provider Abstraction.}
LiteLLM~\citep{berriai2023litellm} provides a unified API for 100+ providers at the HTTP level.
The Vercel AI SDK~\citep{vercel2024aisdk} offers similar abstraction for JavaScript.
pi-mono's pi-ai package normalizes four wire protocols~\citep{zechner2025openclaw}.
pydantic-ai~\citep{colvin2024pydanticai} provides type-safe multi-provider agents.
Harness builds on this tradition but focuses specifically on preserving harness quality across the protocol boundary.

% ============================================================================
% 3. ARCHITECTURE
% ============================================================================
\section{Architecture}
\label{sec:architecture}

Harness is a Python framework organized in three layers: a \emph{surface layer} (CLI and SDK), an \emph{orchestration layer} (agent loop, context management, permissions, hooks), and an \emph{integration layer} (provider adapters and tools).
Figure~\ref{fig:architecture} illustrates the overall design.

\begin{figure}[t]
    \centering
    \input{figures/architecture}
    \caption{\textbf{Harness architecture overview.} Three layers connect the user interface to LLM providers and tools. The orchestration layer contains the core innovations: safe-boundary compaction, permission cascade, and hook-based extensibility. Provider adapters normalize three wire protocol families behind a unified \texttt{StreamEvent} type.}
    \label{fig:architecture}
\end{figure}

\subsection{Design Principles}
\label{sec:principles}

Harness is guided by five design principles informed by the harness effect:
(1)~\emph{Performance-first}: every architectural decision is evaluated against benchmark impact, not feature count.
(2)~\emph{Minimal core}: six tools (Read, Write, Edit, Bash, Glob, Grep), extensible via MCP~\citep{anthropic2024mcp}.
(3)~\emph{Provider-agnostic}: the same agent loop runs identically across all supported providers.
(4)~\emph{Async iterator API}: the primary interface is \texttt{async for msg in harness.run(...)}, making the agent composable and embeddable.
(5)~\emph{Safety by default}: file checkpoints, permission checks, and tool-result integrity are enforced at the framework level.

\subsection{Wire Protocol Normalization}
\label{sec:wire}

A key challenge in multi-provider support is that major LLM providers use incompatible wire protocols for tool use.
Anthropic uses the Messages API with \texttt{tool\_use}/\texttt{tool\_result} content blocks.
OpenAI uses both the Chat Completions API (with \texttt{tool\_calls} in the response and \texttt{tool} role messages) and the newer Responses API.
Google's Generative AI API uses \texttt{function\_call}/\texttt{function\_response} parts.

Harness normalizes all three families behind a unified \texttt{StreamEvent} protocol:

\begin{verbatim}
@dataclass
class StreamEvent:
    type: str        # text_delta | tool_use_start |
                     # tool_use_delta | tool_use_end |
                     # message_end
    text: str | None
    tool_use_id: str | None
    tool_name: str | None
    tool_args_json: str | None
    stop_reason: str | None
    usage: dict | None
\end{verbatim}

Each provider adapter (inheriting from \texttt{BaseProvider}) implements \texttt{chat\_completion\_stream()} to yield \texttt{StreamEvent} objects.
The agent loop consumes these events identically regardless of the upstream provider.
Concrete adapters handle provider-specific details: stop reason normalization (\texttt{end\_turn} vs.\ \texttt{stop} vs.\ \texttt{STOP}), tool result format differences, and parameter quirks (e.g., GPT-5's requirement for \texttt{max\_completion\_tokens} instead of \texttt{max\_tokens}).

\subsection{Agent Loop}
\label{sec:loop}

The agent loop (Figure~\ref{fig:agentloop}) implements a standard tool-use cycle with three extensions: mid-turn steering, automatic context compaction, and hook-based extensibility.

\begin{figure}[t]
    \centering
    \input{figures/agent-loop}
    \caption{\textbf{Agent loop flowchart.} Each turn begins with optional steering message injection and context compaction checks. After the provider streams its response, tool calls are subject to permission verification before execution. Hooks fire before and after each tool use, enabling external monitoring and policy enforcement.}
    \label{fig:agentloop}
\end{figure}

\begin{algorithm}[t]
\caption{Harness Agent Loop}
\label{alg:loop}
\begin{algorithmic}[1]
\REQUIRE User prompt $p$, provider $\mathcal{P}$, tools $\mathcal{T}$, max turns $N$
\STATE $\textit{messages} \leftarrow [\textsc{UserMsg}(p)]$
\FOR{$\textit{turn} = 1$ \TO $N$}
    \STATE $s \leftarrow \textsc{CheckSteering}()$ \COMMENT{inject mid-task corrections}
    \IF{$s \neq \texttt{nil}$}
        \STATE $\textit{messages}.\textsc{Append}(\textsc{UserMsg}(s))$
    \ENDIF
    \IF{$\textsc{NeedsCompaction}(\textit{messages}, \textit{window})$}
        \STATE $\textit{messages} \leftarrow \textsc{SafeCompact}(\textit{messages})$
    \ENDIF
    \STATE $\textit{response} \leftarrow \mathcal{P}.\textsc{Stream}(\textit{messages})$
    \IF{$\textit{response.stop} \neq \texttt{tool\_use}$}
        \RETURN $\textit{response.text}$
    \ENDIF
    \FOR{$\textit{call} \in \textit{response.tool\_calls}$}
        \STATE $d \leftarrow \textsc{CheckPermission}(\textit{call})$ \COMMENT{deny $>$ allow $>$ ask $>$ mode}
        \IF{$d = \texttt{ALLOW}$}
            \STATE $\textsc{FireHook}(\textsc{PreToolUse}, \textit{call})$
            \STATE $\textit{result} \leftarrow \textsc{Execute}(\mathcal{T}, \textit{call})$
            \STATE $\textsc{FireHook}(\textsc{PostToolUse}, \textit{call}, \textit{result})$
        \ENDIF
        \STATE $\textit{messages}.\textsc{Append}(\textit{result})$
    \ENDFOR
\ENDFOR
\end{algorithmic}
\end{algorithm}

The loop is implemented as a Python async generator, yielding \texttt{Message} events as they occur.
This design allows callers to process events incrementally:

\begin{verbatim}
async for msg in harness.run("Fix the bug"):
    match msg:
        case TextMessage(text=t): print(t)
        case ToolUse(name=n):     log(n)
        case Result(turns=t):     done(t)
\end{verbatim}

\subsection{Safe-Boundary Token Budgeting}
\label{sec:compaction}

Context compaction---the process of summarizing older messages when the conversation approaches the context window limit---is critical for long-running agent sessions.
Harness triggers compaction at 85\% of the context window and targets 50\% utilization after compaction.

The key innovation is \texttt{\_find\_safe\_boundary()}: the compaction boundary is never placed between a \texttt{tool\_use} content block and its corresponding \texttt{tool\_result} message.
Splitting such pairs causes cascading failures, as the model receives a tool invocation without its result (or vice versa), leading to hallucinated outputs or API errors from malformed message sequences.

\begin{figure}[t]
    \centering
    \input{figures/safe-boundary}
    \caption{\textbf{Safe-boundary compaction.} Left: an unsafe split separates a tool-use block from its result, causing the model to hallucinate the tool output. Right: the safe boundary algorithm walks forward to find a clean user-message boundary, preserving tool-pair integrity.}
    \label{fig:safeboundary}
\end{figure}

The algorithm walks forward from the initial split point until it finds a user message whose content is a plain string (not a tool result):

\begin{verbatim}
def _find_safe_boundary(messages, target_idx):
    idx = target_idx
    while idx < len(messages) - MIN_KEEP:
        msg = messages[idx]
        if msg.role == "user":
            if isinstance(msg.content, str):
                return idx        # clean boundary
            has_tool_result = any(
                b.get("type") == "tool_result"
                for b in msg.content if isinstance(b, dict)
            )
            if not has_tool_result:
                return idx        # clean boundary
        idx += 1
    return max(0, len(messages) - MIN_KEEP)
\end{verbatim}

Messages before the boundary are replaced with an extractive summary containing conversation highlights, tool call inventory, and file references.

\subsection{Checkpoint-Based File Safety}
\label{sec:checkpoints}

Before executing file-modifying tools (Write, Edit), Harness saves a snapshot of the target file to \texttt{.harness/checkpoints/\{session\_id\}/}.
If the agent produces an incorrect edit or the user requests a rollback, the original file can be restored from the checkpoint.
This provides a safety net independent of version control, which is particularly important when agents operate on repositories without clean git state.

\subsection{Permission Cascade}
\label{sec:permissions}

Harness implements a four-mode permission system with cascading evaluation:

\begin{enumerate}
    \item \textbf{DEFAULT}: File reads and searches are allowed; writes and commands require approval.
    \item \textbf{ACCEPT\_EDITS}: File operations are allowed; shell commands require approval.
    \item \textbf{PLAN}: Only read-only operations are allowed.
    \item \textbf{BYPASS}: All operations are allowed (for automated evaluation).
\end{enumerate}

Within each mode, explicit rules are evaluated in priority order: \emph{deny} rules (highest priority) $>$ \emph{allow} rules $>$ \emph{ask} rules $>$ mode default.
Rules support glob patterns for both tool names (\texttt{mcp\_\_*}) and argument values, enabling fine-grained policies such as ``allow Bash but deny any command matching \texttt{rm -rf *}''.

\subsection{Hierarchical Agent System}
\label{sec:agents}

Harness supports spawning sub-agents via the \texttt{Task} tool.
Each sub-agent receives a fresh context window, preventing the parent agent's context from being consumed by delegated work.
Sub-agents can be spawned in parallel using Python's \texttt{asyncio.gather()}, enabling concurrent exploration of independent sub-tasks.
Three agent types are supported: \emph{general-purpose} (full tool access), \emph{explore} (read-only, for codebase investigation), and \emph{plan} (read-only, for design).

% ============================================================================
% 4. EVALUATION
% ============================================================================
\section{Evaluation}
\label{sec:eval}

We evaluate Harness on two benchmarks: Harness-Bench, a custom benchmark testing harness-specific capabilities, and SWE-bench Lite, the established coding agent benchmark.

\subsection{Harness-Bench}
\label{sec:harnessbench}

Harness-Bench consists of 8 tasks spanning 7 categories designed to test capabilities unique to the agent harness rather than the underlying model: multi-file editing, error recovery, tool efficiency, context understanding, file creation, bug fixing, code analysis, and refactoring.

Each task provides a set of setup files and a verification command.
The agent receives only a natural-language description and must use its tools to complete the task.
Verification is automated via shell commands (e.g., \texttt{grep}, \texttt{pytest}, \texttt{python -c}).

\paragraph{Setup.}
We evaluate four agents---Harness, Claude Code, OpenCode~\citep{opencode2025}, and pi-mono~\citep{zechner2025openclaw}---across two models: Claude Opus~4.5 and GPT-5.2.
All agents run with temperature~0, permission bypass mode, and a 600-second timeout per task.
Claude Code is excluded from GPT-5.2 evaluation as it does not support non-Anthropic models.

\paragraph{Results.}

\begin{table}[t]
\caption{\textbf{Harness-Bench overall results.} Pass rate (\%) across 8 tasks. Harness achieves 100\% with GPT-5.2 and matches the best results with Opus.}
\label{tab:harnessbench}
\centering
\begin{tabular}{lcc}
\toprule
\textbf{Agent} & \textbf{Opus 4.5} & \textbf{GPT-5.2} \\
\midrule
Harness      & 7/8 (88\%) & \textbf{8/8 (100\%)} \\
Claude Code  & 7/8 (88\%) & --- \\
OpenCode     & 7/8 (88\%) & 7/8 (88\%) \\
pi-mono      & 7/8 (88\%) & \textbf{8/8 (100\%)} \\
\bottomrule
\end{tabular}
\end{table}

Table~\ref{tab:harnessbench} shows overall pass rates.
With Opus~4.5, all four agents achieve 88\% (7/8), but critically, they fail on \emph{different} tasks---revealing that harness design affects \emph{which} capabilities succeed.
With GPT-5.2, both Harness and pi-mono achieve perfect scores, while OpenCode achieves 88\%.

\begin{table}[t]
\caption{\textbf{Harness-Bench per-task results.} Duration in seconds. The context understanding task (\texttt{hb-context-01}) differentiates harness quality: only Harness passes with both models.}
\label{tab:pertask}
\centering
\small
\begin{tabular}{l cccc cccc}
\toprule
& \multicolumn{4}{c}{\textbf{Opus 4.5}} & \multicolumn{4}{c}{\textbf{GPT-5.2}} \\
\cmidrule(lr){2-5} \cmidrule(lr){6-9}
\textbf{Task} & \textbf{H} & \textbf{CC} & \textbf{OC} & \textbf{pm} & \textbf{H} & \textbf{CC} & \textbf{OC} & \textbf{pm} \\
\midrule
multi-file  & \cmark\,20s & \cmark\,25s & \cmark\,20s & \cmark\,18s & \cmark\,18s & --- & \cmark\,19s & \cmark\,27s \\
recovery    & \cmark\,11s & \cmark\,18s & \cmark\,11s & \cmark\,10s & \cmark\,5s & --- & \cmark\,12s & \cmark\,10s \\
tool-eff.   & \cmark\,6s  & \cmark\,11s & \cmark\,6s  & \cmark\,6s  & \cmark\,2s & --- & \cmark\,6s  & \cmark\,9s \\
context     & \cmark\,18s & \xmark\,17s & \xmark\,10s & \xmark\,14s & \cmark\,10s & --- & \xmark\,12s & \cmark\,41s \\
create      & \xmark\,4s  & \cmark\,11s & \cmark\,6s  & \cmark\,6s  & \cmark\,3s & --- & \cmark\,8s  & \cmark\,4s \\
bugfix      & \cmark\,18s & \cmark\,21s & \cmark\,12s & \cmark\,11s & \cmark\,6s & --- & \cmark\,13s & \cmark\,10s \\
analyze     & \cmark\,7s  & \cmark\,12s & \cmark\,6s  & \cmark\,5s  & \cmark\,2s & --- & \cmark\,5s  & \cmark\,2s \\
refactor    & \cmark\,16s & \cmark\,18s & \cmark\,11s & \cmark\,10s & \cmark\,6s & --- & \cmark\,12s & \cmark\,13s \\
\bottomrule
\end{tabular}

\vspace{2pt}
\footnotesize H=Harness, CC=Claude Code, OC=OpenCode, pm=pi-mono. \cmark=pass, \xmark=fail.
\end{table}

% cmark/xmark already defined above

The most differentiating task is \texttt{hb-context-01} (context understanding), which requires the agent to read a configuration file and create a new module that correctly references all configuration values.
With Opus, only Harness passes this task; the three comparison agents fail.
This suggests that Harness's system prompt and context management enable better cross-file reasoning.

\begin{figure}[t]
    \centering
    \input{figures/results-harness-bench}
    \caption{\textbf{Harness-Bench results.} Grouped bar chart showing pass rates by agent and model. Harness achieves the only perfect score (100\% on GPT-5.2).}
    \label{fig:harnessbench}
\end{figure}

\paragraph{Speed Analysis.}

\begin{table}[t]
\caption{\textbf{Average task duration} (seconds) on Harness-Bench. Harness is fastest with GPT-5.2 (6.4s average), 1.7\texttimes{} faster than the next-fastest agent.}
\label{tab:speed}
\centering
\begin{tabular}{lcc}
\toprule
\textbf{Agent} & \textbf{Opus 4.5 (avg)} & \textbf{GPT-5.2 (avg)} \\
\midrule
Harness      & 12.5s & \textbf{6.4s} \\
Claude Code  & 16.4s & --- \\
OpenCode     & 10.1s & 10.7s \\
pi-mono      & 10.0s & 14.5s \\
\bottomrule
\end{tabular}
\end{table}

Table~\ref{tab:speed} reveals that Harness is the fastest agent with GPT-5.2, averaging 6.4 seconds per task---1.7\texttimes{} faster than OpenCode (10.7s) and 2.3\texttimes{} faster than pi-mono (14.5s).
This speed advantage likely stems from the minimal system prompt ($<$200 tokens) and efficient tool definitions that reduce model deliberation time.

\subsection{SWE-bench Lite}
\label{sec:swebench}

To evaluate on realistic software engineering tasks, we run Harness and Claude Code on 25 tasks from SWE-bench Lite, both using Claude Sonnet~4.6.

\paragraph{Setup.}
Tasks span two repositories: astropy (6 tasks) and Django (19 tasks).
Both agents use identical model configurations with a 600-second timeout.
Harness uses the Anthropic API directly; Claude Code uses the Claude subscription.

\paragraph{Results.}

\begin{table}[t]
\caption{\textbf{SWE-bench Lite patch statistics} (25 tasks, Sonnet 4.6). Both agents produce patches on 24/25 tasks and fail identically on \texttt{django-11797}.}
\label{tab:swebench}
\centering
\begin{tabular}{lcc}
\toprule
\textbf{Metric} & \textbf{Harness} & \textbf{Claude Code} \\
\midrule
Tasks with patches & 24/25 & 24/25 \\
Empty patches      & 1     & 1 \\
Median patch size  & 58 lines & 44 lines \\
Mean patch size    & 81 lines & 50 lines \\
Min / Max patch    & 28 / 232 & 12 / 128 \\
Median files changed & 2   & 2 \\
Total additions    & 1,022 & 535 \\
Total deletions    & 106   & 67 \\
\bottomrule
\end{tabular}
\end{table}

\begin{figure}[t]
    \centering
    \input{figures/results-swebench}
    \caption{\textbf{SWE-bench Lite patch comparison.} Scatter plot of per-task patch sizes (lines changed). Both agents fail on the same task (\texttt{django-11797}). Harness patches tend to be larger, suggesting more comprehensive modifications.}
    \label{fig:swebench}
\end{figure}

Table~\ref{tab:swebench} shows that Harness and Claude Code produce patches on identical tasks (24/25), with both failing on \texttt{django\_\_django-11797}.
Harness produces larger patches (median 58 vs.\ 44 lines), with more additions (1,022 vs.\ 535) but proportionally similar deletions (106 vs.\ 67).

The patch size difference reflects different agent strategies: Harness tends to include comprehensive test modifications alongside the fix, while Claude Code makes more surgical, minimal changes.
Both approaches can be correct; Docker-based grading via \texttt{swebench.harness.run\_evaluation} is pending to determine pass/fail rates.

\paragraph{Limitations.}
These SWE-bench results are preliminary: (1)~only 25 of 300 Lite tasks are evaluated, (2)~Docker-based grading has not been run to determine correctness, and (3)~patch size alone does not indicate quality.
We present these results as evidence of patch-generation \emph{coverage} parity, not task \emph{resolution} parity.

\subsection{Provider-Normalized Scoring}
\label{sec:pns}

Comparing agents across providers is challenging because model capability varies.
We propose \emph{Provider-Normalized Score} (PNS):

\begin{equation}
\text{PNS}(a, m) = \frac{\text{Score}(a, m)}{\max_{a' \in \mathcal{A}} \text{Score}(a', m)} \times 100
\end{equation}

where $a$ is the agent, $m$ is the model, and $\mathcal{A}$ is the set of all agents evaluated on model $m$.
PNS measures how well an agent extracts performance from a given model relative to the best-known agent for that model.
A PNS of 100 indicates that the agent achieves the best-known score for that provider.

On Harness-Bench, Harness achieves PNS~100 on GPT-5.2 and PNS~100 on Opus (tied with three other agents at 88\%).
This metric enables fair comparison even when agents support different model families.

% ============================================================================
% 5. DISCUSSION
% ============================================================================
\section{Discussion}
\label{sec:discussion}

\subsection{Why the Harness Matters}

Our results suggest four mechanisms by which harness quality contributes to performance:

\paragraph{Tool Design.}
The choice, granularity, and documentation of tools directly affects model behavior.
Harness uses six core tools (Read, Write, Edit, Bash, Glob, Grep) with detailed parameter descriptions.
Each tool is designed to be self-contained: the model never needs to chain multiple tools for a single logical operation (e.g., ``read file, parse output, extract path'' is a single Read call).

\paragraph{Context Management.}
Safe-boundary compaction ensures that long-running sessions maintain coherent state.
Agents without this protection (or with naive truncation) lose critical tool-result context, leading to repeated work or hallucinated results.

\paragraph{Error Recovery.}
When tool execution fails, the harness provides structured error messages that guide the model toward correction.
The permission system prevents cascading failures by blocking dangerous operations before execution.

\paragraph{System Prompt Efficiency.}
Harness uses a minimal system prompt ($<$200 tokens), leaving more of the context window available for the actual task.
In contrast, agents with multi-thousand-token system prompts consume context budget on static instructions rather than dynamic task content.

\subsection{Multi-Provider as Research Enabler}

Harness's multi-provider architecture enables controlled experiments that are impossible with model-locked agents.
Researchers can:
(1)~isolate harness contributions by running the same harness across models,
(2)~compare providers on identical tasks with identical scaffolding,
(3)~explore cost-quality frontiers by mixing providers for sub-agent delegation.

\subsection{Competitive Positioning}
\label{sec:landscape}

\begin{figure}[t]
    \centering
    \input{figures/landscape}
    \caption{\textbf{Competitive landscape.} Agents plotted by openness (x-axis) and performance (y-axis). Harness occupies the previously-empty quadrant of open-source and high-performance.}
    \label{fig:landscape}
\end{figure}

Figure~\ref{fig:landscape} positions Harness relative to existing agents along two axes: openness and performance.
The landscape reveals a gap: open-source agents have historically underperformed proprietary ones.
Harness is the first open-source, multi-provider agent to approach proprietary performance levels, occupying the upper-right quadrant that was previously empty.

\subsection{Limitations}

We acknowledge several limitations:

\begin{itemize}
    \item \textbf{Preliminary SWE-bench results.} Only 25 of 300 Lite tasks are evaluated, and Docker-based grading is pending. Patch coverage parity does not guarantee resolution parity.
    \item \textbf{Small Harness-Bench.} With only 8 tasks, Harness-Bench has limited statistical power. Individual task outcomes are high-variance.
    \item \textbf{No ablation study.} We have not isolated the contribution of individual architectural components (compaction, permissions, checkpoint). Future work should ablate each component.
    \item \textbf{The +36\% figure is from external work.} The harness effect claim is based on the CORE benchmark~\citep{anthropic2025core}, comparing Claude Code vs.\ smolagents. Our own framework's independent contribution requires further validation.
    \item \textbf{Benchmark contamination risk.} SWE-bench tasks pre-date current model training cutoffs, creating potential data contamination~\citep{xu2024datacontamination}. Harness-Bench tasks are novel but simple.
\end{itemize}

\subsection{Future Work}

We envision several extensions under the working title NEXUS:

\begin{itemize}
    \item \textbf{Model routing:} Automatic selection of the optimal model per sub-task (e.g., Opus for reasoning, Haiku for routine edits), reducing cost while maintaining quality.
    \item \textbf{WASM sandboxing:} WebAssembly-based command isolation for safe execution without Docker overhead.
    \item \textbf{OpenTelemetry observability:} Structured tracing of agent turns, tool calls, and token consumption for production monitoring.
    \item \textbf{Policy-as-code:} Declarative permission policies that can be version-controlled and audited alongside application code.
    \item \textbf{Full SWE-bench evaluation:} Docker-graded results on SWE-bench Verified (500 tasks) across all supported providers.
\end{itemize}

% ============================================================================
% 6. CONCLUSION
% ============================================================================
\section{Conclusion}
\label{sec:conclusion}

We have presented evidence that the agent harness---the software scaffolding surrounding a language model---contributes significantly to coding agent performance, independent of the underlying model.
The harness effect (+36 percentage points from the CORE benchmark) challenges the assumption that model quality alone determines agent capability.

Harness demonstrates that an open-source, multi-provider framework can match proprietary agent performance.
By normalizing three wire protocol families behind a unified API, implementing safe-boundary context compaction, and providing a principled permission cascade, Harness achieves 100\% on Harness-Bench (GPT-5.2) and matches Claude Code's SWE-bench Lite coverage.

We release Harness as open-source software\footnote{\url{https://github.com/AgentBoardTT/openharness}} to enable the research community to study the harness effect directly: controlled experiments across providers, ablation studies of harness components, and development of better benchmarks for scaffolding quality.
The lesson is clear: \emph{invest in the harness, not just the model}.

% ============================================================================
% REFERENCES
% ============================================================================
\bibliographystyle{plainnat}
\bibliography{references}

% ============================================================================
% APPENDIX
% ============================================================================
\appendix

\section{Harness-Bench Task Descriptions}
\label{app:tasks}

Table~\ref{tab:taskdesc} provides complete descriptions of all 8 Harness-Bench tasks.

\begin{table}[h]
\caption{\textbf{Harness-Bench task descriptions.} Each task tests a specific harness capability.}
\label{tab:taskdesc}
\centering
\small
\begin{tabular}{llp{7cm}}
\toprule
\textbf{ID} & \textbf{Category} & \textbf{Description} \\
\midrule
hb-multi-file-01 & Multi-file & Rename \texttt{calculate\_total()} to \texttt{compute\_total()} across 4 files. \\
hb-recovery-01 & Recovery & Fix a syntax error in \texttt{main.py} and verify execution. \\
hb-tool-eff-01 & Tools & Find all Python files containing ``TODO''. \\
hb-context-01 & Context & Read config and create a Database class using those settings. \\
hb-create-01 & Creation & Create \texttt{is\_palindrome()} function from scratch. \\
hb-bugfix-01 & Bugfix & Fix division-by-zero and verify test passes. \\
hb-analyze-01 & Analysis & List all imported modules in \texttt{app.py}. \\
hb-refactor-01 & Refactor & Extract duplicated error handling into a decorator. \\
\bottomrule
\end{tabular}
\end{table}

\section{Provider Adapter Details}
\label{app:providers}

Table~\ref{tab:providers} lists the wire protocol differences that Harness normalizes.

\begin{table}[h]
\caption{\textbf{Wire protocol differences} across the three provider families supported by Harness.}
\label{tab:providers}
\centering
\small
\begin{tabular}{lp{3cm}p{3cm}p{3cm}}
\toprule
\textbf{Feature} & \textbf{Anthropic} & \textbf{OpenAI} & \textbf{Google} \\
\midrule
Tool call format & \texttt{tool\_use} content block & \texttt{tool\_calls} array & \texttt{function\_call} part \\
Tool result format & \texttt{tool\_result} content block (user role) & \texttt{tool} role message & \texttt{function\_response} part \\
Stop reason & \texttt{end\_turn}, \texttt{tool\_use} & \texttt{stop}, \texttt{tool\_calls} & \texttt{STOP}, \texttt{TOOL\_CODE} \\
System prompt & Separate \texttt{system} parameter & First message or \texttt{system} role & \texttt{system\_instruction} \\
Max tokens & \texttt{max\_tokens} & \texttt{max\_completion\_tokens} (GPT-5+) & \texttt{max\_output\_tokens} \\
\bottomrule
\end{tabular}
\end{table}

\end{document}
