This AI Agent Got Better Without Training the Model

International access: https://4sapi.com
Domestic access: https://4sapi.org

Abstract

Harness², an open-source framework jointly released by Google Cloud AI Research and Northwestern University, redefines agent self-improvement by separating model weights from surrounding execution scaffolding. Published in the paper *Recursive Agent Harnessing for an Open World* (arXiv:2610.14224), the system freezes model parameters entirely. No weights are updated or trained. Instead, optimization targets eight file-backed components wrapped around the base large language model: system prompts, safety guardrails, memory modules, tool instruction sets, sub-agents, skills, scripts and plugin middleware. Its recursive two-layer mechanism, Propose-Probe-Compose, enables iterative harness refinement without ground-truth answers or predefined test suites. Benchmark results across WorkBuddy-Bench, JobBench and Harvey LAB show consistent gains for four distinct model-executor combinations after a single optimization pass. In some configurations, optimized GLM-5 surpasses native Gemini 3.5 Flash. This article unpacks the architecture, recursive workflow, benchmark metrics, core innovations, practical limitations, and paradigm shift toward harness-centric agent competition.

1. Introduction: The Bottleneck of Traditional Agent Improvement

Most existing approaches to improve agent performance focus on model weight updates. Fine-tuning, reinforcement learning from human feedback, and knowledge distillation all modify the internal parameters of the base model. These methods carry high costs. They require large datasets, heavy compute resources, careful hyperparameter tuning, and lengthy iteration cycles. Once deployed, re-training a model to adapt to new domains or fix behavioral flaws is slow and operationally expensive.

This creates a fundamental constraint for production agent teams. A model trained for one task distribution may fail in new environments. Minor behavioral defects require full retraining cycles. Teams cannot rapidly adjust agent behavior for domain-specific workflows without incurring significant compute overhead.

Harness² addresses this pain point with a different paradigm. The underlying LLM stays unchanged. All improvements live in the outer execution harness, a set of editable, file-based components that govern how the model interacts with tools, memory and sub-systems. The framework’s recursive loop can iteratively revise these components and validate changes purely from observed execution traces. It does not require manually written test cases or reference ground truth. This shifts agent optimization from parameter space into the editable scaffolding layer.

2. Core Architecture: Eight File-Backed Harness Components

The core design principle of Harness² is strict separation between frozen model weights and modifiable harness components. The base model remains static throughout all optimization runs. Tuning operates exclusively on eight modular components, all stored as files. This modular design makes every change auditable, version-controllable, and reversible.

1. System prompts: Core behavioral instructions defining agent goals, constraints and response format.
2. Safety guardrails: Boundary rules to restrict risky actions and limit out-of-scope behavior.
3. Memory: Persistent storage for task history, lessons learned and cross-task experience.
4. Tool instruction sets: Definitions for available tools, input schemas and usage guidance.
5. Sub-agents: Delegated helper agents assigned to subtasks inside the main workflow.
6. Skills: Reusable atomic action sequences for repeated operations.
7. Scripts: Executable automation logic that runs alongside model outputs.
8. Plugin middleware: Interfaces that mediate communication between the model, tools and external services.

Because all eight components are files, engineers can track every revision using version control. A bad harness modification can be rolled back instantly, without model redeployment. This is a stark contrast to fine-tuning, where reverting a model checkpoint is far more complex and resource-intensive.

This architecture targets a key insight: many agent failures do not stem from insufficient model capability. They arise from poorly written prompts, incomplete tool guidance, weak memory handling or missing guardrails. These issues are located in the harness layer, not inside model weights. Fixing the harness can deliver meaningful performance gains without touching model parameters.

3. Two-Tier Recursive Mechanism: Propose-Probe-Compose

Harness²’s optimization runs on two recursive layers: task-level refinement and cross-task experience accumulation. The central algorithm is the contrastive Propose-Probe-Compose loop.

3.1 Task-level Propose-Probe-Compose

For a single target task, the workflow starts by running the existing base harness and capturing the full execution trace. The system then proposes two candidate revisions. One candidate addresses general behavioral defects observed in the trace. The second injects domain-specific knowledge relevant to the task.

Both candidate harness variants are tested independently on the same task. The framework compares the execution trajectories from the base harness and the two modified harnesses. It retains only those edits that deliver measurable improvements visible in behavioral evidence. Critically, this evaluation does not require manually labeled correct answers or handwritten test cases. Judgement relies purely on observable differences in execution traces.

This removes one of the biggest barriers for agent iteration: the need to build large labeled evaluation sets before optimization. Teams can run harness improvements on tasks where reference answers are unavailable or hard to construct.

3.2 Cross-task knowledge storage: EXPERIENCE.md

When a harness revision delivers consistent gains, Harness² saves the updated harness together with an experience note file EXPERIENCE.md into a shared improvement repository. This repository accumulates lessons from past tasks. The stored knowledge guides harness generation for future, unseen tasks.

This cross-task layer creates recursive improvement. As more tasks are processed, the repository builds a library of proven harness patterns. The system learns which prompt edits, guardrail rules or tool definitions reliably fix certain failure modes across different domains. The learning accumulates entirely in the harness repository, not inside model parameters.

4. Benchmark Results and Measured Performance Gains

The paper evaluates Harness² with a single-step optimization pass, denoted k=1. The experiment tests four separate model-executor combinations across three benchmark suites: WorkBuddy-Bench, JobBench and Harvey LAB.

The results reveal an important trend: weaker executors capture larger relative performance gains. After harness optimization, GLM-5 paired with OpenCode outperforms the native Gemini 3.5 Flash baseline. This demonstrates that the quality of the surrounding scaffolding can close capability gaps between different base models.

Harness² is released under Apache 2.0 license. It has native compatibility with OpenCode and Codex executor backends. The open licensing allows engineering teams to deploy, modify and integrate the framework for custom agent workloads.

5. Paradigm Shift: Competition Shifts From Model Weights to Harness Evolution

One of the most consequential takeaways from Harness² is its redefinition of agent competition. Traditionally, teams compete by building larger, better-trained base models. Harness² shows that self-improvement can happen at a much lower cost in the harness layer. The competitive battlefield moves from model pre-training and fine-tuning toward harness evolution.

This creates new strategic options for teams. Instead of investing heavily in model weight updates, teams can focus engineering work on iterating prompt structures, tool definitions, memory rules and sub-agent coordination. The base model remains frozen, while the surrounding scaffolding evolves continuously.

For production systems, this has practical operational advantages. Updates are faster, cheaper and fully auditable. A harness change can be deployed, tested and rolled back in minutes. Model fine-tuning often requires days or weeks of compute work, with irreversible changes to model weights.

6. Limitations and Boundary Conditions

Harness² delivers promising benchmark gains, but the framework carries clear constraints that practitioners must consider.

First, all reported metrics come from self-conducted benchmark tests in the original research paper. Results may not directly transfer to production workloads. Real-world tasks contain more distribution shift, edge cases and messy inputs than curated benchmark suites.

Second, the Propose-Probe-Compose loop requires multiple execution runs for every candidate harness revision. Each iteration re-runs the full task trajectory. This increases total token consumption and end-to-end latency. The performance gains come with higher inference costs. Teams must balance accuracy improvements against increased token budget.

Third, sandbox environments for tool execution are not bundled with Harness². Users are responsible for configuring and securing isolated execution sandboxes for code and tool calls. Secure sandboxing remains a separate engineering burden.

Fourth, Harness² is a research open-source release, not an officially supported Google Cloud product. Production adopters cannot rely on official Google maintenance, support or service level agreements. Teams need to maintain the framework and handle bug fixes independently.

7. Deployment Considerations for Multi-Model Agent Stacks

When building agent platforms that orchestrate multiple base models and executors, teams need centralized routing, credential control and usage auditing. 4sapi serves as an API gateway that consolidates access to diverse LLM endpoints, standardizes request handling and centralizes consumption logging for composite agent stacks. Systems like Harness², which frequently swap model backends and iterate on harness configurations, benefit from unified gateway management instead of separate integration logic for every model vendor.

This kind of gateway abstraction simplifies experiments where the same harness is evaluated across multiple model-executor pairs. Developers can keep the harness components unchanged while switching underlying models via gateway routing, making benchmark comparison easier to implement.

8. Conclusion

Harness² presents a powerful alternative to weight-based agent optimization. By freezing model parameters and treating the entire surrounding execution scaffolding as the tunable object, it unlocks low-cost, auditable recursive improvement. Its Propose-Probe-Compose loop can refine agent behavior using only execution traces, removing dependence on ground truth labels and test suites. Benchmark data confirms meaningful gains: GLM-5 + OpenCode improves by 11.5 points on WorkBuddy-Bench, with up to 11.8 percentage point gains on Harvey LAB. In this setup, optimized GLM-5 overtakes native Gemini 3.5 Flash.

The broader industry implication is a shift in competitive dynamics. Agent capability is no longer determined solely by base model quality. The evolution speed of the harness, prompt layers, memory and tool orchestration becomes equally critical.

The framework is not a universal solution. Multi-pass execution raises token cost and latency, sandboxing must be implemented separately, and results are validated on paper benchmarks. Still, it offers a compelling path for teams seeking to iterate rapidly on agent behavior without expensive model retraining.