HarnessDev: Can AI Agents Build and Optimize Their Own Infrastructure?

An AI agent's performance depends on more than its underlying language model. Tool execution, context management, error recovery, and orchestration all influence whether it can complete a task reliably.

The paper HarnessDev investigates whether large language models (LLMs) can build and iteratively improve this infrastructure, known as an agent harness, using execution feedback.

What Is an Agent Harness?

An agent harness is the runtime software that connects an LLM to its working environment. It typically manages tool calls, context construction, execution loops, error handling, and task verification.

For example, a coding agent may generate a correct patch but fail to run tests or interpret compiler errors. A better harness can enforce a verification step, return structured error output, and allow the model to retry.

The model's weights remain unchanged; the surrounding execution system changes how effectively it uses the model's capabilities.

How HarnessDev Works

HarnessDev evaluates harness development through two stages.

1. Creation

The model starts with a minimal seed implementation and a small set of examples. It must construct a runnable harness for a target domain rather than simply solve individual tasks.

2. Evolution

The model modifies its previously generated harness using downstream execution feedback. The objective is to improve task performance through iterative changes to the execution infrastructure.

The benchmark measures two primary metrics:

• Capability: Task success on held-out benchmarks.

• Efficiency: Execution-token cost.

The Creation experiments cover six creator LLMs, four domains, five downstream benchmarks, and 2,207 unique task instances. Evaluation tasks are withheld from development to test generalization.

Key Findings

The results show that automatically generated harnesses are not equally effective across domains.

• Code and research: Generated harnesses lag behind mature, human-engineered systems.

• Writing and ML experimentation: Some generated harnesses match or exceed the selected reference systems.

• Iterative optimization: Evolution produces gains, but improvements are unstable and transfer only partially to unseen tasks.

• Cross-model transfer: Performance gains depend significantly on the runtime model executing the harness.

These findings suggest that harness quality is domain-dependent and model-dependent. A workflow optimized for one model cannot be assumed to perform equally well with another.

Why Does This Matter for AI Engineering?

HarnessDev shifts part of the optimization problem from model capability to system design.

When an agent fails, the cause may be inadequate reasoning, but it may also be poor tool orchestration, missing context, weak error recovery, or insufficient verification. Improving the harness can address some of these failures without retraining the underlying model.

However, automated harness optimization introduces its own challenges. A modification may improve development benchmarks while degrading performance on unfamiliar tasks. It can also increase token consumption or introduce regressions into previously working workflows.

A practical implementation should therefore use held-out evaluation sets, regression tests, execution-cost tracking, versioned harnesses, and controlled deployment.

Conclusion

HarnessDev demonstrates a promising direction for agent engineering: allowing LLMs to construct and refine the software infrastructure that governs their execution.

But generating a working harness is not the same as producing a consistently better one. Unstable optimization gains and limited transfer across models remain significant challenges.

The central engineering question is no longer just how capable a model is, but how effectively the complete agent system converts that capability into reliable task execution.

Paper: HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?