HarnessTTT · Self-Adaptive Harness Learning for Test-Time Discovery

Results

A frozen 9B executor, pushed by a learned harness.

With a frozen Qwen3.5-9B executor, HarnessTTT leads the compared open-weight systems of up to 10B parameters on nine of eleven tasks and improves on Finch-9B on all eleven.

9 / 11

Best among open-weight ≤10B

Executor-update systems on similar backbones reach at most two of eleven.

0.921

Hadamard

Up from 0.481 for Finch-9B and 0.576 for the next-best open-weight system.

2.635983

Circle packing (n = 26)

Matches the best value reported for systems of this size, starting from 0.960.

OPEN-WEIGHT COMPARISON · NATIVE UNITS

Systems up to 10B parameters

Method	Erdős ↓	AC1 ↓	AC2 ↑	CP26 ↑	Hadamard ↑	AHC039 ↑	AHC058 (M) ↑	EPLB ↑	PRISM ↑	LLM-SQL ↑	Txn ↑	Best
Initial program	0.495056	1.518600	0.855800	0.959764	0.143275	534,850	0.00	0.1265	21.89	0.6856	2824.86	–
OpenEvolve, proprietary executors (reference only)
Claude Opus 4.6	0.381880	–	–	2.629300	–	–	–	0.1270	26.26	0.7160	3774.00	–
Gemini 3 Pro	–	–	–	2.541400	–	–	–	0.1272	26.24	0.7258	4273.50	–
GPT-5	–	–	–	2.541400	–	–	–	0.1272	26.23	0.7155	4237.30	–
Open-weight executors (≤10B)
Qwen3.5-2B	0.381737	1.518600	0.864600	1.253056	0.478009	546,078	0.00	0.1265	21.89	0.6856	2832.86	0/11
Finch-2B	0.381346	1.518400	0.892000	1.535134	0.400476	545,256	329.36	0.1269	22.26	0.6860	2949.85	0/11
Qwen3.5-4B	0.416924	1.518600	0.880200	1.680787	0.384332	542,077	0.00	0.1266	21.89	0.6856	2732.24	0/11
Finch-4B	0.386460	1.517300	0.893300	1.806808	0.146199	551,844	331.47	0.1267	22.87	0.6857	4761.90	1/11
Qwen3.5-9B	0.385512	1.518600	0.880100	1.172702	0.397184	553,582	134.49	0.1269	22.36	0.6858	3584.23	0/11
Finch-9B	0.381100	1.514100	0.912200	1.936000	0.480585	553,759	525.29	0.1265	23.93	0.7024	3636.36	0/11
TTT-Discover^† Qwen3-8B	0.380932	1.505250	0.947200	2.635983	–	–	–	–	–	–	–	2/11
ThetaEvolve^† R1-Qwen3-8B	–	1.506803	0.946900	2.635983	0.576400	–	–	–	–	–	–	1/11
Nano-Discover^† Qwen3-8B	0.380935	1.505395	0.945766	2.635983	–	–	–	–	–	–	–	1/11
Ours · frozen Qwen3.5-9B executor
HarnessTTT-9B	0.380919	1.504680	0.933900	2.635983	0.921053	559,534	713.55	0.1272	26.26	0.7415	4255.32	9/11
Δ vs. Finch-9B (%)	+0.05	+0.62	+2.38	+36.2	+91.7	+1.04	+35.8	+0.55	+9.74	+5.57	+17.0	–

Bold: best open-weight value per task; underline: second. Proprietary rows are context only. ^†Obtained with a different evolution pipeline and shown for reference only. AHC058 is in millions. The last row is the relative change over Finch-9B in each metric’s direction.

The idea

Move test-time learning one level outward.

Test-time RL writes the reward into the executor’s weights. Frozen-executor systems keep a better artifact but leave their proposal policy fixed. HarnessTTT trains that policy.

01

Propose an executable harness

The proposer edits a complete harness: system prompt, skills, generated tool code, middleware, and control parameters. Every generated tool passes static checks and a sandboxed self-test before it runs.

02

Score it with a frozen executor

Each harness gets one rollout of the frozen executor under a fixed evaluator budget. Its reward is the gain over the incumbent, normalized by the remaining gap to a reference score.

03

Update only the proposer

A leave-one-out baseline with max-sharpening turns the K scores of a group into advantages for a LoRA update of the proposer. The best program and harness carry over to the next round.

HarnessTTT method: a trainable proposer writes K harnesses, a frozen executor rolls out each one, and a group reward updates only the proposer — **The HarnessTTT learning loop.** The proposer samples K harnesses from a shared starting state. The frozen executor rolls out each harness once, and a group reward updates only the proposer. The best solution, the selected harness, and a fixed-size experience digest are retained for the next round.

Weak-to-strong transfer

A harness learned on a small model lifts a stronger one.

The learned harness is explicit and separate from the executor’s weights. We take the harnesses learned with Qwen3.5-9B and Qwen3.8-27B, switch the executor to GPT-5.5, and continue the search.

0.9636

Autocorrelation II

Reported frontier: 0.9629.

571,450

AHC039

Reported frontier: 567,503.

850.80M

AHC058

Reported frontier: 849.33M.

3.430204

V2 (lower is better)

Reported frontier: 3.430214.

STRONGEST REPORTED SYSTEMS · ANY BACKBONE

Against the reported frontier

	Reported systems										HarnessTTT (ours)
Task	OpenEvolvemultiple	AlphaEvolveGemini 2.0	HyraHy3	SimpleTESgpt-oss-120B	TTT-Discovergpt-oss-120B	ShinkaEvolvemultiple	CORALOpus 4.6	ALE-AgentGemini / GPT	GEPAGPT-5 / Gem. 3	LEVImixed	9BQwen3.5-9B	9Btransfer → GPT-5.5	27BQwen3.8-27B	27Btransfer → GPT-5.5
Mathematical discovery
Erdős ↓	0.380910	0.380924	0.380859	0.380868	0.380875	0.38156	0.38089	–	–	–	0.380919	0.380923	0.380902	0.380868
AC1 ↓	1.506604	1.503170	1.502850	1.503871	1.502863	–	–	–	–	–	1.504680	1.506805	1.503778	1.503871
AC2 ↑	0.944900	0.961020	0.962901	0.962694	0.959180	–	–	–	–	–	0.933900	0.934210	0.934868	0.963589
CP26 ↑	2.635983	2.635983	2.635983	2.635983	–	2.635983	2.6360	–	–	–	2.635983	2.635983	2.635983	2.635983
Hadamard ↑	–	–	–	0.935673	–	–	–	–	–	–	0.921053	0.935673	0.806604	0.935673
Algorithm engineering
AHC039 ↑	–	–	–	567,503	567,062	558,026	–	550,647	–	–	559,534	571,450	564,457	569,409
AHC058 (M) ↑	–	–	–	849.33	848.41	–	–	848.37	–	–	713.55	849.68	841.39	850.80
System performance
EPLB ↑	0.1272	–	–	–	–	0.1290	0.1490	–	0.1445	0.1523	0.1272	0.1294	0.1285	0.1288
PRISM ↑	26.26	–	–	–	–	26.26	26.26	–	26.23	26.26	26.26	26.26	26.26	26.26
LLM-SQL ↑	0.7258	–	–	–	–	0.7240	0.7310	–	0.7134	0.7985	0.7415	0.7375	0.9676	0.9672
Txn ↑	4,273.50	–	–	–	–	4,329.00	4,566.00	–	3,984.10	4,464.29	4,255.32	4,651.16	4,370.00	4,273.50
Astrodynamics and biology
V2 ↓	–	–	–	3.430214	–	–	–	–	–	–	3.446415	3.432584	3.430206	3.430204
scRNA ↑	0.710	–	–	0.740	0.730	–	–	–	–	–	0.709	0.740	0.714	0.736

Bold: state of the art per task; underline: second best. For each HarnessTTT size, the first column searches with its own frozen executor and the second moves the learned harness to GPT-5.5 and continues the search. HarnessTTT matches or exceeds the frontier on eight of thirteen tasks, is within 1% on four more (Erdős, AC1, CP26, scRNA), and is behind on EPLB. LLM-SQL and Txn are quarantined in the paper, which leaves six. Because the executor changes and the search continues, these gains do not isolate a pure transfer effect.

Cross-executor replay: custom-tool calls and mean normalized score with the initial and the evolved harness on five frozen executors — **Replaying a fixed harness on other executors.** The harness selected on Qwen3.5-9B is replayed unchanged on five frozen executors. Left: calls to the harness’s generated tools per 100 turns. Right: mean normalized score of an OpenEvolve replay with the initial and the evolved harness; the evolved harness scores higher on all five.

Ablations

Where the reward goes, how wide to search, how to score.

Routing the rollout reward to the proposer’s weights reaches the highest endpoint, and reaches it sooner, than routing it to the proposer’s context or to the executor’s weights.

Three ablations: reward routing, number of proposals per round, and the reward objective — **Three ablations.** (a) Reward routing on Erdős and autocorrelation II: normalized score against total rollouts spent; the open marker is a held value, not a measurement. (b) Best score over 15 rounds on circle packing and Hadamard for K = 2 to 32 proposals per round; round cost grows with K. (c) Proposer objective, normalized per panel; the RLOO curve is the K = 32 run in (b).

ADAPTATION STAGES AND MODEL SCALE · NATIVE UNITS

What each stage of proposer learning adds

Condition	Erdős ↓	AC1 ↓	AC2 ↑	CP26 ↑	Hadamard ↑	AHC039 ↑	AHC058 (M) ↑	EPLB ↑	PRISM ↑	LLM-SQL ↑	Txn ↑	V2 ↓	scRNA ↑
Initial harness 9B	0.456591	1.518200	0.896100	1.477767	0.360961	557,225	134.49	0.1265	24.02	0.0934	3610.11	3.839529	0.641
Context update 9B	0.395849	1.509800	0.926300	2.370463	0.517212	559,534	562.82	0.1278	23.04	0.7267	3906.25	3.458023	0.701
vs. initial (%)	+13.3	+0.55	+3.37	+60.4	+43.3	+0.41	+318.5	+1.03	−4.08	+678.1	+8.20	+9.94	+9.36
Weight update 9B	0.380919	1.504680	0.933900	2.635983	0.921053	559,534	713.55	0.1272	26.26	0.7415	4255.32	3.446415	0.709
vs. context (%)	+3.77	+0.34	+0.82	+11.2	+78.1	0.00	+26.8	−0.47	+14.0	+2.04	+8.94	+0.34	+1.14
Weight update 27B	0.380902	1.503778	0.934868	2.635983	0.806604	564,457	841.39	0.1285	26.26	0.9676	4370.00	3.430206	0.714
vs. 9B weight (%)	+0.004	+0.06	+0.10	0.00	−12.4	+0.88	+17.9	+1.02	0.00	+30.5	+2.69	+0.47	+0.71

Bold and underline: best and second best of the four conditions. Grey rows give the relative change over the row above, in each metric’s direction. In this table, in-context editing improves on the initial harness on twelve of thirteen tasks, and training the proposer’s weights improves further on eleven. The 27B campaign samples 8 harnesses per round instead of 32 and uses 5 evaluator calls per rollout instead of 20; it also uses revised Hadamard, LLM-SQL, and AHC evaluators, so those columns mix model scale with evaluator changes.

Search from different initial programs: autocorrelation II from two seeds, and search continued from strong programs on autocorrelation II and Erdős — **Search from different initial programs.** (a) Autocorrelation II from the default seed and from the ThetaEvolve StrongInit program. (b) Search continued from already-strong programs. Curves show the incumbent score after each outer round; round 0 is the initial program. Each curve is a single run.

Discovered harnesses

What the proposer left behind.

Two final harnesses from the 27B campaign. Each tab shows how the harness was assembled round by round, and the prompt policy, tool code, and skill playbooks the proposer wrote.

FINAL HARNESS

Executable components

compiled & validated

——

EXECUTABLE AGENT PACKAGE

Harness H_k

compiled & valid

mount authorityagent.yamlbinds every component below

01prompt.mdstrategy & workflow

02tools/schemas & generated code

03skills/task playbooks

04middlewares/automatic runtime control

05samplingtemperature, top-p, max tokens

06control policybudget, stalls & restarts

validate→runnable agent→frozen executor

READING THE NUMBERS

What these results do and do not show.

The paper reports the selected outcome of each search campaign and states where a comparison is not clean.

Single campaigns. Tables report one selected run per task, not a mean over independent searches.
Quarantined tasks. LLM-SQL and Txn are excluded from the paper’s frontier claims.
Transfer is not isolated. Switching to GPT-5.5 changes the executor and adds search.
Controls describe the protocol. The released artifact does not yet include the trusted runtime needed to verify the validation gates independently.

Back to the results →