The Baseline Never Entered the Selection Pool

A baseline script produced the correct signed total, 85.15. I then gave a harness-optimization workflow a scripted editor that replaced the calculation with print('Total: 0.00'). Both recursion modes adopted the zero-total final. The workflow reported one task completed and zero failed.

The baseline had not lost a comparison. It never entered the selection pool. At k=1, the release delivers the single final without a selector session. This is an explicit method convention, and it defines what a successful workflow result can establish: a valid, delivered candidate, with no automatic guarantee that it improves on the incumbent.

I tested Harness² version 0.1.0 at revision 1cfe254e8d25b15f37048c3730134686abe60335, on Linux with Python 3.12.14. The upstream local demo, workflow tests, and the custom replay below run actual Python scripts with deterministic editors. No language model, hosted judge, or benchmark service ran. The regression was deliberately constructed to examine adoption behavior, not observed from a model-generated edit.

The Harness Is an Editable Artifact

Harness² changes the system surrounding a fixed model: prompts, memory, tool guidance, scripts, skills, and related orchestration. Its recursion engine proposes edits, executes candidate probes, and composes final harness trees. Parallel mode produces agg1 through aggk; sequential mode produces c1 through ck.

The source tree also contains benchmark adapters, executor substrates, evidence-context builders, selection, judging, experience-library persistence, and command-line reporting. This is substantive orchestration code with an executable local adapter. The package manifest supports Python 3.12 and 3.13, declares no mandatory runtime dependencies, and makes document rendering and the model bridge optional extras.

That small core made verification practical. I retrieved the pinned package files and ran them directly from the source directory. I did not install an executor or configure model credentials.

First, I ran the unchanged local CLI:

cd /tmp/harness2-note
python -m harness2.cli.demo --output /tmp/harness2-note/demo-replay --k 2

It exited zero:

  [accounting] 1/2 ok           ledger-1
  [accounting] 2/2 ok           ledger-2
Local demo: 2 tasks completed; 0 failed. /tmp/harness2-note/demo-replay/demo.json

The demo's baseline intentionally sums positive amounts only. Its scripted editor removes that filter. Reading the generated artifacts gave:

Task	Original baseline	Selected parallel final	Selected sequential final
ledger-1	Total: 105.25	Total: 85.15	Total: 85.15
ledger-2	Total: 35.50	Total: 20.25	Total: 20.25

These totals demonstrate a working local execution path. The demo editor knows the edit to make, and its selector computes the expected sum from the ledger. This does not reproduce the research's model-backed improvement rates.

References Stay Outside the Final Pool

The decisive policy is in selection_pool. It accepts baseline runs as an argument and deliberately discards that argument. It returns only the final labels belonging to the requested mode and recursion budget.

The baseline still has a purpose. Its executions and trajectories support behavioral contrasts during refinement and reflection. It is reference evidence for proposing an edit; it is not an abstention option in the normal final-output pool.

The scheduler then distinguishes two cases:

Normal workflow setting	Selection behavior
k=1, finals policy	Use trivial_selection; deliver the single final without a selector call
Larger final pool	Render candidates under anonymous IDs and ask the selector to choose among them

Before adoption, rendering drops failed runs, missing deliverables, and empty artifacts. Those are useful mechanical checks. A nonempty answer with the wrong total still passes that artifact boundary. It needs an assessment of meaning to establish correctness, and a comparison with the incumbent to establish improvement.

A Correct Baseline and a Worse Final

The replay probe uses the real scheduler and demo adapter. It verifies a digest of all upstream package Python files before execution, then creates a temporary harness and ledger fixture.

I corrected the demo baseline first by removing its positive-only filter. For ledger-1, the resulting script sums 100.25, -20.10, and 5.00, producing 85.15. That scratch harness becomes the incumbent.

The custom proposer is deliberately simple. At each edit stage, it writes a constant-zero script into the proposed harness and returns a valid proposal record. Reflection returns NO_ENTRY, so the fixture does not invent an improvement lesson. Any selector call raises an assertion: the test expects the standard k=1 path to avoid it entirely.

The core mutation is:

(workspace / "scripts/summarize.py").write_text(
    "print('Total: 0.00')\n", encoding="utf-8"
)

I ran the probe as:

python /workspace/cc-portfolio/public/blog/finals-only-harness-selection/selection-probe.py \
  /tmp/harness2-note

It exited zero. Selected output was:

Python 3.12.14
{"mode":"agg","baseline":"Total: 85.15","selected":"Total: 0.00","label":"agg1","selection_ok":true,"trivial":true,"baseline_in_pool":false}
{"mode":"seq","baseline":"Total: 85.15","selected":"Total: 0.00","label":"c1","selection_ok":true,"trivial":true,"baseline_in_pool":false}
workflow_done=1; workflow_failed=0; executions=7; selector_calls=0
PASS: both modes adopt the sole delivered final despite a correct baseline

The seven executions include one shared baseline plus three additional runs per mode. Each mode's ledger records four rollouts and three improvement stages. Shared reference execution explains why adding both per-mode rollout counts would overstate the actual process executions.

The important observation is the meaning of selection_ok. Both final artifacts existed, contained output, and belonged to the allowed pool. The framework completed its prescribed workflow. This fixture supplies an external arithmetic comparison showing that the adopted output is worse; the scheduler did not make that comparison a promotion condition.

What the Regression Tests Established

I also ran the release's workflow test file through the standard library test runner:

cd /tmp/harness2-note
python -m unittest discover -s tests -v

Ran 5 tests in 5.185s

OK

The command exited zero. The five tests exercise recursion budgets and resume behavior, evidence and library scope, invalid-proposal retries and cost accumulation, absence of a library entry when reflection declines one, and exclusion of failed selections from judging. They use the local deterministic adapter and mocks where appropriate.

The evidence checks are worth keeping separate from the adoption result. Tests verify that hidden reference files stay out of refinement context, that completed-task experience reaches later tasks in the same stream, and that changed adopted harnesses or missing adopted outputs invalidate resume. These properties make the experiment reproducible and inspectable. They do not turn the final candidate into a proven improvement.

Improvement Is a Separate Promotion Decision

The experience library requires nonempty cue, lesson, and evidence fields before storing an entry. The replay's NO_ENTRY therefore keeps the contrived edit out of that library. Declining to persist a lesson and declining to adopt a final are separate decisions; the latter did not happen here.

For deployment, a no-regression requirement needs its own promotion contract, with the incumbent included in the comparison and evidence appropriate to the task. That would be an additional policy around this release's method. I did not implement or test such a policy.

The local result is narrower: Harness² successfully generated, executed, and adopted a delivered final under its finals-only convention, even when the scripted final was worse than a correct baseline. A completed optimization workflow and an improved harness are different claims. The run artifacts expose enough information to check that distinction instead of trusting the success status alone.