bluecollarEvaluation
An agent is a model and a harness together, so a benchmark score alone does not say which half earned it. bench holds the model, the task and the verifier…
An agent is a model and a harness together, so a benchmark score alone does not say which half earned it. bench holds the model, the task and the verifier fixed and swaps only the harness.
Run metrics
bench.RunMetrics is derived from the event ledger a turn already writes, so a measured run is the run that ran. bench.Runner drives tasks through any agentcontract.Harness and asks the benchmark's own verifier; a task nobody checked stays unverified.
| what it reports | why |
|---|---|
| prompt tokens per turn | what a harness puts in front of the model is what a pass rate hides |
| cached prompt tokens | two harnesses can send the same bytes and pay very differently |
| turns, tool calls, failed tool calls | how much work it took |
| approval holds | held calls are counted apart from failures |
| recovery attempts | whether it recovered or thrashed |
| cost per passed task | a harness that burns money failing is not the cheap one |
| wall clock, model latency | how much of the wait was the model |
Terminal-Bench
bench/terminalbench/README.md puts cmd/bluecollar on Terminal-Bench, reaching each task container through docker exec, and compares it with another open-source loop on the same model, tasks and verifier. It is a running log of every sweep. Its pass rates are not terminal-bench-core scores, because both harnesses run with raised timeouts.
The column the write-up keeps returning to is failure reporting: across 188 runs the grader failed, bluecollar told the requester it could not finish in 91; across 30 failed runs of the other loop, in none.
Where the budgets come from
bench/derive-budgets reads the step and tool call distributions off runs that succeeded. The first working tier is their 95th percentile and each tier doubles, and loop/task_level_profile_test.go fails when the first tier drifts from that percentile. The measurement came from container coding tasks, and a thin sample of successes, so it is a starting point to re-derive from product runs.
Prompt budget
loop/testdata/prompt-budget.json records what two fixture turns assemble, and maximumTotalBytes (16 KiB) is the ceiling. Any change that grows or shrinks what the model reads fails the test until the file is updated, so the difference lands in the diff a reviewer reads. Today the fixtures assemble 11.4 KB and 9.7 KB, and the largest item in each is the action schema, larger than the instruction and the tool catalog together. The test needs no credentials and calls no model.
Guarantees as tests
The loop's guarantees are written as tests, so their names are the specification:
go test -run 'Checkpoint|Resume|Approval' -v ./loopCI runs gofmt, go vet, go build and go test, then the same inside the ACP module, with no network, credentials or database. Live model evaluations sit behind the llmeval build tag and never run by default.