Repeatable is not the same as correct

An earlier Ackren candidate replayed all 716 recorded turns identically, yet qualified on only 94 of 120 scored episodes. It passed 14 of 40 meaning-contrast episodes. These are counts from an exposed, agent-authored regression set, not estimates of performance with people. The candidate failed its overall and contrast qualification thresholds.

This is a useful result to publish because the successful replay and the unsuccessful language evaluation concern different properties. A system can follow the same steps every time and still misunderstand the sentence it was given. Repeatability makes a failure easier to reproduce; it does not make the answer true.

Sources: Recorded Ackren evaluation: counts, scope and source hashes

The recorded run contains 476 scored checkpoints and 240 diagnostic turns. All 716 turns passed the structural explanation audit. That audit checked the shape of the explanation and its public record and source references. It did not establish that every explanatory sentence was faithful, that the English interpretation was correct, or that a person would find the conversation useful.

The contrast cases are especially revealing. They change meaning while keeping much of the wording similar. The candidate qualified on 40 of 40 paraphrase episodes, but only 14 of 40 contrast episodes. A parser that tolerates alternate wording must also preserve consequential differences between statements. Success on the former does not establish the latter.

Sources: Recorded Ackren evaluation: counts, scope and source hashes

The fixtures had been exposed during development, and the evaluator itself required repairs. The final figures describe regression performance on that material. They are not a fresh generalisation test. No human participants took part. Earlier partial evaluator scores must not be presented as a clean before-and-after improvement curve.

These results belong to the frozen candidate identified in the evidence note, not to every Ackren build. The newer compositional prototype has different evidence and outstanding qualification work. Combining their strongest numbers would create a result that no single candidate achieved.

Sources: Recorded Ackren evaluation: counts, scope and source hashes · Recorded prototype example and development checks

Our editorial conclusion is that a convincing demonstration should expose the interpretation as well as the answer. Ask what the system understood, which premises it used, whether a changed premise changes the result appropriately, and how it responds when the wording is outside its coverage.

A stronger follow-up study would freeze the implementation and evaluator before independent people supply unfamiliar examples. It would assess intended meaning and explanation usefulness separately from replay. That is proposed work. The existing record supports an inspectable engineering case study, not a claim that Ackren has solved general conversation.

Sources: Recorded Ackren evaluation: counts, scope and source hashes · Recorded prototype example and development checks