Reliability layers designed to catch errors in language model output sorted answers by trustworthiness but corrected almost none, according to a new study published by participants in the InfoQ Certified Architect Program. The researchers tested six different verification layers—checks wrapped around a language model to flag wrong answers—on the same clinical text records, each one priced. The fundamental theory they set out to test, that these layers stack together to progressively raise accuracy, did not hold. Instead, the layers functioned primarily as filters, withholding questionable answers rather than repairing them.

The study used the CADEC corpus, 1,250 forum posts about two drugs with every adverse reaction marked and coded, to measure performance. The language model being tested was correct about one answer in three on the underlying task and reported full confidence on two-thirds of its answers, regardless of whether they were right or wrong. When the model was asked to recall medical codes from memory, it invented codes that don't exist in any SNOMED release 13 to 18 percent of the time. But when given a menu to pick from—20 nearest concepts retrieved by a separate system—it scored ten times higher. The zero-shot extractor reached an end-to-end F1 score of 0.47 to 0.50 under lenient span matching on the development split, far below the 0.72 achieved by a supervised system fine-tuned on 875 annotated files. Four of five model families tested reproduced their output exactly at temperature zero across three runs; the fifth, a mixture-of-experts model, agreed on 70 percent of mentions.

The authors write that accuracy alone "flatters any withholding layer" because it measures only what passes through, not what's held back. They introduced a second metric, yield—right answers as a share of all records—to expose what accuracy hides. The vocabulary check, which cost zero tokens, changed no answer but gave each one a verdict: reject if the code doesn't exist or the quoted span isn't in the post, accept if the span's words match one of the vocabulary's own names, or band if there's no evidence either way. Self-correction, which sends provable failures back to the model, fired only two or three times per run and corrected nothing. Voting—sampling the model three times and taking the majority—changed about 27 codes per run, as many right-to-wrong as wrong-to-right, for a net effect of minus one to plus one correct answers. According to the report, deterministic code "can prove an answer wrong, never right: it catches nonexistent codes and spans missing from the post, but plausible answers still need a model or a person."

The central problem sits at the detection stage, not in the reliability layers themselves. Nearly half the annotated mentions were never quoted as the annotators marked them—not because of missing domain knowledge, but because of reading errors. Single words went untouched where annotators marked them, boundaries stretched to include extra adjectives, and figures of speech were read literally. Once a mention is found, the pipeline proves reliable: the right concept reaches the menu for 93 percent of found spans, and the model picks it 81 percent of the time. But no layer above the extractor can add a mention it missed in the first place, so the maximum any stack can ever get right is what the extractor found—just over half the annotated mentions on the held-out split, a detection F1 of 0.521. Two domain-adapted models, BioMistral-7B and SapBERT, did not help. A worked example showing the corpus's annotation conventions and a rule for denied reactions did.

The report concludes with six practices that transfer beyond this specific task: read the data before adding a layer, never ask the model for an identifier, anchor confidence in something that doesn't resample like a vocabulary or schema, measure your noise floor with three runs minimum, treat disagreement as a diagnostic to-do list rather than paying to resolve it online, and find the reader of every field a layer writes. The authors tested their findings on seven additional corpora across five model families and found four of the five lessons held, though where the model loses varied—on clinical vocabularies the free check's endorsed lane was right 76 to 100 percent of the time, but on gazetteers the same lane was right only 7 to 26 percent because a place name can match the vocabulary and still be the wrong place. The verdict nesting they observed means each additional verdict mostly confirms the first and can only remove records, never recover one the first held back. The researchers state they set out believing that stacking reliability layers buys reliability, but measured end to end, the layers sorted answers by how far to trust them and fixed almost none—worth a great deal, but not what they bought them for. Organizations deploying these systems face a choice between shipping everything with known error rates or holding back answers for human review, a trade that reliability layers illuminate but cannot eliminate. The real engineering work happens upstream, where reading quality determines the ceiling no downstream layer can breach.