Most organisations have a quality function. Someone reviews the work, catches errors, signs things off. That is normal and it mostly works.
Fewer have an answer to the next question. Who reviews the reviewer, and against what standard, and how would anyone know if the checking itself had drifted?
American healthcare has run that experiment at national scale, and the results are public, because the checker in that system is a federal agency that publishes what it finds.
Three layers, each checking the one below
In US Medicare Advantage, insurers are paid more for covering sicker patients. Illness is established by diagnosis codes drawn from clinical notes.
Layer one is the clinician, writing the note. Layer two is the software and reviewers who check those notes and decide which diagnoses to submit. Layer three is the federal audit that checks layer two.
The interesting finding is not about layer one. It is that layer two, the checking function, was doing something structurally incomplete, and nobody inside the system could see it because the checking looked like it was working.
What the audit found
The US Office of Inspector General published its detection method in December 2023: a toolkit setting out the selection logic, including the database queries it uses to flag diagnosis codes at high risk of lacking support. Then it ran it.
Across its audit work, roughly 70 percent of those codes were not supported by the associated medical records, with some categories above 90 percent. One national audit sampled 97 records carrying an acute stroke diagnosis. None were supported as acute strokes. In 68 of those cases the patient genuinely had had a stroke, recorded in their history, with the submitted code describing it as active.
Thirty months sat between the method being published and the audit landing. Any organisation could have run the same check on itself. Almost none did.
Why the checking function could not catch itself
This is the part that transfers to any organisation with a quality process.
The software doing layer two was built to find diagnoses that had been missed, because finding more generated revenue. It predicts: given a note, estimate which codes probably apply, rank by confidence.
That output has no slot for a negative. A ranked list cannot contain an entry meaning “this existing diagnosis is unsupported, remove it.” You can score something low. You cannot mark it wrong.
So the checking function only ever added. Run it a hundred times and nothing comes out, not because anyone configured it badly, but because there was no output for a removal. It looked like quality control and functioned as a completeness check.
Three questions for your own review process
- Can your check fail something, or only flag what is missing? If it has no negative output, it is not a control, whatever it is called internally.
- Is there a standard your reviewers are measured against that they did not write themselves? A checking function evaluated on its own terms will always look adequate.
- Could an outsider run your check and get the same answer? If the method is not written down clearly enough for that, you cannot verify it and neither can anyone else.
What changed in the healthcare case
Organisations in that market have been rebuilding the checking layer so it can return a negative result, and preparing for the external check on the assumption that it is coming. The term for that external check is risk adjustment data validation, and the practical shift has been from reviewing to prove things were found, to reviewing to prove things hold up.
The useful detail for anyone else: the regulator published the standard before it enforced it. Most organisations never get that courtesy, which makes writing down your own checking standard a cheaper exercise than waiting to be handed one.
