A US federal regulator recently published a list of practices it considers potentially abusive in one corner of the health insurance market. Buried in it is a line that should interest anyone deploying AI inside their own business.
It names prompts “generated by artificial intelligence algorithms” that encourage clinicians to add diagnoses patients did not have.
Read that again. The regulator is not objecting to the AI. It is objecting to what the AI ethics asks the employee to do.
Most AI governance conversations are about whether a model is accurate. This is a different question, and a harder one. It is about software that sits between your staff and their professional judgment, and quietly leans on it.
The mechanism is duller than it sounds
Nobody built these systems to pressure anyone. Here is how it happens anyway.
The software reviews a patient record, spots something that might support a diagnosis, and generates a question for the doctor. Entirely reasonable. Records are incomplete, and asking for clarification is normal practice.
The trouble starts with how the question is phrased. A prompt that says “the record mentions X and Y, please clarify what you observed” is asking. A prompt that says “this patient appears to have condition Z, please confirm” has supplied the answer and asked for a signature.
Same software. Same clinical situation. Different sentences. One is decision support, and the other is a nudge with a paper trail.
Why does this scale badly?
A human colleague who leans on your judgment does it occasionally, visibly, and you can push back in the moment.
Software does it every time, identically, to everyone, and it looks like process rather than pressure. Nobody in the chain feels they are doing anything unusual. The doctor is answering a question the system asked. The system is asking a question somebody configured two years ago. The person who configured it was optimising for completeness, not steering anyone.
By the time it becomes a pattern, the pattern is thousands of records deep and looks exactly like normal operations.
Four questions that separate asking from leading
These transfer to any workflow where software prompts a person for a judgment call, whether that is a clinician, an underwriter, a loan officer, or a compliance reviewer.
- Does the prompt name the answer? If the system supplies the conclusion and asks for confirmation, the desired outcome is obvious to the person answering.
- Can they say no? If the workflow accepts one answer easily and makes the other awkward, the judgment has been reduced to a formality.
- Does the prompt mention what the answer is worth? Introducing the financial consequence into a professional judgment changes the nature of the question entirely.
- Would the system ask if there was no money in it? This is the revealing one. If prompts cluster around the outcomes that pay and ignore equally important ones that do not, the logic is following revenue rather than evidence.
The part most businesses get wrong
The usual response to all this is an AI ethics policy. Values, principles, a commitment to responsible use.
A policy states an intention. It does not change what the software asks. If your system generates leading prompts, it will keep generating them under a policy that says fairness matters, because the policy and the prompt template are different objects maintained by different people.
There is a sharper version of this in the same regulatory guidance. The AI ethics guidance itself is voluntary and says so plainly. The reporting obligation that sits alongside it is not: organisations must report unsupported entries and the resulting overpayments, under statute and regulation. Your commitment to doing the right thing is optional. Your duty to report what you find is not.
That asymmetry is worth sitting with. It means the risk is not really about intent. An organisation with excellent intentions and software that only ever prompts in one direction is still generating an obligation it may not know it has.
What to actually check?
You do not need to audit the model. You need to read what it says to your staff.
Pull a hundred prompts the system generated last month. Count how many names a conclusion inside the question. That share tells you how the system behaves, and you can run it this week without asking the vendor for anything.
The market response has been architectural rather than procedural. Organisations moving to a risk adjustment platform that tests records against explicit criteria, rather than scoring how closely they resemble a pattern, made the change because an explicit test can come back negative. A scoring system cannot. It has no output that means “no.” Ask your vendor to show you theirs, and watch which of those two things you get.
