This is my exact concern, patients will feel compelled to say they consent even if they do not, whether it is because of politeness or deferring to authority
When Rebecca Green arrived for her first appointment with a urologist, she was asked whether she consented to having the appointment transcribed by artificial intelligence (AI).
She said yes without hesitation.
“I don’t want to be in her bad books or be that annoying patient who takes up more of her time,” she said.
From what I got talking to friends who work in the field, the problem starts at this point
Insurance and bureaucratic pressures have increased the number and size of notes that doctors are expected to create. Then, instead of hiring scribes or compensating doctors appropriately for taking notes, they are expected to meet those ever increasing requirements with the limited time that they have.
We already had audio processing “AI”, and doctors have been using them for years. That’s actually one area where the recent advances around open models have really helped, because now it’s cheaper and easier to run those systems locally and doctors (or scribes) can document faster than typing.
IMO it should be the administrative body that should be apologizing for this one.
The part I disagree with is this
This would be fair if we were comparing typed notes to speach-to-text notes. However when talking about these LLM systems, the type of error that a doctor makes is different from the types of errors an LLM makes, and so the scale of the damage can be much higher. A doctor isn’t going to hallucinate new conditions, or make up that a patient is being abused (or abusing a child) out of nowhere. I don’t have a source on the later, but it’s something I remember from a research article that was shared here in the past. That kind of thing can have severe consequences on a person because doctors are often relied on as experts to determine how someone should be treated by other systems (civil/legal etc).
Also, human errors are generally predictable and so we can create some safeguards against them. For example, writing 5mg instead of 5.0 mg in hand written charts, double checking numbers during shift changes, etc. It’s another thing to try and account for unpredictable hallucinations.