AI said it could save thousands on medical bills, then it checked its math. What went wrong and how to use AI safely
4分で読めます

A Yahoo Tech editor handed two surprise medical bills to an AI chatbot. It found three "huge" errors and wrote her dispute letters and a phone script. Then she asked it to double-check its math, and almost everything fell apart. Here's what happened, what the AI got right, and how to use AI on your own bills without getting burned.
The experiment comes from Danica Creahan, a streaming editor at Yahoo, in a new column called AI, IRL, where Yahoo puts AI to work on everyday problems and reports the honest results. Her verdict: AI can help with the boring parts, but you have to check its work.
Quick facts
- The AI: Anthropic's Claude
- The setup: two procedures quoted at $0, then two real bills arrived
- What she uploaded: both bills and a 168-page insurance benefits document
- Bill 1: $60.30. Claude suggested disputing $14 by certified mail, which costs about $9–12
- Bill 2: just over $2,000. Claude first found three "huge" errors
- After one "check your math": only a $23.46 duplicate charge held up
- Claude's grade for itself: B-

What happened
Creahan says she's not the biggest AI fan, but she likes surprise medical bills even less. After two procedures her provider and hospital calculator had both quoted at $0, two bills arrived anyway.
So she gave Claude everything: the bills plus her full benefits booklet, and asked four questions. Is this bill correct? Are there errors? Do I really need to pay? Can I negotiate?
Bill 1: spend $12 to save $14
On the smaller $60.30 bill, Claude confirmed most charges were right. But it also drafted a long letter disputing an oddly labeled $14 charge, and told her to send it by certified mail with a return receipt.
That costs roughly $9 to $12. In other words, the dispute would cost almost as much as it could win back.

Bill 2: three "huge" errors, then one
The bigger bill, just over $2,000, looked like a win at first. Claude found three discrepancies, with the strongest claim focused on a $399.68 charge. It built comparison charts, drafted letters, wrote a call script and listed billing phone numbers.
Then Creahan typed one simple request, with no new information: "Can you review all of the math in the second procedure to make sure our dispute is correct?"
The math didn't hold up. Claude had built its case on numbers it made up and coverage her plan didn't actually have. What survived was a single duplicate charge of $23.46.

What Claude did get right
It wasn't all bad. According to Creahan, Claude:
- Found the right phone numbers, with extensions, for her insurer, medical facilities and billing departments.
- Spotted that one bill wasn't itemized, something she'd missed.
- Wrote a clear call script, with which questions to ask and what to do if the rep says no.
- Found fine print fast in a long, dense insurance document.
When she asked it to grade itself, Claude gave itself a B-. It admitted that she shouldn't have had to ask "are you sure" twice, and that someone who accepted its first answer would have mailed her insurer a letter citing a state law that doesn't apply to her plan.

How to use AI on your own medical bills
Based on what went wrong, here's a safer way to try it:
- Ask for an itemized bill first. You can't check charges you can't see.
- Make the AI show its math, line by line. Then ask it to check that math again, every time.
- Ask it to quote the exact line from your insurance document that supports each claim, and look that line up yourself.
- Use it for the boring stuff. Finding the right department, the right phone number and the right questions to ask is where it shines.
- Protect your privacy. Remove your member ID, date of birth and other personal details before you upload anything.
- Weigh the cost. Don't spend $12 and an hour on hold to dispute $14.
The bottom line
AI can turn a scary pile of paperwork into a plan. But in a task where confidence costs real money, a chatbot's first answer is a draft, not a fact. Creahan's next plan: use AI to find the right people to call, and do the math herself.
Sources: Yahoo Tech: AI told me I could save thousands on my medical bills. Then I asked it to check its math.