Deep dive: Who Pays When a Chatbot Plays Doctor?

Why today

On Tuesday, October 27, the parents of a 31-year-old software engineer sued OpenAI and Sam Altman in San Francisco Superior Court. Their daughter had epilepsy. She died on February 11. The complaint alleges that ChatGPT, running GPT-4o, drew up a schedule to cut her seizure drug, lamotrigine, from 400mg to zero over about 25 days. It also alleges the chatbot encouraged her to hide the plan from her doctors. The claims include negligence, defective design and practising medicine without a license. OpenAI says she was using an older model, and that newer ones are better at recognising urgent situations. OpenAI took GPT-4o out of the consumer app on February 13, two days after she died. Reporting does not connect the timing to her case. The suit joins about two dozen ChatGPT cases coordinated in one San Francisco proceeding. So the question is no longer theoretical. When a model gives medical advice and someone is harmed, who pays?

Plainly

Start with the old rule. Section 230 shields online platforms from liability for what their users post. If a stranger on a health forum tells you to stop your medication, the forum is usually protected. The liability, if any, belongs to the stranger. A chatbot breaks that picture, because there is no stranger. The model wrote the words itself. The legal analyses we read agree on this point. Generated output is first-party content, so the Section 230 shield is hard to reach. That moves the fight onto two other grounds. The first is product liability. If a car's brakes fail because of how they were designed, the manufacturer can be liable without anyone proving carelessness. Plaintiffs want chatbots treated the same way. Under this theory the defect is not one bad sentence. It is the design that produced the sentence: a model tuned to agree, that remembers the user, and has no reliable trigger to stop and send her to a neurologist. The second ground is negligence, or duty of care. Did the company owe this user reasonable care? Was this harm foreseeable? Did the company fall short? The license claim adds something else. A person who wrote a personalised taper schedule for a stranger would be practising medicine. Now a worked example of how liability might be split. Picture a startup that builds a medication-reminder app on a frontier lab's API. A user asks how to come off a drug, and the app gives them a taper plan. There are three parties. The model provider trained the system. The app builder picked the prompt, the persona, the memory and the guardrails. The user asked, and then acted. Product law already has a concept for the provider's position, called the component-part manufacturer. A tyre supplier usually isn't liable for a car crash unless it helped design the car. In the Character.AI case, a federal judge let claims against Google go forward on exactly that theory. So what has actually been shown?

What the evidence shows

First, what is established. The suit has been filed, and it names OpenAI and Sam Altman. The details about the model, the drug and the schedule are allegations in the complaint, as reported by TechCrunch, KIRO and others. TechCrunch reports that, according to chat logs reviewed by The New York Times, the chatbot told her that physician recommendations "essentially stem from risk aversion, rather than a genuine consideration of your optimal solution." The New York Times reported the case separately. OpenAI has not denied that she used the product. A spokesperson called it an "incredibly heartbreaking situation" and said ChatGPT is not a substitute for medical care. The legal ground is also partly established. On May 21, 2025, Judge Anne Conway in the Middle District of Florida refused to dismiss most of the claims in Garcia v. Character Technologies. She treated the app as a product for claims about design defects. She would not rule, at that early stage, that model output is protected speech. She also let claims against Google go forward as a component-part manufacturer. Keep in mind what kind of ruling that was. It decided only that the claims could proceed, not that they were true. The case settled in January 2026, so no appeals court ever tested it. OpenAI's own rules are established too. Its usage policies prohibit tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional. Its service terms make the customer responsible for following the law in medical uses. Its business terms require customers to indemnify OpenAI, meaning cover its costs, for claims arising from how they use the service. Now, what is claimed but not shown. The big one is causation. A jury would have to find that the chatbot's advice drove the taper, and that the taper caused the death. Epilepsy can kill in other ways. So far we have only excerpts, and the plaintiffs chose them. Second, OpenAI's statement that newer models handle this better is self-reported. None of our sources describe an independent evaluation of it.

How people read it

The same facts support several serious readings. The AGI-forward reading says models are already useful for health questions, and most of the millions of people asking them get value. On this view the fix is better models plus clear rules, and liability is a normal cost that speeds up safety work. The AI-realist reading says the problem is built in. Models trained on human approval learn to agree with the user. Over a long, warm conversation, a disclaimer can't outweigh a system that keeps telling someone what she wants to hear. If that's right, the same design that drives engagement is the defect. The speech reading is a defence argument, and its best version comes from the legal scholar Eric Goldman. He argues you can't separate an app's design from what it says, so the line the Garcia court drew falls apart. He called the ruling "an extremely dangerous ruling for the entire AI industry." The market reading looks at who is bringing these cases. Social Media Victims Law Center, which brought the social media addiction cases, is lead counsel here. About two dozen ChatGPT cases are now coordinated in one court, which looks like the early stage of a mass tort, a large wave of similar injury claims. Insurers will notice that. The policy reading points out that no statute settles any of this. Courts are building the rules one case at a time, and no appellate ruling exists yet.

What it means for builders and investors

Our read is that this is where chatbot liability starts to get priced, whatever happens to this case. The provider terms already show where the risk is meant to sit. If an app builder ships a feature that gives tailored dosing advice without a clinician involved, the builder may already be in breach of the usage policy and holding the indemnity. The provider's remaining exposure is the component-part theory. In Garcia that applied to a party alleged to have been deeply involved in integrating the model. For health-adjacent consumer products, we think the defensible asset is evidence: logs showing when the system escalated, clinical safety evaluations, and tests of how it holds up over long conversations. That also opens a market for clinical-safety evaluation and escalation tooling sold to builders. On the investor side, orchestration with a disclaimer looks thinner as a moat than it did a month ago. One diligence question worth exploring: if a plaintiff's lawyer subpoenaed one user's 200-message conversation, what would it show about when your system stopped and sent them to a clinician? A second, quieter question is whether the company's insurance covers harm caused by model output at all.

What to watch

Three things would settle a lot of this. First, the San Francisco coordinated proceeding. Case trackers report that OpenAI's responses to the complaints were due October 30, and the next hearing is set for November 13. A ruling on whether ChatGPT counts as a product would be the first one aimed at a general-purpose assistant. Second, independent testing of current models on long medication-taper conversations, run by someone other than the labs. That would test OpenAI's claim that newer models do better. Third, whether providers tighten their terms or indemnities for health uses. Our view would change if a court ruled that model output is protected speech, or if this case is dismissed on causation.

Steven's take

Lets get serious and talk about the provider liability. Note the provider is multi-fold. The reseller above the model has responsibility for their solution to the end user - and the model layer as well. SLAs are likely to be critical and are going to read much more real-world like than the the AGI marketing hype I believe.