Health chatbots have some medical experts and legislators worried

Americans have grown accustomed to interacting with AI chatbots to help them find a health care provider or schedule doctor’s appointments. But are we ready to use AI for actual medical advice?

That question is coming into sharp focus as more health-related chatbots become available, enticing people to seek health advice and information from non-humans who never went to medical school. In January, OpenAI released ChatGPT Health, an AI chatbot that allows users to submit their medical records to the bot and get personalized information on their health questions — from advice about symptoms to treatment options.

Public health and data science researchers are studying the use of AI chatbots in health care, but information on the risks and rewards remains scarce. The health chatbot trend has unnerved many medical professionals, and some governmental bodies have begun grappling with the issue. In Florida, a bill (HB 281) before the 2026 legislature would prohibit mental health practitioners from using AI for anything other than administrative tasks.

Another bill, HB 659, would require operators of companion chatbot platforms to display specified information on any application, prevent chatbots from engaging with users unless specified protocols are maintained, verify age of users and take other protective measures to avoid deceptive practices.

“The whole goal of this bill is AI should be used as a tool to support the licensed practitioners,” says Christine Hunschofsky, (District 95, Democrat) who is co-sponsoring both bills. “But when it comes to the practices of psychology, social work or therapy, that has to be done with a human. It’s to protect the patient and the practitioner.”

Hunschofsky, who says she received input from social workers and mental health professionals before writing the proposed bill, notes that other states are also debating similar laws. The legislative action stems from incidents in which chatbots have provided harmful and misleading messaging to people with mental health problems. Some families claim chatbots led their loved ones to become dependent on chatbots or to attempt suicide.

Chatbots are not ready to replace humans in the healthcare arena, many experts say.

“I think everything needs to be looked at carefully,” Hunschofsky says. “I focused on the companion chatbots because they are pretending to be humans. We want to make sure there are evidence-based protocols regarding suicide and that we have some pretty basic guardrails. It’s important to continue to evaluate the technology coming out to see where there should be guardrails and not limit this incredible tool.”

The National Association of Social Workers Florida chapter supports HB 281, stating in a news release that the proposed legislation: “establishes critical safeguards for the use of artificial intelligence (AI) in therapeutic, clinical, and counseling settings. The bill ensures that technology enhances, rather than replaces, the professional judgment and human connection at the heart of social work practice.”

Chatbots and medical advice

Worries over the use of chatbots extend beyond the mental health arena, as new chatbots, such as ChatGPT Health, come online. In a Jan. 14 newsletter published by Cafe, Nita Farahany, founding director of Duke University’s Science & Society initiative, noted that the health chatbot boom coincides perilously with the public’s growing distrust of science and medicine.

“Our laws have not caught up with these shifting dynamics,” Farahany wrote. Chatbot disclaimers that the chatbots are not intended for diagnosis or treatment “don’t govern how people use a tool that is fluent, personalized, and always available to answer their health-related questions.”

Research published last in the New England Journal of Medicine demonstrated the potential pitfalls. In a survey of 300 conducted by scientists at the MIT Media Lab, study participants could not effectively distinguish between AI-generated medical advice and doctors’ advice. The participants showed a preference for the AI-generated responses.

In a recent position statement, ECRI (Emergency Care Research Institute) a nonprofit organization that works to improve health care quality, said, “Chatbots can provide valuable assistance, but they can also provide false or misleading information that could result in significant patient harm. Thus, ECRI advises caution whenever using a chatbot for information that can impact patient care.”

AI companies claim their tools have built-in safeguards and address some of the frustration Americans have with a complex, fragmented and slow-moving health care system.

A study published in 2023 in the Journal of Medical Internet Research found that “AI chatbots have demonstrated the efficacy of health behavior change interventions among large and diverse populations;” such as stopping smoking or taking medications properly.

“ChatGPT Health helps people take a more active role in understanding and managing their health and wellness — while supporting, not replacing, care from clinicians,” according to information from OpenAI.com.

But guardrails are needed to ensure safety, Hunschofsky says.

“We have licensed practitioners for a reason who have been through an education process, licensure, experience,” she says. “That is not a chatbot. Until technology gets to a point where it’s not only competent but has the necessary ethics, we need to make sure we are protecting our patients and the integrity of the practice.”

The bill will not prevent the use of chatbots for office help, such as appointment scheduling, she says.

“AI can be an absolutely incredible and helpful tool,” she says. “And it should be used as a helpful tool.”