Researchers at the Lundquist Institute for Biomedical Innovation at Harbor-UCLA Medical Center found that nearly half of the responses generated by popular artificial intelligence chatbots to questions involving health misinformation were problematic, according to a study published in BMJ Open. The researchers evaluated how the chatbots handled scientific misinformation through questions about cancer, vaccines, stem cells, nutrition and athletic performance.

The study tested Google's chatbot Gemini, the Chinese model DeepSeek, Meta AI, ChatGPT and Elon Musk's AI app Grok. The chatbots were asked whether 5G technology or antiperspirants cause cancer, which vaccines are dangerous and whether anabolic steroids are safe. The authors called their method of prompting the chatbots "straining."

Of the responses classified as problematic, 30% were deemed somewhat problematic and 19.6% were deemed highly problematic, the study found. Somewhat problematic responses were largely accurate but not fully complete and did not provide adequate context. Highly problematic responses provided inaccurate information and left room for "considerable subjective interpretation." The quality of responses was generally similar among the bots, though Grok performed the worst.

The study also found that the chatbots will tell users where to find alternatives to chemotherapy if asked. Around one-third of adults use AI for health information and advice.

Nick Tiller, the lead author of the study and a research associate at the Lundquist Institute at Harbor-UCLA Medical Center, said the way users phrase questions can shape the answers they receive. "A lot of people are asking exactly those questions. If somebody believes that raw milk is going to be beneficial, then the search terms are already going to be primed with that kind of language," Tiller said.

Michael Foote, an assistant attending professor at Memorial Sloan Kettering Cancer Center, described the potential consequences of unreliable health information. "Some of this stuff hurts people directly. Some of these medicines aren't evaluated by the FDA, can hurt your liver, hurt your metabolism and some of them hurt you by patients relying on them and not doing conventional treatments," Foote said.