Millions of people now ask artificial intelligence chatbots for health advice every day, and major technology companies have taken notice. OpenAI announced ChatGPT Health in January, while Amazon on Tuesday said it was expanding access to its health chatbot, and Microsoft has launched Copilot Health. These tools promise to synthesise medical records, test results and wearable data into personalised guidance, potentially transforming how people navigate healthcare.
The appeal is clear. With healthcare costs rising and appointments hard to access, an always-available AI assistant that can explain lab work and help prepare for doctor visits addresses real gaps in the system. Yet for all their promise, these tools carry significant risks that should give users pause before handing over sensitive medical information.
Where AI Chatbots Can Help (and Where They Fall Short)
The chatbots can summarise and explain complicated test results, help prepare for a doctor's visit or analyse important health trends buried in medical records and app metrics. These are genuinely useful functions, especially for people trying to make sense of complex information before a consultation. Some medical experts encourage patients to use these tools, arguing that with healthcare difficult to afford and access, consulting AI is still often better than the alternatives, and the advice people get from the tools is substantially better than nothing and better than what they would get from their second cousin.
But the gap between promise and practice is substantial. Research published in recent months reveals consistent problems with accuracy and judgment. In 52 per cent of emergency cases, the bots "under-triaged," meaning treated the ailment as less serious than it was. In one example, it failed to direct a hypothetical patient with diabetic ketoacidosis and impending respiratory failure to go to the emergency department. In a documented case, AI advised a patient to try the anti-parasitic drug ivermectin as a treatment for testicular cancer, which probably wouldn't hurt, but what would hurt the patient is not getting appropriate treatment for their cancer that is treatable.
A significant part of the problem lies with how ordinary people interact with these systems. A study published recently in the journal Nature Medicine asked participants to consult AI tools with medical scenarios. After conversing with the bots, participants correctly identified the hypothetical condition only about a third of the time, and only 43 per cent made the correct decision about next steps, such as whether to go to the emergency room or stay home. Doctors are trained to recognise which details are relevant and which can be ignored; the general public is not, which makes it more difficult for them to use AI chatbots for health advice effectively.
The Privacy Trap Nobody Talks About
Many people uploading their medical histories to health chatbots believe they are receiving HIPAA protection, the federal privacy law that governs doctors and hospitals. They are wrong. The law allows for fines and even prison time for doctors, hospitals, insurers or other health services that disclose medical records, but the law doesn't apply to companies that design chatbots.
The difference is crucial. When someone is uploading their medical chart into a large language model, that is very different than handing it to a new doctor, and consumers need to understand that they're completely different privacy standards. While both OpenAI and Anthropic say users' health information is kept separate from other types of data and is subject to additional privacy protections, and the companies do not use health data to train their models, and users must opt in to share their information and can disconnect at any time, researchers have documented how little transparency these commitments actually include.
According to a new study, six leading U.S. companies feed user inputs back into their models to improve capabilities. AI developers' privacy documentation is often unclear, making it difficult for users to understand their data rights. The research stresses the importance of establishing comprehensive federal privacy regulations to protect users' data in AI interactions.
A Pragmatic Path Forward
The honest assessment from medical experts is that AI chatbots work best in limited roles. They are good at guidance, but not so much judgment, and can help you understand next steps, but shouldn't be used for making medical decisions. A good use case is to think through questions to ask your doctor. This approach lets you harness the tool's ability to synthesise information without delegating critical medical judgment to an algorithm.
There are situations when people should skip the chatbot and seek immediate medical attention. Symptoms such as shortness of breath, chest pain or a severe headache could signal a medical emergency. Beyond acute situations, patients and doctors should approach AI programs with a degree of healthy scepticism, and should never be relying just on what you're getting out of a large language model when talking about a major medical decision.
The real-world utility of health AI tools depends on using them with clear eyes about both their capabilities and their limitations. They can help busy people understand their health better and prepare for medical conversations. But they cannot replicate the contextual reasoning, clinical experience and accountability that doctors provide. Until federal privacy frameworks catch up with the technology, and until these systems demonstrate more reliable performance in real-world conditions, treating them as a starting point for further research is far safer than treating them as a substitute for professional medical advice.