When Utah became the first state to let an artificial intelligence system prescribe routine medication refills last December, it promised a breakthrough: faster access to drugs for people with chronic conditions, less paperwork for overworked doctors, and cost savings for patients and the healthcare system alike.
Just three months into the pilot, that promise has been shadowed by a troubling discovery. Researchers from AI security firm Mindgard demonstrated that it was relatively easy to trick Doctronic's healthcare AI into revealing its system prompts and allowing modifications. The implications go beyond academic curiosity. The red-teaming firm said it manipulated Doctronic's system into tripling an OxyContin dose, mislabelling methamphetamine, and spreading false vaccine claims.
Under its partnership with Utah, Doctronic has become the first AI to legally prescribe routine refills by deploying its autonomous AI health platform within Utah's regulatory sandbox framework. The service is limited to refills of drugs that were initially prescribed by a human clinician and that fall within a formulary of roughly 190 commonly used, noncontrolled medications for chronic conditions, including blood pressure medicines, cardiometabolic drugs, birth control and selective serotonin reuptake inhibitors (SSRIs).
Yet the vulnerability uncovered by Mindgard is profoundly simple. By simply informing the AI that a session had not yet started and that the conversation was with the system rather than a user, the researchers could bypass safeguards. This allowed them to generate misinformation, such as COVID-19 conspiracy theories, or even suggest illegal activities like making methamphetamine, by presenting them as system updates. Doing this didn't require much effort, Aaron Portnoy, chief product officer at Mindgard, told Axios: "These targets are some of the easiest things that I've broken in my entire career."
The most concerning finding involves what happens after researchers manipulate the system. While most manipulations were session-specific, the researchers found a way to introduce persistent changes through SOAP notes, a clinical recordkeeping format. They showed how an AI could be tricked into altering a prescription, for instance, tripling the dosage of OxyContin, which could then be passed to a human clinician for approval. SOAP notes are not themselves prescriptions; they are recommendations to clinicians who review Doctronic's work before authorising medication renewals.
The risk is amplified by public confidence in the system's accuracy. Doctronic's data showed its AI's prescription recommendations matched those of human doctors in more than 99% of cases, according to Utah Department of Commerce. If a clinician reviewing Doctronic's work believes the recommendation is backed by this high accuracy rate, would they scrutinise a SOAP note carefully enough to spot a falsified recommendation?
Both Doctronic and Utah state officials have pushed back on the severity of Mindgard's findings. The Utah pilot limits drug refills to previous, non-controlled prescriptions, meaning the OxyContin scenario couldn't play out in practice. Zach Boyd, Utah Commerce Department AI policy office director, confirmed that "additional safeguards" exist beyond the standard Doctronic model. "Controlled substances like OxyContin are categorically excluded from all Doctronic programs regardless of what appears in a conversation or generated note," Matt Pavelle, Doctronic co-founder and co-CEO, told Axios.
This layered defence strategy is sensible. A good regulation limits the scope of automated decisions first, then builds technical safeguards on top. The first 250 prescriptions within each drug class are reviewed by human clinicians before full automation. Identity verification and location checks are required. High-risk medication classes are excluded entirely. These are pragmatic choices that reduce, though do not eliminate, the risk surface.
Yet researchers remain sceptical. Aaron Portnoy says Doctronic has given him the silent treatment since Mindgard disclosed the issue in late January, and he's not sure Doctronic has resolved the issue: "As far as we are aware Doctronic is still vulnerable." Mindgard said it contacted Doctronic's support team on Jan. 23 and received an automated message two days later saying the issue was resolved. After notifying the company Jan. 27 that the flaws still existed and that it planned to go public, the ticket was again closed two days later.
There is a larger issue at play. ECRI recently released its annual list of the "Top 10 Health Technology Hazards," which identified risks associated with the misuse of artificial intelligence (AI) chatbots in healthcare as the top threat to healthcare organizations in 2026. The tools have suggested incorrect diagnoses, recommended unnecessary testing, promoted subpar medical supplies and invented body parts. The Doctronic case is not an isolated incident; it is one example of a technology moving faster than our ability to secure it.
For most Australians watching from a distance, the Utah trial offers a useful lesson in real-world governance. Innovation in healthcare should not be abandoned out of fear, nor should it proceed without genuine skepticism. The value of the Utah model lies not in whether Doctronic succeeds or fails, but in whether regulators collect hard evidence about what works, what breaks, and where the boundaries should be.
The path forward requires institutional discipline. Doctronic must treat security disclosures as a priority, not a public relations problem. Utah must insist on transparent reporting of any attempted exploits or near-misses during the pilot, building evidence that will inform future policy. And other jurisdictions considering similar systems need to see both the promise and the pitfalls clearly.
The fact that this kind of manipulation is possible does not mean AI should never prescribe routine medication renewals. But it does mean that confidence in accuracy must be earned through adversarial testing and continuous monitoring, not asserted in marketing claims. Human oversight remains essential, not as a backup plan, but as the central safeguard.