At QCon London, Birgitta Böckeler, global lead for AI-assisted software delivery at Thoughtworks, warned that AI is in a dangerous state where it is too useful not to use, but by using it developers are giving up the experience they need to review what it does.
The concern reflects a genuine bind facing organisations deploying advanced coding agents. Advances in agent orchestration, sub-agents, and reduced supervision are forming strong forces that are tempting humans out of the loop. Yet this apparent efficiency masks a deeper problem. Böckeler noted that the industry is getting into a dangerous state where AI is so useful that developers want to use it, but they cannot and maybe never will be able to give it everything, and at the same time they are getting less experience because they are not doing the work themselves anymore.
The practical risks are substantial. AI is not safe: it makes errors and is vulnerable to issues such as prompt injection. Independent research reinforces this concern. Research indicates that approximately 25-30% of code generated by models like GitHub Copilot contains Common Weakness Enumerations (CWEs). More broadly, a recent study found that 62% of AI-generated code solutions contain design flaws or known security vulnerabilities, even when developers used the latest foundational AI models.
The danger extends beyond code quality to developer psychology. A recent study found that coders who used an AI assistant wrote significantly less secure code than those who did not, and even though they created less secure coding, these participants believed their code was actually more secure than the manual creation group. This combination of reduced scrutiny and inflated confidence amplifies risk.
Böckeler identified what she framed as a critical security concern. When an agent has exposure to untrusted content and access to private data and can externally communicate, there is a high risk of data and security problems; giving an agent read and send rights to email alone is enough to trigger this problem. This scenario, sometimes called the "lethal trifecta," represents a structural vulnerability in how autonomous agents are deployed.
The industry is not ignoring these warnings. Whilst adoption is widespread at 84% among developers with around 51% of professionals using AI tools daily and material productivity gains of approximately 3.6 hours per week, independent code analysis raises a clear caution that AI-assisted code can increase issue counts by 1.7 times and security findings if not paired with governance.
Developers are in the business of risk assessment, which involves a combination of three things: probability, impact, and detectability. The challenge is that these variables shift as tools become more powerful. Böckeler observed that the longer an agent goes without supervision, the more review it requires afterwards.
Solutions exist but require discipline. Training developers on secure prompting is essential, since prompts are now the code design specification, and developers need to be taught not just how to use AI coding assistants but how to guide them with specificity. Conducting thorough code reviews and exercising control over LLM outputs helps secure AI-driven development against evolving threats.
The tension between productivity and risk is real and cannot be wished away. Organisations deploying AI coding agents must treat them not as fully trusted collaborators but as tools requiring active governance. The productivity gains are genuine, but so are the risks. The question for development teams is not whether to use AI, but how to use it responsibly.