AI Coding Agents: Why They Cannot Replace Human Programmers Yet
New research shows that AI models complete only 26% of tasks without errors, making human intervention a necessity.
Real-world environments and AI limitations
While advancements in artificial intelligence have revolutionised programming, recent research suggests it is too early for total optimism. A team of researchers from Alibaba and several Asian universities has developed a new testing system, Co-Design Bench, to evaluate how effectively AI agents handle real-world programming tasks. The results indicate that even the most advanced models, such as GPT-6 Astra, complete tasks perfectly only 26.4% of the time.
Unlike traditional tests, where models often report success rates exceeding 99%, Co-Design Bench relies on actual user requirements. These tasks involve editing web applications, utilising not only text but also images, audio, and video as inputs. This approach more accurately reflects the complexities developers face in their daily work.
Why human supervision is necessary
According to the study, although AI agents successfully handle a large portion of a task, they almost always leave behind small but critical errors. This means programmers must still review and manually correct the code. GPT-6 Astra, which performed best in the tests, met an average of 70% of task requirements, yet a perfect result remained rare.
Researchers note that task difficulty increases when design, interaction, and content changes are combined within a single project—a standard requirement in real-world scenarios. Interestingly, increasing computational resources does not always lead to better results. For instance, the GLM-5.2 model consumes six times more resources than GPT-6 Astra, yet significantly underperforms in comparison.
Environment and outcomes: What companies should consider
The study also revealed that a model's effectiveness depends not only on the algorithm itself but also on the harness in which it operates. Results for the same model can vary drastically depending on the software environment in which it writes and executes code. Therefore, companies planning to implement AI must evaluate the model and the working environment as a unified system.
While artificial intelligence is already a powerful tool, it cannot yet replace qualified specialists. Real-world workflows are far more flexible and unpredictable than academic tests. For developers, this means that while AI tools increase productivity, ensuring final quality and perfecting code remains a human competency.