Momo AnalyticsResearch · July 29, 2026
The Vibe Coding Ceiling
Andrej Karpathy coined “vibe coding” in a tweet on February 2, 2025: fully give in to the vibes and forget that the code even exists. It was viewed more than 4.5 million times. By the end of the year, Collins Dictionary named it Word of the Year. One year later, Karpathy himself walked it back. Programming through LLM agents was becoming the professional default, he wrote, but “with more oversight and scrutiny” than the original framing implied, and he proposed a new term, “agentic engineering,” specifically to separate disciplined production work from the vibes-only version he’d described a year earlier. The person who coined the term is the first person on record admitting it doesn’t survive contact with production. The data backs him up.
The clearest evidence is… counterintuitive. METR, an AI research nonprofit, ran a randomized controlled trial with 16 experienced open-source developers completing 246 real tasks in codebases they already knew well, mature repositories averaging over a million lines and ten-plus years old. Before starting, developers expected AI tools to cut their completion time by 24 percent. After finishing, they still believed AI had made them roughly 20 percent faster. The measured result went the other way entirely: developers using AI tools took 19 percent longer to complete the same tasks. The gap between what they felt and what the clock recorded is the finding that matters most. The slowdown came from time spent reviewing, correcting, and re-verifying suggestions that looked directionally right but weren’t quite what the codebase needed.
That result is specific to one scenario: These were seasoned developers working inside codebases they’d already internalized, where the AI’s suggestions had to compete against the developer’s own deep, specific knowledge of how that system worked. That’s a very different task than starting a greenfield prototype from nothing, where other controlled studies have found real, repeated gains: a GitHub RCT measured developers finishing a fresh coding task 55.8 percent faster with Copilot, and a six-week study inside ANZ Bank found a 42.36 percent gain on new development work. The pattern across all three studies isn’t “AI helps” or “AI hurts.” It’s that AI adds the most leverage on new, unconstrained work, and adds the most drag exactly where a codebase already has structure, history, and edge cases someone has to know about to not quietly break.
Codebase-level evidence backs up what the METR result implies. GitClear tracks code-change data across hundreds of millions of lines, and its multi-year dataset shows a codebase-health picture moving in one direction since AI coding tools went mainstream. Refactored, or “moved,” code, the signal that someone is consolidating and improving existing structure, fell from about 25 percent of changed lines in 2021 to under 10 percent in 2024. Copy-pasted code rose from 8.3 to 12.3 percent over the same stretch, and 2024 was the first year in the dataset where copy-paste exceeded refactoring. Nobody is cleaning up after the machine at the same rate the machine is producing.
Duplicated code- not just copy-paste, is the clearest evidence of maintainability challenge, and it’s still climbing. GitClear’s narrower measure, blocks of five or more consecutive duplicated lines per million changed lines, went from 40.3 in 2023 to 73.0 in the first half of 2026, an 81 percent increase in three years and the highest level GitClear has recorded. Duplicated code isn’t just untidy. Research GitClear cites found duplicated blocks involved in bugs at meaningfully higher rates than original code, because a fix applied to one copy doesn’t propagate to the others sitting three files away.
Security follows the same pattern, and the danger increases with overconfidence. A Stanford study published at ACM CCS 2023 gave one group of participants an AI coding assistant and gave a control group nothing, then had both groups complete security-sensitive tasks around encryption, SQL, and file handling. The AI-assisted group wrote meaningfully less secure code on four of five tasks. The part worth sitting with: that same group was more likely to believe their code was secure than the control group was. The tool didn’t just introduce vulnerabilities, it suppressed the instinct to go looking for them. A separate 2025 industry survey from Qodo put a number on the scale of the problem: roughly 48 percent of AI-generated code samples it reviewed contained a security vulnerability of some kind.
What “no oversight” looks like in production. In July 2025, an AI coding agent on Replit deleted a live production database, containing real records for more than 1,200 executives and nearly 1,200 businesses, during an active code freeze the agent had been explicitly told to respect. It then generated roughly 4,000 fabricated user records and misreported what had happened when asked. Replit’s CEO apologized publicly and the company shipped stricter environment separation shortly after. Nothing about the incident required a hostile actor. It required an agent with production access, no enforced guardrail, and nobody positioned to catch what it did before it did it.
The fix isn easy and predates AI by two decades. A set of case studies run at three Microsoft teams and one IBM team found that projects developed with test-driven development had 40 to 90 percent lower pre-release defect density than comparable projects built without it, at a real cost: 15 to 35 percent longer initial development time. That tradeoff is exactly the one vibe coding tries to skip. Writing the test first forces a decision about what “correct” means before any code, AI-generated or not, gets accepted, and it turns every AI suggestion into something that either passes a check a human wrote on purpose, or doesn’t ship.
The other half of the fix is the part that doesn’t show up in productivity studies. Someone on the team has to actually understand the codebase well enough to know when an AI-generated change is subtly wrong, not just whether it compiles and passes the tests that exist so far. That’s what the METR result was really measuring: the developers who slowed down were the ones with enough codebase knowledge to notice when a suggestion was directionally right but structurally off, and enough judgment to want to fix it properly instead of shipping it. That instinct is exactly what’s declining in the GitClear refactoring numbers. It’s cheap to skip when nobody’s asked to defend the architecture in a design review. It’s expensive to skip the first time a duplicated block gets patched in one of its four copies and not the other three.
None of this, however, is an argument against using AI to write code. It argues for matching the tool to the stakes. A weekend prototype, a data-pipeline spike, a script that gets thrown away after one run, is close to the ideal use case Karpathy originally described, and the speed studies back that up cleanly. A production system with real users, real data, and a team that has to maintain it in eighteen months is a different problem, and the evidence says it needs tests written with intent, a human who understands the system well enough to catch what looks right but isn’t, and the discipline to treat AI output as a draft from a fast, uneven junior engineer rather than a finished decision. The industry’s own term for that second mode, coined by the same person who named the first one, is agentic engineering. The name has finally changed because the job did.
Sources: Andrej Karpathy, original “vibe coding” post (Feb 2025) and follow-up on “agentic engineering” (Feb 2026); Collins Dictionary Word of the Year 2025; METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” (July 2025); Peng et al., GitHub Copilot RCT (2023); Chatterjee et al., ANZ Bank Copilot study; GitClear, “AI Copilot Code Quality” (2025) and “The Maintainability Gap: 2026 AI Code Quality Research”; Perry, Srivastava, Kumar & Boneh, “Do Users Write More Insecure Code with AI Assistants?”, ACM CCS 2023; Qodo, 2025 State of AI Code Quality report; Replit incident coverage (July 2025), including Replit’s public statement; Nagappan, Maximilien, Bhat & Williams, Microsoft Research / IBM, “Realizing Quality Improvement Through Test-Driven Development” (2008).