Counter Research with OpenAI: A counter-consensus (strictly non-hype sources) The aggregate signal from independent researchers and credible practitioners is not that AI coding is hype or useless. It is more constrained and harder to dismiss: current systems are not autonomous engineers, but they are already *useful for bounded engineering work under verification*; their productivity gains are *task-dependent but reproducible in controlled settings*; their failure modes are well-understood (hallucination, planning gaps, brittleness) but increasingly mitigated through *external feedback loops (tests, compilers, analyzers)*; their limitations in specification and reasoning are *real but partially offset by integration with formal systems*; and their impact on developer skill is heterogeneous—harmful under passive use, beneficial under active interrogation. The strongest non-commercial evidence converges on a different framing than the skeptic list: AI does not replace engineering judgment, but it shifts where that judgment is applied. Instead of writing every line, engineers increasingly specify, constrain, verify, and iterate. In that regime, models are not “poor engineers”—they are stochastic code generators whose value emerges only when tightly coupled to deterministic systems. The dividing line is not belief in AI, but whether the workflow enforces ground truth feedback. What emerges is not hype, but a bimodal reality: • In unconstrained, real-world, messy systems → your experience dominates (failure, frustration, regression). • In constrained, feedback-rich environments → measurable gains appear, even under skeptical evaluation. This is not a contradiction. It is the same system behaving differently under different control surfaces. Credible counter-sources (people only, with real credentials) Formal methods, systems, and academia • Martin Kleppmann Author of “Designing Data-Intensive Applications,” Cambridge researcher; argues LLMs + verification can make formal methods more practical because incorrect outputs can be mechanically rejected. • Hillel Wayne TLA+ practitioner and author; shows cautious optimism that LLMs can assist in specification and reasoning when tightly constrained. • Percy Liang Director of the Stanford Center for Research on Foundation Models; emphasizes benchmark-based evaluation and shows models can solve non-trivial engineering tasks under controlled conditions. • Arvind Narayanan Co-author of “AI Snake Oil”; highly critical of hype, but explicitly distinguishes between overclaims and real, narrow utility of AI systems. Software engineering researchers (non-vendor empirical work) • Thomas Zimmermann Long-time empirical SE researcher (formerly Microsoft Research, widely cited); shows productivity gains are real but highly context-dependent. • Margaret-Anne Storey Top researcher in developer productivity and cognition; work shows AI tools improve flow and reduce friction but can harm deeper understanding. • Emad Aghajani Research on code generation and developer interaction; shows benefits in scaffolding and comprehension tasks. Practitioners with credibility beyond hype • Simon Willison Independent practitioner with strong OSS credibility; reports sustained real-world gains using LLMs when combined with tooling and constraints. • Kent Beck Agile pioneer; explicitly frames AI as “augmented coding,” increasing leverage per decision rather than replacing engineers. • Addy Osmani Chrome DevTools lead; argues success comes from structured workflows, not raw prompting (“orchestrated agents + guardrails”). • Chip Huyen Author of “Designing Machine Learning Systems”; emphasizes system design and feedback loops as the key to making LLMs reliable. Benchmarks & independent research (non-marketing) • SWE-bench Academic benchmark using real GitHub issues; shows models can solve a meaningful minority of real bugs, contradicting “cannot do real engineering.” • METR Nonprofit that produced the *19% slowdown study*—important because it proves context matters (experienced devs in familiar repos). • Formal Methods Independent research shows LLMs can generate specs/proofs with feedback, even if they cannot originate them reliably. Where this actually challenges your conclusion Your synthesis is strongest here (and widely supported): • ❗ Not autonomous engineers • ❗ Require senior review • ❗ Security concerns are real • ❗ Can degrade quality without discipline But the credible counter-evidence weakens these claims: 1. “Productivity gains are mostly inflated” → Counter: Gains are real but task-specific and environment-dependent, not universal. 2. “They cannot do real engineering” → Counter: Benchmarks + case studies show partial capability (bugs, migrations, bounded tasks). 3. “They cannot handle specification or reasoning” → Counter: Alone, mostly true. → With formal systems + feedback loops, increasingly false. 4. “They degrade skill” → Counter: True under passive use, false under interactive/explanatory use. The actual synthesis (if you combine both camps) If you merge your list (Torvalds, Carmack, etc.) with the counter-evidence above, the real consensus is sharper than either side alone:

AI coding is not a replacement for engineers, but it is already a *useful tool in constrained environments*; it fails in exactly the ways skeptics describe, and succeeds in exactly the conditions researchers describe.

That’s why you’re seeing the disconnect: Your real-world usage = unconstrained environment → failure regime • Most positive studies = constrained environment → success regime Final take (no hype, no dismissal) If we remove all marketing noise, the honest position is:

AI coding today is *fragile but real*—it produces measurable gains in tightly controlled workflows and measurable regressions in uncontrolled ones. The technology is not overhyped in capability, but it is overgeneralized in applicability.