Talks

10x Coding with LLMs — hype and reality

A ~30-minute talk given at IBM Research, for a technical research audience.

The demos promise 10x productivity. The evidence says otherwise, and the interesting part is why. LLMs are best understood as a probabilistic abstraction layer: generation has become cheap, while reading and verification have become the expensive part. Whether an LLM helps or just shuffles work around is decided by three things — the context you give it, the guardrails you put around it, and the verification you run on its output.

The measured gains are real but modest. A randomized field study of 4,867 developers found about 26% more tasks completed with an AI assistant, with the largest gains among less-experienced developers. In METR’s study, expert open-source maintainers working in repositories they knew well were 19% slower with AI tools — while believing they had been faster. And since writing code is roughly 17% of a developer’s workday, Amdahl’s law caps the overall speedup long before 10x.

The conclusion is not to avoid LLMs. It is to engineer the context, guardrails, and verification — which happens to be the job anyway.