Research note
Research note: AI coding agents add 30% more code but not more finished software, Harvard researchers find
How we checked a Harvard working paper on AI coding agents through two independent reports, and why we stress its limits.
What we set out to check
Who did the study, how big it is, what it found, and what it does not show.
What we found
- Two independent reports give the same core figures: about 30% more code, 20% more commits, 23% more pull requests, no significant rise in completed tasks, and 49% longer time to merge.
- Netzwoche lists the limits and notes the paper is not peer reviewed.
Numbers that differ
One report counts “more than 700” companies and another gives 718. We say “more than 700”. On AI review comments, one source says 23% of comments were AI-written and another says about one in nine changes got an AI comment; these measure different things, so we describe it only as a small share.
Labels we chose and why
- Reported for the main findings, carried by two outlets.
- Unconfirmed for peer-review status, from one outlet.
What we could not verify
- The paper itself.
- Whether results apply beyond Jellyfish customers.
What we searched
- Harvard study AI coding agents Jellyfish 700 firms
- AI coding agents more code not more software
- Chen Stratton AI coding agents pull requests merge time
Every source we read
| Source | Used? | Why |
|---|---|---|
| AI Weekly | USED | Summary of the paper with figures and a quote from the authors. |
| Netzwoche | USED | Independent report with the same figures, the limits, and a link to the paper. |
| Ars Technica | NOT USED | Could not be opened by our tools. |