↵ select ↓ ↑ navigate esc close

marginal revolution read out: AIs vs. humans in economics papers

marginal revolution read out ·

Listen: https://serve.buildtall.systems/b4d87ac7488072b0ba4a6f1e076168006df05f76bbd7aac85bf34191d04061d6.mp3

Source: https://marginalrevolution.com/marginalrevolution/2026/11/ais-vs-humans-in-economics-papers.html?utm_source=rss&utm_medium=rss&utm_campaign=ais-vs-humans-in-economics-papers


marginal revolution read out. Today's post is AIs vs. humans in economics papers, from Marginal REVOLUTION. The author opens by asking whether agentic research is any good, then works through several recent checks on how AI-generated economics research compares with work by human economists. The post collects findings about errors, idea generation, refereeing and reproducibility, and it closes with a pointer to a longer Economist article. The post is less a single study and more a roundup of evidence from different teams.

The first article carries the same title, AIs vs. humans in economics papers. It begins with a study by Dr Yanagizawa-Drott and Olaf Willner, who used an AI verifier to check one thousand APE papers. Their results were stark: none of the papers was error-free, and eight hundred forty-two contained at least one severe error, including errors that invalidated headline results. That is a striking failure rate for a research pipeline meant to lower the cost of producing economics work. To see how APE stacks up against human primates, The Economist teamed up with Refine, the research-verification firm founded by Dr Golub. Refine chose one hundred papers at random from those accepted for publication in the top five journals, including the Quarterly Journal of Economics and the Journal of Political Economy, in the first nine months of 2026. Those papers were then subjected to thorough review using several frontier AI models, plus a once-over for serious errors by a human reviewer. The exercise flagged four thousand one hundred fifty-eight issues across the one hundred papers. One in ten papers contained at least one grave error, while other problems concerned interpretation or inconsistencies between tables and text. So even established human-published journals are not clean.

The author's take is that the two exercises are not directly comparable, but the lower error rate among human economists might give them some hope. On ideas, people still keep an edge. In an experiment by Alexis Akira Toda of Emory University, AI lowers the cost of checking for a candidate flaw, but it does not yet reliably substitute for the human judgment that locates the flaw in the first place. The author highlights this as a limit: the machine can test a suspicion, but the suspicion itself still comes from a person. That distinction matters because checking a flaw is cheaper than finding one. Another comparison of APE papers with published papers found that the difference is mostly in ideas, not execution. Michele Zampa of the Geneva Graduate Institute used an AI referee to grade nine hundred eighty-one AI papers and forty-one human papers on things like the novelty of research questions and acknowledgment of methodological limitations. AI did better in crowded fields with lots of existing research to digest, while humans performed at the same level regardless of the size of the literature. The finding suggests AI's advantage is search and synthesis, while human judgment is more general. Finally, Benjamin Kohler of ETH Zürich, another Swiss university, and colleagues let AI reproduce forty-eight social-science papers using only the studies' raw data and a description of the method. The discrepancies between the original papers and the AI versions were due in roughly equal measure to problems with the AI and with the human work. But the better the AI model, the more the balance tilted against humans. This implies that as models improve, the residual errors increasingly point back to the original research.

The second article is more from The Economist. The author's commentary is simply that it is more from The Economist, interesting throughout. Because the article text failed to fetch, nothing can be said about the specific reporting or evidence in that piece. From the title and commentary alone, it appears to be the Economist's broader treatment of AI's move into economics, but no further detail should be inferred.

Taken together, the post suggests that AI is becoming a useful verifier, referee and reproducer in economics, but it still falls short on locating the original flaw or asking the original question. Human economists may have a lower error rate and a persistent edge in ideas, even as better models narrow the gap. That is a cautious, concrete picture of where AI and human researchers stand relative to one another. It leaves the listener with a sense that AI is not yet an autonomous economist, but it is becoming an indispensable research assistant.

  1. AIs vs. humans in economics papers
  2. more from The Economist