↵ select ↓ ↑ navigate esc close

AIs vs. humans in economics papers

Marginal REVOLUTION ·

Is agentic research any good? Dr Yanagizawa-Drott and his colleague, Olaf Willner, used an AI verifier to check 1,000 APE papers. None was error-free; 842 contained at least one severe error, including ones invalidating headline results. To see roughly how Ape
stacks up against human primates, The Economist teamed up with Refine, the research-verification firm founded by Dr Golub. Refine chose 100 papers at random from those accepted for publication in the top five journals (including QJE and JPE ) in the first nine months of 2026. It then subjected them to thorough review using several frontier AI models plus a once-over for serious errors by a human reviewer. The exercise flagged 4,158 issues in the 100 papers. One in ten papers contained at least one grave error (other problems concerned interpretation or inconsistencies between tables and text).

The two exercises are not directly comparable, but human economists’ lower error rate might give them some hope. In ideas, too, people retain the edge. One experiment, by Alexis Akira Toda of Emory University, found that when considering papers on economic theory alone, AI lowers the cost of checking for a candidate flaw but “does not yet reliably substitute for the human judgment that locates the flaw in the first place”. Another, which compared APE papers with published papers found the difference is mostly in ideas, not execution. In a further study, Michele Zampa of the Geneva Graduate Institute used an AI referee to grade 981 AI papers and 41 human ones on things like the novelty of research questions and acknowledgment of methodological limitations. AI did better in crowded fields with lots of existing research to digest. Humans performed at the same level regardless of the size of the literature.

…Benjamin Kohler of ETH Zürich, another Swiss university, and colleagues, let AI reproduce 48 social-science papers, using only the studies’ raw data and a description of the method. The discrepancies between the original papers and the AI versions were due in roughly equal measure to problems with the AI and human work. But the better the AI model, the more the balance tilted against humans.

Here is more from The Economist, interesting throughout.

The post AIs vs. humans in economics papers appeared first on Marginal REVOLUTION.