Listen: https://serve.buildtall.systems/2d7851b764f7fe6214fdcaaef22b641040da2ed46efa6e76803131edf6503a1c.mp3
This episode of marginal revolution read out covers the post Checking the AI audit of Saez and Zucman’s research. The author opens by pointing to a set of linked documents about an anonymous AI audit of work by Emmanuel Saez and Gabriel Zucman, and about the effort to check that audit.
The first linked item is From SaneCite. This is a post on X in which SaneCite describes its own check of the anonymous AI audit. According to the fetched text, SaneCite checked all 147 problems the audit called serious enough to change a headline number, against the papers, their sources, and the authors’ own data. For 131 of the 147, every fact that could be checked was right, and none had its core facts wrong. Twenty-one change a headline number, ninety-eight depend on a disputed choice of method, twenty change no number, and eight could not be checked on that question. A second pass graded the ninety-nine flags initially counted as depending on a disputed choice of method: eleven standard, seventy defensible, eleven questionable, six could not be graded, and one contested. That second pass found one item, number fifty-four, that does change a headline number, which is why the counts are now twenty-one and ninety-eight. SaneCite also says Saez and Zucman were sent the findings on October 29 and did not reply by November 6. The author introduces this link with the phrase From SaneCite, and quotes the statement as the centerpiece of the post.
The next link is the audit of the audit, hosted on SaneCite. The author’s remark is that it has open code and data, and the author gives kudos to the authors. The fetched page shows one example, labeled number six, of how the check works. It looks at a March 2022 note by Neef and others from the EU Tax Observatory and World Inequality Lab, titled Effective Sanctions against Oligarchs and the Role of a European Asset Registry. The note says about half of Russian household wealth is held abroad; the study it cites gives about a quarter, twenty-four percent, of total wealth. The audit of the audit concludes that the note’s summary claim survives. This is a useful illustration: the audit of the audit is not a blanket rejection or endorsement of the original audit, but an item-by-item test of which facts hold up, which methods are disputed, and which claims really change a headline number.
The next link is the original AI audit. The author’s commentary is brief: here was the original AI audit. The fetched text failed, so this episode can only describe what the title suggests and what the post already tells us. The title points to the earlier Marginal Revolution post from October 28 that first featured the anonymous AI audit of Saez and Zucman’s research. The SaneCite statement gives the relevant context: that original audit listed 147 problems it considered serious enough to change a headline number. Since the fetched copy could not be retrieved, the discussion here stays with the way the post links back to that earlier piece as the starting point of the debate.
The final link is the October 30 revision of that audit. The author points to it as the updated version of the original AI audit. The fetched page displays a detailed entry for one academic paper. The paper is Taxing the Wealth of the Poor: Evidence from the Danish Old-Age Support Asset Test, by Johannesen, Sæverud, and Saez, published in American Economic Review: Insights. The audit lists eight defects and six contested choices, with twenty-seven of thirty-three checks reproduced and five not checkable, giving it a moderate defect rating. The main issue is that the savings limit studied in the paper governs two benefits at once: a yearly cash payment and a health subsidy for medicine, dental care, and glasses. The paper credits the whole savings response to the cash payment, but the 2003 and 2010 rule changes used as tests also changed the health subsidy. The response is real, but how much of it comes from the cash payment alone is not known. The audit also notes that the twenty percent figure compares savings on December 31, the day the government checks them, when these pensioners’ bank balances are lowest because the yearly payment arrives in January and is spent over the year. Year-round the effect is about minus fifteen to minus sixteen percent. Another contested choice is that the paper compares the same people at ages sixty to sixty-four and seventy to seventy-four, assuming their savings would otherwise look the same ten years later. The same comparison that shows lower savings near the limit also shows ten to seventeen percent higher savings among the poorer half of these pensioners. An earlier version of the paper used a different method and found an effect about half as large, though over a shorter period and when the payment was smaller.
Taken together, the post shows a replication debate playing out in real time: an AI audit flags errors, a third party checks the audit itself, and a revised version of the audit refines the defects and separates factual fixes from contested methodological choices. The author’s stance is mostly to curate: quote SaneCite, link to the open audit of the audit, and point back to the original and revised audits.
