replication - David Roodman

Disappointment about the war on worms in the American South 100 years ago

On GiveWell.org, I just blogged a new study revisiting the evidence on whether the campaign in the 1910s to rid the South of hookworm brought major benefits. A great 2007 paper by Hoyt Bleakley suggests that it did: after eradication school attendance rose disproportionately in historically hookworm-heavy areas; and adult earnings of babies born in affected areas also later rose.

The new study revisits Bleakley’s original by reconstructing its database from primary sources, and replicating and revising the analysis. I ended up strongly questioning the original study’s conclusion. These two pairs of graphs show why. The first graph in each pair is from the original study, the second from the new version. The original graphs seem to show jumps in outcomes of interest—school attendance, earnings—but the new ones do not.
Continue reading “Disappointment about the war on worms in the American South 100 years ago”

Who has the power to count chickens?

In March I recounted how former colleagues Michael Clemens, Steven Radelet, and Rikhil Bhavnani wrote an excellent paper in 2004 on the impact of foreign aid on economic growth, “Counting Chickens When They Hatch.” The idea captured in that title is that it is important to think about the likely timing of the impacts of aid. Don’t design your analysis as if you expect that funding for teaching 6-year-olds will raise economic growth in four years. Match the follow-up period to the type of aid. Count your chickens only when they hatch.

Some years later, and joined by another CGD recruit, Samuel Bazzi, those authors overhauled their paper and published it in the top-flight Economic Journal (ungated version). The final version is quite different but also excellent (it won the journal’s best-article prize). Instead of doing its own econometrics afresh, it modifies the three most-cited studies in the aid-growth literature in light of the “counting chickens” insight. Although those studies disagree on whether and when aid “works,” in the sense of boosting growth, Clemens, Radelet, Bhavnani, and Bazzi (CRBB) conclude that revising the studies to take timing into account causes all results to converge, to a ginger but positive appraisal. (Listen to Michael speak cautionarily about “Chickens” in this Library of Economics and Liberty podcast.)

I say “excellent” and I mean it. But, true to type, I actually doubt the econometric reasoning. I am not persuaded by these results that “aid inflows are systematically associated with modest, positive subsequent growth.” Continue reading “Who has the power to count chickens?”

Rejoinder to Pitt

The April issue of the Journal of Development Studies includes the final version of my article with Jonathan Morduch replicating the study of the impact of microcredit in Bangladesh by Mark Pitt and Shahidur Khandker. Properly, the journal also carries a reply from Mark Pitt. (Ungated versions of the dueling documents are here and here.) To my surprise, JDS did not solicit a rejoinder from us the way they did in a nearly identical situation involving a JDS editor as replicating author. Perhaps this is a sign of the strength the editors see in our paper…which is to say, maybe I should have just chilled.

But as usual, Pitt’s arguments are strongly worded even as their subject remains technical. So the average reader will absorb the style more than the substance, and wonder, I fear, who are these fools Roodman and Morduch? So for the public record, here is a rejoinder from yours truly. It’s a quote-and-response.

“RM [Roodman & Morduch] have backed off many of their prior claims and methods.”

No. The first version of our paper questions the exogeneity of the core intent-to-treat variables; highlights that an asserted discontinuity in treatment, central to Pitt & Khandker’s (PK’s) claim to quasi-experimental status, is absent from the data; observes that the magnitude of the impact estimates depends on an arbitrary censoring choice for the “log of 0”; and demonstrates that a more-robust linear estimator produces no evidence of impact. Those arguments stand.

Continue reading “Rejoinder to Pitt”

Comment on “Counting Chickens”

In 2012, the top-flight Economic Journal published an article by my former CGD colleagues about the impact of foreign aid on economic growth in receiving countries. It is called Counting Chickens When They Hatch: Timing and the Effects of Aid on Growth (ungated version on cgdev.org). The journal’s publisher, the Royal Economic Society, awarded the authors its annual prize for best article. I think you’ll see why, if you read the paper. It’s sharply written and firmly argued, yet judicious in its conclusions.

Continue reading “Comment on “Counting Chickens””