Rejoinder to Pitt

The April issue of the Journal of Development Studies includes the final version of my article with Jonathan Morduch replicating the study of the impact of microcredit in Bangladesh by Mark Pitt and Shahidur Khandker. Properly, the journal also carries a reply from Mark Pitt. (Ungated versions of the dueling documents are here and here.) To my surprise, JDS did not solicit a rejoinder from us the way they did in a nearly identical situation involving a JDS editor as replicating author. Perhaps this is a sign of the strength the editors see in our paper…which is to say, maybe I should have just chilled. But as usual, Pitt’s arguments are strongly worded even as their subject remains technical. So the average reader will absorb the style more than the substance, and wonder, I fear, who are these fools Roodman and Morduch? So for the public record, here is a rejoinder from yours truly. It’s a quote-and-response. “RM [Roodman & Morduch] have backed off many of their prior claims and methods.” No. The first version of our paper questions the exogeneity of the core intent-to-treat variables; highlights that an asserted discontinuity in treatment, central to Pitt & Khandker’s (PK’s) claim to quasi-experimental status, is absent from the data; observes that the magnitude of the impact estimates depends on an arbitrary censoring choice for the “log of 0”; and demonstrates that a more-robust linear estimator produces no evidence of impact. Those arguments stand. Of course, I have also learned from the debate with PK. RM (2009) failed to replicate PK’s original impact estimates, getting opposite signs. Pitt (2011) showed us how to match, by swapping in a missing control and adopting a different censoring value. As it...

Shoddy microcredit impact reporting in Economist

The Economist just posted an article on the state of microfinance in Bangladesh. I’m surprised at how weak it is: The article reports that on April 6, the government of Bangladesh arrogated for itself the right to appoint members of the board of the Grameen Bank, while providing almost no interpretive context for this move. It describes a new study of the impact of microcredit in Bangladesh by Shahidur Khandker and Hussain Samad as the “biggest” so far, by which it seems to mean the one with the most households in it. Not so. The new study has 1,509–2,322 households depending on how you count (see the paper’s Table 1); as a counterexample, a study in Hyderabad, India, had about 6,850. The new study is however the longest, tracking families for a remarkable 20 years. The article describes the positive results of this study as breaking a pattern of research finding “limited or no benefits” of microcredit. Actually, it continues a pattern. Shahid Khandker has authored many papers with similar conclusions: Pitt and Khandker (1998); Pitt and Khandker (2002); Pitt, Khandker, Chowdhury, and Millimet (2003); Khandker (2005); Pitt, Khandker, and Cartwright (2006); Pitt and Khandker (2012); Khandker and Samad (2013); Khandker, Samad, and Ali (2013)… The pattern continued after randomized studies began appearing in 2009 that produced the less-positive findings mentioned. More fundamentally, the article evinces no understanding of why those randomized studies are more trustworthy (see below). The article appears to commit what Dierdre McCloskey and Stephen Ziliak dub the “standard error of regressions,” which is to confuse statistical significance with real-world significance. Statistical significance, as meant here,...