Shoddy microcredit impact reporting in Economist 

The Economist just posted an article on the state of microfinance in Bangladesh. I’m surprised at how weak it is:

  • The article reports that on April 6, the government of Bangladesh arrogated for itself the right to appoint members of the board of the Grameen Bank, while providing almost no interpretive context for this move.
  • It describes a new study of the impact of microcredit in Bangladesh by Shahidur Khandker and Hussain Samad as the “biggest” so far, by which it seems to mean the one with the most households in it. Not so. The new study has 1,509–2,322 households depending on how you count (see the paper’s Table 1); as a counterexample, a study in Hyderabad, India, had about 6,850. The new study is however the longest, tracking families for a remarkable 20 years.
  • The article describes the positive results of this study as breaking a pattern of research finding “limited or no benefits” of microcredit. Actually, it continues a pattern. Shahid Khandker has authored many papers with similar conclusions: Pitt and Khandker (1998); Pitt and Khandker (2002); Pitt, Khandker, Chowdhury, and Millimet (2003); Khandker (2005); Pitt, Khandker, and Cartwright (2006); Pitt and Khandker (2012); Khandker and Samad (2013); Khandker, Samad, and Ali (2013)… The pattern continued after randomized studies began appearing in 2009 that produced the less-positive findings mentioned.
  • More fundamentally, the article evinces no understanding of why those randomized studies are more trustworthy (see below).
  • The article appears to commit what Dierdre McCloskey and Stephen Ziliak dub the “standard error of regressions,” which is to confuse statistical significance with real-world significance. Statistical significance, as meant here, is the certainty that the impact of microcredit is not zero. Real-world significance is whether the effect is big enough to matter. “A 10% increase in men’s borrowing raises household spending by 0.04%….Borrowing by women pushes up household spending by one and a half times as much.” Let’s see…because of compounding, seven 10% increases would about suffice to about double borrowing. So doubling female borrowing will lift household spending by \(7 \times 0.0 4\% \times 1.5 = 0.42\%\). To me, that seems small—about a sixth of the impact found in the first study of these families.

Alright, I guess that’s enough beating up on an anonymous journalist. The important question is what to make of the new study. I say: unfortunately, not much.

The issue is standard. Correlation is not causation. The paper makes a strikingly confident statement about how one thing affects another: “microcredit programs have continued to benefit the poor by raising household welfare.” The problem is that in families and villages, everything affects everything. Taking more microloans can make people wealthier or poorer. Being wealthier or poorer can make people take more microloans. The arrows go in circles. Statistics can measure correlations. How do we make the leap to causation?

The burden is on the researcher evincing such certainty to rule out competing explanations such as reverse causation. In this paper, that requires demonstrating and exploiting the equivalent of an experiment. If, for some arbitrary reason, certain families borrowed more than others, a comparison between them could be persuasive. The classic example is some being randomly offered microcredit. But that is not what happened in Bangladesh. Rather, the asserted basis for a “quasi-experiment” in this study is copied from Pitt and Khandker’s first analysis of these families, the one published in 1998. It is that only “landless” families (owning less than half an acre of land) could borrow, at least from the Grameen Bank.

As the launchpad for an experiment that reveals the impact of microcredit, there is a big problem with this splitting. As Jonathan Morduch showed in 1998, the half-acre rule wasn’t followed in practice. Lots of people with well more than half an acre borrowed. Of course, not all did. This leads to the strong possibility that the real basis for the split between borrowers and non-borrowers was differences in entrepreneurial talent (capacity to use credit productively) or demonstrated reliability (capacity to obtain credit). Khandker and Samad’s results may simply be telling us is that people who were more driven and trustworthy did a little better in life 20 years later.

Pitt and Khandker have never truly rebutted Jonathan’s criticism. In fact, Pitt (1999) wrote that the rule dividing borrowers from non-borrowers is “unknown.” Owning more or less than half an acre is asserted to be one factor, but there is no sign of that in the data, and it could easily be confounded by other factors such drive and reliability.

Unfortunately, the Khandker and Samad paper does not even address this issues; it does not shoulder the burden I described. It never defends the assumption that you need to believe in order to buy the paper’s conclusions. It is like asking you to accept a proof in geometry while avoiding scrutiny of the starting axioms about lines and angles.

The statement of the key assumption is at the top of page 15: “The necessary assumption is that the availability of a credit group by gender in a village is uncorrelated with the differenced household errors, ε, conditional on X.” To be fair, few journalists reading that could tell what is missing.

print

  • Asif Dowla

    Couple of quick reactions:

    1. The half-acre rule suggested that the land will be of medium quality. It is hard to implement it empirically because the quality of the land varies geographically. So, people borrowing with more than half an acre of land are actually not violating the rule; it is an adjustment for the quality of the land.

    2. You are suggesting an impossible condition for the proof that microfinance in Bangladesh works that it will never be able to meet–randomization. You cannot randomize in Bangladesh–you would not find any control group because the control group is contaminated. This means that this long standing debate between Khandker, Pitt and other authors and Mourduch and you will never be resolved.

    You guys should call it quits and move on..

  • David Roodman

    Asif, it is great to have you back, keeping me honest.

    I understand that the actual eligibility rule is more complicated than “half acre,” but that doesn’t change the fact that there is no statistical demonstration of quasi-experimental arbitrariness.

    And I agree the experiment-like standard is tough to meet in Bangladesh. So research like what Khandker has doing may be the best that can be done, and may well be worthwhile. But that does make the confidence in the Khandker & Samad conclusion more justified.

    –David

    • Asif Dowla

      Then isn’t it unfair to characterize this kind of research as “shoddy?”

      • David Roodman

        Asif, actually I believe I reserved that adjective for the journalism.

    • Asif Dowla

      I could never figure out from Pitt and Khandker (1998) and Morduch (1999), if they took into account of years of membership. Remember the half-acre cut off rule was an entry condition–you have to own less than half an acre of land to be member. However, over the years of membership, if the borrowers acquired more land that will be a testament to the success of microcredit, not an example of violation of eligibility rule.

      Since you agree with my earlier response that the eligibility rule had positive variance, your criticism that borrowers who had more than half an acre of land represents entrepreneurial ability is moot.

      • David Roodman

        Asif, the landholding variable they use is land ownership before starting microcredit. So in theory they’ve avoided that concern.

        No the point is not moot. You’ve offered one explanation for why borrowers would own more than half an acre. That does not disprove other explanations, such as the ones I put forward as possible.

        • Asif Dowla

          So, at the end of the day, it is an empirical question. I have not seen any study that prove that people who own slightly more than half an acre of land is more entrepreneurial than people who own slightly less than half an acre. An econometrician would say that the difference is not identifiable.

          • David Roodman

            Asif, I don’t think it’s just an empirical question, a matter of being more evidence. It’s also a question of what we are willing to assume in the absence of evidence. We don’t have the evidence to rule out competing explanations for the Khandker results. This is in contrast with the randomized studies. Until a strong case can be made against competing theories, I am unwilling to rule them out. Or to put that in less binary terms, the serious plausibility of alternative explanations, because of the lack of any clear, sharp line dividing borrowers from non-borrowers, makes this study’s causal interpretations less convincing.

  • Hi David,

    Great to have you back in the microfinance impact science world, as a voice of reason. Just help out my intuition here, please: is it correct to interpret the study as saying it takes 20 years of microlending, with for instance a $1,000 loan each year, to make a $42 difference in consumption at the end? I would agree that the difference appears too small to matter. All the sampling problems (yadda yadda) notwithstanding…

    Phil

    • David Roodman

      Hi Phil,

      Well…the full truth is that I think you can’t precisely answer that question from reading the study, unless I missed something. There are a couple of complexities here, which are very easy to miss.

      One is the definition of amount of credit. I believe, based on my examinations of the earlier generations of this work, that amount of credit is the simple sum of loans taken. So if a woman borrowed and repaid $1,000 20 times in 20 years, that would count as $20,000 of credit, not $1,000.

      The second is technical, having to do with logarithms. See my post http://www.cgdev.org/blog/logging-business, that tries to explain (and maybe fails miserably). The regression study elasticities which is not unusual, asking what is the % increase in household spending caused by a given % increase in microcredit borrowing. One disadvantage of this mathematical structure is that it assumes everyone uses microcredit: there are no zeroes. Because if you have 0 borrowing, even a gazillion % increase in your borrowing still leaves you at 0. This is why it is hard for me to answer your question about what happens to someone who goes from borrowing nothing to borrowing $1,000. In practice, Pitt and Khandker and coauthors, have assigned a very small non-zero amount to non-borrowers–usually 1 taka, or a few cents. But the results they get are sensitive to this undocumented choice, which is one reason Jonathan and I had trouble matching their results back in 2009.

      –David

      • Hi David,

        Thanks for this non-technical explication of this very technical stuff… and your well-documented overall patience in these matters.
        I think I can begin to see now why looging the variables may be more than just a statistical choice, in lumping (A) minimal microborrowers (“toe-dippers”) closer together with (B) not-quite-so-micro-borrowers (“million-taka men”) than with (C) non-borrowers (people who borrowed effectively just barely less than (A)).

        Another dimension of comparison perhaps better suited to the data would be to note that a 10% increase in borrowing raises the interest the borrower pays by 2.428% (1/10 of Grameen’s annual rate, http://www.cgdev.org/blog/quick-whats-grameen-banks-interest-rate ). So a female borrower borrowing 10% more would pay 2.428% more each year over 20 years, just to gain an 0.42% rise in income at the end. While statistically significant, this outcome could nonetheless make for fairly troubled cost-benefit calculations from the perspective of a borrower.
        Phil

        • David Roodman

          Hi Phil,

          Another good query, to which the answer is not at all obvious. As far as I can see, the definition of household spending used in the study excludes interest, i.e., is net of interest. The data for the 1998/99 survey round are at http://go.worldbank.org/E9WWFZIXJ0. The questionnaire on spending asks about dowry, education, food, etc., but not interest on loans. So my best understanding is that the spending variable in the regressions does not include interest either. In the bottom half of Table 10 of the original version of my replication with Jonathan of Khandker (2005), the predecessor to the present study, our reconstructed spending variable produced similar averages to those reported in Khandker, suggesting that my understanding of their definition of the variable is mostly right–though I must say the matches are not exact.

          –David

  • Hugh Sinclair

    David,

    Standing back a moment from the technicalities, it appears you are coming to a similar conclusion that I came to in my analysis of the Karlan et al. paper on Compartamos. It appears these authors have a strong pre-disposition to extract any positive result possible from their data. One can almost feel the frustration in the Compartamos study, as the authors produce similarly unimpressive results, and make as much as possible out of negligible positive conclusions, while brushing over rather obvious negative conclusions. I am not an academic, and don’t understand the process of peer-review and preventing such obvious bias from contaminating the conclusions of a paper. To cite just one example, Karlan has written extensive about microcredit. In his book “More than Good intentions”, he discusses it at length, but in the concluding chapter he proposes “Seven Ideas That Work” and microcredit doesn’t even get a mention!

    Do you think there are potentially incentives to promote such ideas as microcredit that are not supported by rigorous academic research? Could the funders behind universities, or grants for fieldwork, have an impact on the interpretation of the findings? Do academics face pressure, perhaps invisibly, to “spin” their research in certain directions? Have you ever faced such pressure (no need to mention names!). Obviously in the case of P&K, with a vintage of criticism and counter-criticism that you have been directly involved in, the over-arching question that I have, as an outsider and non-academic, is simply “why do they seem so intent on defending their initial pro-microcredit stance? And is this related to their work at, and the overarching agenda of, the World Bank?”.

    In the Karlan Compartamos paper, for example, they spend a huge amount of time physically in the desert of Northern Mexico. The project was extensive, cost a substantial amount in terms of both funding and time, and yet when you see the results, they are “non-dramatic” to say the least, as are the results you mention above. I entirely understand the frustration they must have felt. Faced with such mediocre results, they could either present them as “mediocre results”, or spin them in one direction or another. It is not that the research per se is poor, or the results are flawed, but rather that the subjective conclusions drawn do not match the (more) objective results presented. Is this a reasonable conclusion to arrive at? Is this the implied conclusion you are alluding to here? And why does peer-review not weed this out?

    • David Roodman

      Hugh,
      I see the two studies as quite different and am less interested than you appear to be in speculating on the researchers’ motives and incentives in the cases at hand. The Karlan, Zinman, and Angelucci Mexico study is in my view very well done. It is randomized. It represents a second generation in this literature in probing the impacts of microcredit on not just average outcomes but the distribution of outcomes. Thus the title, “Win Some, Lose Some?” And as that title suggests, it suggests a pretty mixed picture, if with somewhat stronger evidence of there being some winners than losers. I disagree that the paper’s interpretions go beyond its evidence.

      No, I have never felt pressure to bend what I say about microfinance.

      Of course many incentives are at play on researchers. I’m most ingested in what their research has to teach us about the world.
      David

  • Pingback: The Holy Grail of Microfinance Research? | Center for Financial Inclusion blog()