I’ve avoided reading and writing about the New York Times’ Nicholas Kristof lately as I’ve really been trying to get the stress in my life under control. But this merits a response as apparently somebody still doesn’t understand the difference between evidence and anecdotes.
Here’s a quick rundown on what happened.
Kristof wrote an article extolling the virtues of do-it-yourself (DIY) aid projects, in which amateurs circumvent large aid agencies to implement programs on their own.
Dave Algoso wrote a very measured and kind response pointing out that, actually, aid is a very difficult profession and not one that amateurs are equipped to undertake and do well.
Another excellent response was posted at Tales from the Hood in which it was noted that aid is not about us while DIY aid is often all about the doer.
Kristof responded to this barrage of criticism in a blog post.
He wrote: “Compared to professionals, amateurs tend to be more, well, amateurish. Accountability can be a real problem. But on the whole, I think this concern is misstated. I’ve generally found that grassroots, locally owned aid projects have a better record than large scale, top-down ones that don’t always have the same buy-in. And the truth is that DIY aid projects are more likely to be modest, grassroots efforts undertaken with strong local partners. They often keep their ear to the ground and tinker with their model more than the larger projects. Aid projects often succeed at the experimental level and then have difficulty going to scale, but that’s less of an issue with DIY investments that are never meant to scale up (that’s a separate problem with them, and a legitimate concern).”
What I want to focus on is the problem with Kristof’s claims in terms of research methods – namely that there’s not any evidence for his claims.
“I’ve generally found that grassroots, locally owned aid projects have a better record than large scale, top-down ones that don’t always have the same buy-in” may be true based on Kristof’s limited exposure to local aid projects. But it’s not one that’s supported by any systematically gathered evidence that I’ve seen.
Has anyone else?
The problem here is that Kristof is relying upon anecdotal evidence (the nongovernmental organizations he has encountered) rather than systematically gathered evidence. Even though Kristof has more anecdotes than your average observer, it’s still not evidence.
Why is this problematic?
Because there are exceptions to every rule. In the social sciences, we call these exceptions “outliers.” You can’t base a theory on outliers because then you’d be wrongly explaining general phenomena based on an unusual case. Because of this, we generally place outliers in what is called the “error term.”
The error term is kind of like the remainder in a long division or algebra problem. We leave those cases out of our studies in order to avoid tainting our results. We do so in order to get the right answer – the one that explains what happens most of the time under given conditions.
Because Kristof’s only research method is his personal observation, we can’t be sure that he’s not simply making general claims on outliers. He’s not using data. He’s using anecdotes. And anecdotes are a slippery slope on which to base claims about the kind of aid work that will best aid the world’s poor.
Also, I have to point out that there’s a huge difference between “grassroots, locally owned aid projects” and the sort of DIY aid projects conceived and executed by well-meaning foreigners.
I actually think he’s probably right that the former work much better than many other projects because they’re grounded in the community. But that’s not what Kristof wrote about in the piece. His article was entirely about Whites in Shining Armor, not grassroots, locally-conceived projects. And there’s no evidence that I know of that shows that projects conceived by well-meaning, idealistic foreigners work better than professional, NGO-supported aid.
I met Kristof once in the eastern Democratic Republic of the Congo. Based on that one encounter, I could claim that Kristof’s standard research method is to go to the best local NGO a city has to offer and then to take the word of a few officials at major international NGOs and United Nations agencies as truth.
But I can’t make the claim that that’s how he always works.
Why? Because I only have one case. And that case could be an outlier. It could be that Kristof was having a bad day, was scared to death of the eastern DRC or had accidentally drank the tap water at the Ihusi Hotel. I don’t know.
What I do know is that we shouldn’t be making decisions based upon unreliable, anecdotal evidence. And if you want to be an aid worker, you’d better know what you’re doing.
Laura Seay is an assistant professor of political science at Morehouse College in Atlanta. This column was adapted from her blog, Texas in Africa. She did fieldwork from 2005-07 related to the Congo for her doctoral dissertation, “Authority at ‘Twilight:’ Civil Society, Social Services and the State in the Eastern Democratic Republic of Congo.”
Laura Seay is an Assistant Professor of Government at Colby College. She studies African politics, conflict and development, with a focus on central Africa.