Recently, at a Center for Global Development conference, attendees criticized a wide swath of development programs that, these experts say, provide no evidence they are achieving their objectives.

NPR author Nurith Aizenman provided a nice summary of their conclusions in an article titled “How Do We Know If Aid Really Works? Turns Out …We Often Don’t.” Attendees recommended increasing “impact evaluations” of development programs and incorporating the findings to revise and direct future efforts.

Aizenman compares this to what an entrepreneur would do: “Before spending big bucks on a massive effort to improve life for the world’s poorest … you should run a smaller scale test to make sure the idea actually works.”

Comparing development programs to entrepreneurial activity gives insight into a more troubling problem. Let’s say an entrepreneur, Kaitlyn, comes up with an idea to put Cap’n Crunch cereal on a donut with extra sprinkles and candy bits. She makes a few for her friends, and they can’t stop talking about how amazing they are. “You should sell these. People would line up for blocks to get one!”

After more prodding from her friends and some careful planning, Kaitlyn sets up a small shop and starts creating donut masterpieces in hopes of gaining enough loyal customers to make her enterprise profitable. Is she using resources in a way people value? She’ll soon know if people choose to exchange their hard earned money for her creations.

Competition and Discovery

After noticing the lines outside Kaitlyn’s shop, Shaun comes up with an even better idea — he makes a milkshake and tops it with a cereal and candy coated donut! A few weeks later, he opens his store down the road.

What is he about to discover? Soon, he’ll know if people value his creation over Kaitlyn’s, or perhaps, different people prefer his, while others prefer Kaitlyn’s. Possibly, no one thinks his idea is any good, and he’ll go out of business. How will he learn this? Potential customers will choose to buy or not buy his loaded milkshakes, which indicates how valuable his efforts are to them.

Now imagine this same set of potential customers earned so little that they chose to not spend their money on any desserts. Suppose their kind-hearted mayor, seeing the plight of his electorate, chose to use the city’s tax revenues to provide every citizen with a loaded milkshake. Has the mayor increased the quality of life? It’s very hard to tell.

Perhaps recent studies have shown that increased consumption of Shaun’s loaded milkshakes increases self-reported happiness. That sounds good. But could the mayor have done better? Maybe. The people might have preferred gourmet donuts, or seven-layer cakes, or small batch, hand-churned ice cream. The possibilities are endless.

Impact Evaluations

So was the mayor’s plan effective? If we define the objective as an increase in the number of loaded milkshakes consumed and a corresponding increase in happiness, we would conclude yes. However, if the objective is to increase happiness as much as possible, given the same resource constraints, it’s effectively impossible to measure. There are a nearly infinite number of ways our resources could be combined.

Even if we assume that every person prefers the same dessert as everyone else and that their tastes will never change, we still need to test out all current dessert options to find the one that increases happiness most, and continue the comparison as new delectables are developed.

People in poverty don’t have the same set of choices as those with higher incomes. Many lack sufficient resources to provide for basic things like a bed, clean water, and medicine. Development experts that aim to alleviate poverty with targeted aid face an impossible task of choosing between countless ways to use their resources. Should efforts be directed towards bed nets, or micro loans, or clean water, or vaccinations, or education, or … the list has no end.

The best that “impact evaluations” can do is show specific results of a particular intervention. For instance, distributing zero-priced bed nets decreases incidence of malaria, but they cannot show that this particular use of resources is the best way to increase the quality of life for those they seek to help. But worse, by directing resources towards a potentially inferior solution, they may be impeding the discovery of another idea that could provide much more help. A call for more “impact evaluations” directs focus from the larger issue of eradicating poverty to narrowly defined objectives and, most importantly, turns a blind eye to the significant opportunity cost that may be incurred.