Measuring results and evaluating tradeoffs quantitatively

Written by Michael Thomas (Social Hackers Academy)

Goal of this chapter

• Create metrics to measure your school’s effectiveness at achieving its mission
• Develop a mental model for making hard decisions and tradeoffs Apply rational thinking to emotional situations

Summary of our learnings

• When deciding how money should be spent it can be helpful to chose a single point of optimization like Cost per Placement.
• A school should optimize for small investments that have disproportionate returns

Introduction

In the world of business measuring success is simple: a company exists to deliver shareholder value; therefore every decision should be made in order to optimize profit. For a non-profit or social enterprise things aren’t always so clear. But still, there are frameworks and mental models that can be helpful in thinking about how to optimize time and resources.

How to look at impact

It’s safe to say that the goal of a code school for refugees is to graduate and place as many people at well-paying jobs as possible, with the prospect of a sustainable career. Given that capital is generally limited—especially in the early days—the goal is to do so with as little money as possible. That’s why the metric, or Key Performance Indicator (KPI) that I recommend schools optimize for is the cost per placement.

It’s safe to assume most code schools exist to get students a job, not merely to educate them. In other words, education is the vehicle and employment is the destination. Secondly, many of the tradeoffs that a school must make—for example, whether or not to provide housing or cater meals—impact the graduation and placement rate. Therefore when considering them it's important to ask will this increase our placement rate, and if so, by how much? Of course, it’s impossible to predict the future or know whether or not an individual activity will increase or decrease the graduation rate. But as a mental model I find it helpful to have a single point of optimization.

As an example let’s say that you are deciding whether or not to offer students housing. This would obviously come with many benefits to the students’ lives and to the school’s performance. But so would a full-time instructor, and three catered meals per day, and a therapist that could help them deal with stress and anxiety. Because time and money are limited an organization must choose what services to provide students. In other words they must make tradeoffs. But often this can feel like comparing apples to oranges, which is why it’s important to have a single point of optimization.

Let’s say that housing costs €200 per month per student (as it did in Greece when we started our school). With 10 students going to school for 6 months that adds a cost of €12,000.

Now let’s say that all the other costs to run a school equaled the same amount. The question to answer is this: will doubling the cost of the program double the number of students placed? If not there’s an argument to be made that housing is inefficient because it increases the cost per placement.

You can run the same math on any time or money investment. Let’s say that you find a therapist willing to come to the school and volunteer, but he or she needs her transportation cost subsidized at €100 euros per month. You might hypothesize that having someone to help students suffering from depression, anxiety or Post-Traumatic Stress Disorder (PTSD) will prevent at least one high performing student from dropping out. Using the same €12,000 euro budget for 10 students example, you’d be able to increase your placement rate by 10% for a 5% increase in money spent, which would reduce your cost per placement.

Of course this is a framework and not a rule. In your first semester you’ll only be able to hypothesize what impact certain decisions will make. And even if you determine that something like housing increases your cost per placement you may still want to offer it, simply because it will improve your students lives. The point of this framework is to evaluate decisions rationally in a field overrun by emotions.

Here is a Google Sheet template you can use to think through similar tradeoffs.

Much of this framework was borrowed from the concept of quality-adjusted life-year (QALY) and disability-adjusted life year (DALY). For a brief explanation of this concept see this video: