Multi-Factor Analysis Of Y Combinator’s Summer 2014 Class

Editor’s note: Matt Oguz is managing director of Palo Alto Venture Science.

In my previous article, I wrote about how multi-factor analysis of startups helps avoid decision biases such as availability — anchoring or the famous “herd mentality.” The merits of the theories behind this approach go back to the 1950’s. But based on the feedback our team received, we felt encouraged to share a sample analysis with everyone to explain the inner workings of the process. With that in mind, we ran the Y Combinator Summer ’14 class through our models.

First, let’s take a look at the methodology to refresh our memory. The idea behind Multiple Criteria Decision Analysis (MCDA) is to instill transparency and consistency to the process of picking alternatives among an available batch. Professors Belton and Stewart state the following in their book, MCDA – An Integrated Approach: “Subjectivity is inherent in all decision making. MCDA does not dispel that subjectivity; it simply seeks to make the need for subjective judgments explicit and the process transparent.” A view shared by many others prominent in the field of MCDA is illustrated by the following quotes: 

Simply stated, the major role of formal analysis is to promote good decision-making. A good analysis should illuminate controversy – to find out where basic differences exist in values and uncertainties, to facilitate compromise, to increase the level of debate and to undercut rhetoric.” Keeney and Raiffa, 1972

We’ll share our results for only 15 startups here, which we named “batch one.” First, let’s go over the methodology. Prior to starting to evaluate all startups, we need to weigh the categories against one another to calculate their importance in the decision problem:

Oguz_YC1

From here, we need to determine how to evaluate each startup based on all the criteria. To do that, we can assign numerical values, qualitative scales or utility functions to a given criterion. In this analysis, to make things easier, we defined qualitative scales that correspond to scores:

Oguz_YC2

Next, we need to determine how we’re going to evaluate each criterion. Market size, for example, is a dimension to be maximized. Competition is to be minimized. We assign the appropriate qualitative scale to each criterion. The figure below shows the weighting of categories and criteria and how we assign parameters.

Oguz_YC3

Once we establish the framework comes the fun part of evaluating each startup. This is subjective as mentioned before. Due to space constraints, we’re only able to show four startups, but the rest are evaluated the same way:

Oguz_YC4

After all the startups are evaluated, we calculate scores that captures each startup’s attractiveness (or risk) levels based on their perceived scores and the weights of these scores. This process is a lot more powerful than one member of the deciding team evaluating the startups. That way, if someone missed an attribute, others will pick it up.

But it is important to perform the evaluations without collaborating; otherwise, a number of biases can arise. The group can decide on the importance of each criterion, but each member should evaluate the alternatives on his or her own.

The rankings

Oguz_YC5

We’re all naturally drawn to those companies that top the charts. And out of this batch of 15 startups, Naytev, PicnicHealth and One Codex emerge as the top three. We noticed that even though data, big data or analytics weren’t part of our criteria, all three startups are data plays.

The spider web charting is useful in visualizing the strengths and weaknesses of each startup.

Oguz_YC6

Oguz_YC7

It’s worthwhile to examine the contribution of each criterion to the overall score. This visualization helps with that analysis. Note how the Technology/Product dimension impacts the overall score.

There are two other parameters we didn’t cover here because we didn’t have all the information: Pre-money valuations and the amounts being raised by each startup. Naturally, these two qualifications are very important in order to calculate how much “bang for the buck” one can expect. There are different ways to attack this problem.

One method is to treat these monetary parameters as part of the criteria set and assign them a percentage, as we did with all the other parameters. The second option is to factor them in later as a standalone parameter and compare them to market averages.

Another dimension we didn’t include in the analysis is “popularity.” We believe that both social proof or popularity are irrelevant when it comes to evaluating the potential of a startup, which is ultimately determined by how the market reacts to the value offering. Some VCs consider “who else invested in this” as an important parameter and label opportunities as “hot deals.” We believe that such an approach increases a number of decision biases with which most of us are familiar at this time.

It’s not easy to score attributes like ethics, energy or leadership without an in-depth analysis of a startup. But, given that this group of startups is part of a great program, we weighed those attributes heavily. However, we did wonder if that was a bias. (It is.)

This analysis helps us structure the problem, provides a focus and a language, and establishes an audit trail. It doesn’t seek to replace intuition, but it complements and challenges it. Visualizing the attributes that impact the economic merits of each startup is more important than any single element, such as the type of problem the startup tries to solve.

In this batch, there were quite a few that aimed at big problems. Issues such as legal and regulatory risk and state of technology (concept, MVP, Alpha, Live 1.0, etc.) are important to consider when attacking big problems.

We noticed that this batch of YC startups spans a variety of subverticals. The startups attack problems with varying economic and social impact. You can see the path to monetization much clearly in some of them and not so easily in others. Which ones will emerge as big winners? It’s too soon to tell.

There are three ways to invest in this batch. If you have lots of capital to deploy and are not concerned about early losses, you can invest in a lot of them at the same time. If you’re a thrill-seeker, then you can pick one of the ones you like the most and bet big. If you’re an advocate of a rational investment methodology, the multi-factor framework presented here is a great starting point. In conclusion, we wish good luck to all startups and those who bet in their success.