Teach yourself growth marketing: How to perform growth experimentation through A/B testing

Without customers, there can be no business. So how do you drive new customers to your startup and keep existing customers engaged? The answer is simple: Growth marketing.

As a growth marketer who has honed this craft for the past decade, I’ve been exposed to countless courses, and I can confidently attest that doing the work is the best way to learn the skills to excel in this profession.

I am not saying you need to immediately join a Series A startup or land a growth marketing role at a large corporation. Instead, I have broken down how you can teach yourself growth marketing in five easy steps:

1. Setting up a landing page.
2. Launching a paid acquisition channel.
3. Booting up an email marketing campaign.
4. A/B test growth experimentation.
5. Deciding which metrics matter most for your startup.

In this fourth part of my five-part series, I’ll take you through a few standard A/B tests to begin with, then show which tests to prioritize once you have assembled a large enough list. Finally, I’ll explain how to run these tests with minimal external interference. For the entirety of this series, we will assume we are working on a direct-to-consumer (DTC) athletic supplement brand.

A crucial difference between typical advertising programs and growth marketing is that the latter employs heavy data-driven experimentation fueled by hypotheses. Let’s cover growth experimentation in the form of A/B testing.

How to properly do A/B tests

A/B testing, or split testing, is the process of sending traffic to two variants of something at the same time and analyzing which performs best.

In fact, there are hundreds of different ways to invalidate an A/B test and I’ve witnessed most of them while consulting for smaller startups. During my tenure leading the expansion of rider growth at Uber, we used advanced internal tooling simply to ensure that tests we performed ran almost perfectly. One of these tools was a campaign name generator that would keep naming consistent so that we could analyze accurate data when the tests had concluded.

Some important factors to consider when running A/B tests:

• Do not run tests with multiple variables.
• Ensure traffic is being split correctly.
• Set a metric that is being measured.

The most common reason for tests getting invalidated is confounding variables. At times it isn’t obvious, but even testing different creatives in two campaigns that have different bids can skew results. When setting up your first A/B test, ensure there’s only one difference between the two email campaigns or datasets being tested.

For example, if you want to test whether emojis perform well in your email subject lines, only add an emoji to one sample and change nothing else in the rest of the copy between the two variants. After you’ve selected which variable to test, you’ll want to confirm traffic is being split correctly and evenly between variants.

Email platforms like Mailchimp will have tools to help split email traffic evenly, but if you’re running a test on a channel like Facebook, the easiest way to split traffic is to manually separate the recipients while keeping the budget even.

Finally, make sure that you have a metric to measure the success of your test. If you’re testing subject lines in an email, the correct metric would be the email open rate. By determining this metric at the start, you will make picking a winner much faster once the data starts to come in.

You may also consider testing some secondary metrics and looking further down the funnel. For example, if you’re testing ad creatives and produce a click-bait asset, it may boost the click-through-rate of the ad, but the conversion rate down-funnel may suffer in comparison to the control assets. It is important to consider secondary metrics and not always rely on a single metric for measuring impact.

Start with these tests

If we were brainstorming the types of tests to run for an athletic supplement brand, I’d begin with a quick, basic list like this one:

• Emoji versus no emoji in the email subject line.
• Text-only email versus header image in the email.
• Value proposition 1 versus value proposition 2 in ad creative.

While these are basic tests, I’d recommend them as a starting point because these tests won’t contain confounding variables and can be set up easily. For our athletic supplement brand, here’s an example of what an emoji subject line test could look like:

• Control: Get your greens in one pill
• Test: Get your greens in one pill 💊

To showcase what a deceivingly straightforward test would look like, we will use a test to measure how well ad creatives featuring men perform compared to those featuring women. To control for confounding variables in this test, we would need to make sure that both the male and female actors shoot their videos in the same spot, and the script has minimal differences. We’d also need to control for many more potential variables.

This is an example of a test you’d likely run with more than one actor for each creative before deciding which segment performs best. It’s not a great first test unless you’re feeling the need for a challenge.

Selecting winners

Now for the fun part! It’s time to analyze and select your test winners.

You want to make sure you have achieved statistical significance (stat-sig) before determining a winner. Stat-sig tells us whether our result is due to chance or if we have a consistent winner.

There are many stat-sig calculators on the web. I use Neil Patel’s calculator.

If you ran the email subject line test and are measuring the open rate metric, enter your sends and opens in a stat-sig calculator to determine if you can determine a winner. When I was working at Coinbase, we were comfortable with a minimum confidence level of 70%-80%.

Prioritizing growth tests

It’s common to have a plethora of ideas once you launch your first A/B test. That’s a great problem to have.

When I was leading fleet acquisition at Postmates, I had to constantly prioritize which tests we’d run due to the abundance of ideas and growth mediums we were examining. When a large list of good ideas is paired with the limited bandwidth of a startup, the need for thoughtful prioritization becomes paramount.

A quick eyeball test often works when ranking a list of tests, but if you’d prefer something more methodical, consider the RICE (Reach, Impact, Confidence and Effort) approach.

In the example above, Test 3 has the highest RICE score. This was calculated by multiplying Reach, Impact and Confidence, and then dividing the result by Effort. This means Test 3 should be the first test to conduct given its high impact and the low effort required for launch.

One good habit to develop is to maintain a repository sheet that tracks all your performed tests and their results. This sheet can serve as an information bank if needed later.

Now that you have a few growth tests under your belt, we’ll next learn how to identify which metrics matter the most for your startup in the final part of this series.

Teach yourself growth marketing: Which metrics really matter?