Outages, pandemics and the reengineering of traffic on the internet

NS1 EC-1 Part 4: Customer development

Sales in enterprise infrastructure is all about meeting a customer’s requirements. But what happens if customers don’t even realize they need a startup’s product in the first place?

In the first three parts of this EC-1, we looked at the origins of NS1, how the company built its DNS and DDI services, and how it competes in a hypercompetitive market. This fourth and final part is where the “rubber meets the road” — the actual customers. Without customers, there is no money, there is no growth and there is no NS1.

NS1 kicked off with the idea that DNS could be used for more than it was previously. The company’s technology has continued to evolve since its initial inception and now uses DNS as a leverage point to help organizations like Pinterest, Roblox and hundreds of others improve application delivery.

As we saw, the challenge for NS1 is that most customers are relatively content with their existing DNS service. It’s a space that has many offerings from legacy as well as public cloud providers, and for most customers, there’s never an urgent force pushing them to upend the fundamentals of their network.

NS1 has had a couple of lucky breaks. The first, as we will see with Pinterest and Roblox, was a major outage of one of the company’s largest competitors, DynDNS, back in October 2016. That failure brought NS1 immediate attention from network architects, turning a relatively staid layer of infrastructure into a critical area for re-architecting.

Beyond that outage, NS1 has been able to educate its customers on why a more flexible base for DNS is important for performance. We interviewed Roblox extensively to understand how the social gaming platform has engineered its systems in light of its feverish growth amidst the COVID-19 pandemic.

Finally, we will look at how NS1 continues to expand its product lineup, and what the future holds for the company. NS1 now serves more than 760 customers, including many of the world’s most trafficked applications, such as LinkedIn, Dropbox, JetBlue, Fox and The Guardian. It has become a major infrastructure leader, and an “unusual outcome” likely awaits.

One outage to topple them all

There are a lot of different reasons why any enterprise or organization would choose to buy services from NS1.

Every business and internet user on the planet already has access to DNS in one form or another. NS1 doesn’t sell merely DNS services, as the company repeated time and again in every interview and briefing, but instead sells network resilience and performance. Sure, DNS “just works” in many cases, but “good enough” isn’t good enough when microseconds count and users demand and expect to always be connected at the best possible speed.

In 2017, Roblox was using DynDNS for managed DNS services. When Principal Traffic Engineer Adam Mills joined the gaming company, Dan Williams, VP of Corporate and Production Engineering at Roblox, suggested they change to NS1, since he had seen it in production at his previous employer Dropbox. The October 2016 DynDNS outage was not the primary motivation for the company’s transition, though it was a factor. The big issue, according to Mills, was the actual technology itself.

David Baszucki, founder and CEO of Roblox - Roblox Developer Conference 2019

David Baszucki, founder and CEO of Roblox. Image Credits: Ian Tuttle/Getty Images for Roblox

“It wasn’t great. There was no amazing API framework and there wasn’t a way to dynamically configure the things they had,” Mills said of his experience with DynDNS. “It was a little antiquated.”

In his view, DynDNS back in 2017 was little more than a basic DNS service that connected traffic to locations without any other capabilities. Roblox wanted to use DNS as a key component of its network operations to direct users to the best servers according to their location, with geotargeting as a key goal. That would improve game performance and therefore the user experience.

Roblox already had data centers in major regions with an active user base, enabling connections as close to players as possible. What it needed was a more intelligent layer for determining how to connect a user to its platform. Roblox built its own infrastructure stack for the game platform, but building its own DNS service was not a game Mills had any interest in playing.

“We’re a video game company, so we really weren’t going to do DNS,” Mills said. “I mean, at the time the traffic engineering team was basically two people, so we weren’t going to be deploying our own DNS implementation to solve this problem.” NS1 provided the necessary APIs and performance while allowing the traffic team to optimize performance to the platform’s needs.

The DynDNS outage would prove to be the gift that kept on giving for NS1, and it was an even stronger impetus for change at social networking giant Pinterest. Scott Beardsley, engineering manager at Pinterest, explained that after the outage, it became painfully obvious that there was a need to have a secondary authoritative DNS provider to ensure the company’s uptime performance. Pinterest chose NS1.

Ben Silbermann, co-founder and CEO of Pinterest at TechCrunch Disrupt SF 2017. Image Credits: TechCrunch

Beardsley referred to the initial engagement as providing “dumb DNS” — basic managed DNS that helps route traffic to the right location. Over the last five years, however, Pinterest has moved beyond that simple use case to adopt and benefit from NS1’s Pulsar product, which allows a traffic engineering team to adjust where traffic demand goes algorithmically and in real time.

“Pinterest has a lot of content, as you can imagine, and serving content can be expensive,” Beardsley explained. “So we needed a way to optimize some of our costs by routing to multiple CDN vendors.” With Pulsar, Beardsley said that Pinterest has a feedback loop from users on how they are experiencing the social network so it can optimize traffic delivery across its CDN providers for availability, performance and cost.

How Roblox used traffic engineering to thrive during last year’s growth spike

When the COVID-19 pandemic first hit in early 2020, kids (and plenty of adults) around the world found themselves at home and with a lot more time to play games. For Mills, it was also game time: to prove that the online game creation platform could survive a deluge of traffic that it had not planned for.

“We were about to get hit with millions of new players and kids just staying at home that had more available free time,” he said. “We were going to have capacity constraints as we had already forecasted for the year, and we had not put this at all in our forecast.”

So how did Roblox handle the traffic? It was a combination of DNS traffic steering from NS1 and highly performant infrastructure that Roblox itself built over several years.

The actual game application and platform itself had been evolving since at least 2017 when Roblox built its own infrastructure platform, led by Williams (whose story is well told in another Extra Crunch article). Mills joined at about the same time as Williams and helped to lead the traffic engineering efforts.

Traffic engineering is a particularly vital function at an online gaming company. There are a lot of different things that make a game fast, including the processing power of the gaming server, connection speeds, graphics capabilities as well as myriad other factors. But at the most basic level, the first step for any online game is actually being able to connect.

“You can easily plot player engagement time to how fast the website is — it’s a one-to-one ratio,” Mills said. “If the website loads fast, people stay engaged longer.”

Keeping a nine-year-old engaged is a key motivation for Mills and his team at Roblox. The company’s gaming platform tends to skew toward younger players and Mills himself noted he has kids of his own that enjoy playing Roblox.

“What ends up happening with a nine-year-old, as does with anybody, is if it doesn’t load, they’ll just go someplace else; they’ll immediately pivot to the other game,” he said. “Reliability is important even for a nine-year-old.”

A key part of what helps Roblox with its traffic optimization is NS1’s Pulsar Active Traffic Steering technology, which I described in detail back in part two.

Mills explained that Roblox directly gathers its own latency metrics from players’ browsers as they fetch game assets and connect to the platform. The combination of the company’s user data and broader internet connection speed data from NS1 affords a very accurate picture of performance for any user. Pulsar then uses all the available data to direct users to the best possible server per their location.

Figuring out some of the capabilities of NS1, and Pulsar in particular, involved a lot of trial and error. That wasn’t necessarily a bad thing, as Mills said he wasn’t particularly fond of reading documentation and needed to figure out how things worked for himself. To NS1’s credit, he noted that whenever there was an item that didn’t quite work as he wanted or a capability that Roblox needed, NS1 was responsive in helping to figure things out.

“I said, ‘Hey, this is what I found as a customer,’ and they’re like, ‘Oh, well, we’ll fix that’ and then like, you know, three or four weeks later they came back and said, ‘Okay, we made a new version of the code that does what you want it to do,’” Mills said.

Aside from figuring out how things work, he also had to figure out what didn’t work. Back in 2017, Roblox initially started out running the NS1 Managed DNS service. After three years of solid service, Mills and his team decided it was time to have a secondary DNS provider, to help enable resiliency. “It’s not like we don’t like NS1 or think that they’re doing a bad job, but sometimes in some regions, everybody has an outage — it happens,” Mills said.

That caution proved to be well warranted. In 2019, Mills said that Roblox had an outage in São Paulo, Brazil, with NS1. His team opened trouble tickets with NS1 and they were able to respond, but it took time.

DNS redundancy is critical, and that’s where NS1’s Dedicated DNS comes in. The service provides a set of servers on a separate network, and also pushes updates with a different deploy cycle than NS1’s Managed DNS service. Mills said that the model means it’s highly unlikely that both NS1’s Managed and Dedicated DNS would ever both be broken at the same time.

As to why Roblox didn’t choose a different DNS altogether for a secondary, Mills was unapologetic in stating that he just couldn’t find any other provider that had the same traffic steering capabilities.

“Google has PoPs (points of presence) and they have tendrils in every single market, but as a startup company, you’re not going to have that infrastructure [on] day one,” Mills said. “We didn’t have that infrastructure three years ago and we still don’t have that infrastructure today [ … ] being able to provide DNS and in a robust way and having a company like NS1 that offsets that engineering effort to speed up your connections is great.”

Finally, Mills noted that one key attribute of NS1 is its ability to play nice with the rest of Roblox’s tech stack. After all, DNS and traffic steering is just one part of a larger set of tools that help to make the game platform work. Among the other technologies that Roblox uses are the open-source Ansible configuration management platform as well as Hashicorp tools such as the Nomad application workload scheduling system and the Consul service networking platform. Mills uses Ansible to help configure and automate deployments, while the Hashicorp technologies are used on Roblox’s infrastructure to manage applications.

How NS1 is routing its own future

NS1 kicked off with the idea that DNS could be used for more than it was previously. The company’s technology has continued to evolve since its initial inception and now uses DNS as a leverage point to help organizations like Pinterest, Roblox and hundreds of others improve application delivery.

At this stage of its growth, NS1 founder Kris Beevers says he’s not looking for more funding, at least not yet. Whether an IPO or an acquisition is in the cards is also unknown at this point.

Raj Dutt, the first investor in NS1, former CEO of Voxel and current CEO of Grafana, isn’t worried about whether the startup goes public. “My hope is that they just continue to make decisions that ensure that they’re going to be a long-term, viable, sustainable, dominant company in their space,” he said. “I’m not sitting here being like, ‘Man, I wish that there was an exit for my investment’. … I’m just like — play the long game Kris, that’s what you should do, that’s what I want you to do.”

Dell Capital’s Tyler Jewell hasn’t put an IPO date for NS1 on his calendar either, though he did specifically tell TechCrunch that he’s expecting an “unusual outcome” for NS1. That basically means an exit where investors get a significant return either via an acquisition or IPO. “Our job here is to capture as much of the opportunity as possible and to just stay focused on that, and then all the other stuff will happen, when and how that should happen,” he said.

Regardless of the future public or private status of NS1, there continues to be opportunities for product expansion and riding secular growth trends.

The company last month announced NS1 Connect, a new platform meant to integrate and bring its services together in a more integrated manner. Users will be able to manage multiple NS1 services from a common interface, which makes for a setup that will also provide new cross-selling opportunities.

Jewell said he sees increasing demand as the volume of video collaboration in 4K as well as augmented reality (AR) and virtual reality (VR) is likely to grow in the coming years. All that workload, like all other internet workloads across the entire history of the internet, will rely on DNS to connect. NS1 and its backers are also optimistic about the continued growth of Internet of Things (IoT) as devices proliferate at the edge of the network.

As the company moves forward, what’s next isn’t necessarily about taking a dramatic leap, but rather about continuing to iterate, experiment and adapt as it has since day one.

“One of the things that I really believe in business, life and everything else is, you try something, you get some data, you create the feedback loop, and you iterate on that for what you’re trying next,” Beevers said.

Over the last nine years, since he first scribbled down the idea behind NS1 on a napkin in a NYC bar, Beevers has seen a lot as the company he founded moved from being an idea to being a foundational substrate for the internet.

“There’s a million things that I would do differently with the benefit of all the data that we have from what we’ve learned in the last seven or eight years of business. But we didn’t have the data,” Beevers reflected. “What I am proud of is the way that we continue, eight years into our business, to experiment and tinker the same way as I did with the first 22 lines of code that I wrote on the couch.”


NS1 EC-1 Table of Contents

Also check out other EC-1s on Extra Crunch.