Just a year ago, Alex Roetter took over Twitter’s engineering activities in an executive shuffle. And a year later, Twitter’s now in the middle of finding a new CEO.
But that hasn’t stopped the company from continuing to following the strategy former CEO Dick Costolo and Roetter put in place to continue building out the company’s engineering team and launch new products. It’s a pretty simple approach — hire some of the best engineers and put them in small, agile teams with little top-down structure, Roetter said.
Twitter itself is a complex problem filled with tons of information coming in all at once. The company has to sift through all that data, surface up the best information, and amid all that figure out new ways to make it easier and more approachable for new users. That in of itself is an interesting engineering problem and, in fact, the CEO transition also hasn’t given Twitter any problems bringing in new engineers either, Roetter said.
We caught up with Roetter this week to hear a bit more about where Twitter’s engineering team is and a little bit more about the atmosphere within Twitter right now. Below is a lightly edited transcript of the conversation.
TC: Where is Twitter now in terms of its engineering focus and priorities? What are your priorities?
Alex Roetter: I’ve had this job for about a year, what I’m maniacally focused on is building an environment where world-class engineers can do the best they’ve done in their careers. One way you know it’s working, is we’ve seen a massive acceleration of the pace of execution. We’re seeing a bunch of signs of that so far, but of course there’s more to do. One of the ways as well that, you build a stable culture where you give them basically a really stable vision, really strong teams, and the tools to get their jobs done.
Twitter, as you know, has this total audience strategy. Twitter has really great, broadly appealing, universally interesting content. However, today you have to use the app to get that content, and the Twitter of the future is much broader. What we mean is, we reach everyone wherever they happen to be. You might be logged in, you might see it through a third-party app, through a syndicate site — having a stable strategy is an important part of executing on that vision with those teams. That’s what we started working on last year.
TC: What’s the archetype for engineers that you are looking for in your teams?
AR: The model is, I try to find great people and empower them with the bare minimum of top-down structure they need. Top-down [structure] is great for things like setting a clear direction and explaining what’s important to the company. Other than that, you really want to use [a] top-down [structure] as sparingly as possible. We try to build teams that thrive in an environment with a clear direction, but the details are unspecified. Teams are able to try a bunch of things and fail fast, and hold themselves accountable and measure themselves rigorously. We find that smaller teams are faster, we try to build them in that sort of environment. When there’s no top-down direction around the specific things they can try, they’re left to do bottoms up.
As a result of that being our philosophy, we look for people who are extremely technically strong, people who run in and fill in the gaps. When there’s an unspecified problem, you run in and make it work. You don’t look at it and say, ‘I’m paralyzed’. We look at people with side projects, open source projects, school projects, and look and see if they’re a cultural fit.
TC: How do you reconcile the need to bring relevant content to more casual users with some users still wanting a reverse chronological timeline as their primary interface?
Every time you open Twitter, we should show the best stuff to you. Alex Roetter
AR: I don’t think they’re mutually exclusive. So, it’s a continuum. For people today who have curated their follower lists and open the app really frequently, Twitter works really well. We can see the app and how often they return. For those people we don’t need to make dramatic changes.
But the other end of the spectrum are people we know relatively little about, and the app isn’t as effective as it could be. Every time you open Twitter, we should show the best stuff to you. We’re gonna try different fits on different parts of that spectrum, we think there’s room for both of those to exist. We’ll learn by asking them more targeted questions, observing their behavior. For core users, we don’t want to change much, it’s a good product. For this new wave, we can do a lot better, but to get to it you have to dig in to it, know the right moment to open your phone.
TC: What are the technical challenges for something like that?
AR: Twitter is a giant info retrieval problem. Here’s an example: ‘While You Were Away’ is a special case of that. I know what you’ve historically done on twitter, what the other features you’ve used, and we need to instantly score all the content that I can possibly show you, and rank it, and get it out, and get feedback in real-time. When you dismiss [a tweet], we ask if it’s helpful. But expanding that further, the tech challenge gets harder. Every time you get to twitter, pull something out of your phone, think of Twitter as immediately considering the content on twitter, your history, the content on the internet, and digesting it and figuring out what to get to you. It has to work globally and be fault tolerant.
TC: I haven’t seen the Fail Whale all that much recently. How have you guys attacked that problem?
AR: I’m glad you’ve noticed. It’s a large distributed systems problem. We split the app into a bunch of independent services that can talk to each other, and then run a bunch of tests. If anything fails, it degrade gracefully. ‘I can do what I can, but omit one thing when the service fails’ rather than the service just falling over entirely. Twitter’s very bursty, right — there are events on Twitter that are double sigma events, orders of magnitude heavier. We do load testing, we’re auto injecting failures into the system, so we know we’ll be able to handle them. We take things down, we split the system up. It’s a lot of work but it is paying off.
TC: And what are you guys working on, on that front?
AR: Better ways to handle spikes, better ways to be able to slowly make the experience a little worse for users under cases of heavy load rather than just falling over. Most of the uptime has been solved, but we can always take it to the next level. As we work on ranking twitter content and getting it to the timeline, we want to make sure that works.
Another thing we work on — it’s not obvious — you want engineers to move as fast as possible with bare minimum of process but having automation to catch problems so you don’t break something.
TC: Does Twitter have code reviews?
AR: Yep. All the code that goes into production gets reviewed. That’s one of the fundamental principles we believe in to do high quality work. It’s not so much about catching quality, it’s about sharing knowledge and best practices, and making sure we’re not investing and not reinventing the wheel, and serving as a teaching mechanism.
TC: How have things been since the CEO search process started?
AR: I would say, the shortest is, it’s business as usual. The stuff I talked about, about the total audience strategy and finding them great people, some of the wins you’ve seen like Fabric, While You Were Away, a bunch of others, those are all part of our strategy that were conceived a while ago, those
The good news is [Twitter interim CEO] Jack [Dorsey’s] been the chairman of the board, he’s fully on board, he has confidence in the team, he knows all of us really well. We’re continuing to execute and accelerate the pace. People feel good, people know Jack. The reason I think its exciting time, if we answer your question for you, we get the best stuff, wherever you are, then Twitter’s a much more impactful and bigger company than it is today.
TC: Has the CEO transition had any effect on recruiting?
AR: No, not the transition specially. We go after exclusively world-class engineers, they’ve always had a lot of options. Recruiting is something I spend a lot of time on, but the great thing is, we’re able to bring in tons of really strong new people.Nandini Ramini to run all our engineering effectiveness, that’s a team that’s making sure everyone has the tools to be successful. [VP of platform engineering] Chris Pinkam ran parts of Amazon, he now runs all our backend. [New York engineering lead] Vijay Pandurangan, now the engineering lead on project Lightning.
But recruiting, we live in a competitive environment, that’s still the case with Dick and Jack as the CEO. That’s just life, but for the right people twitter is an amazing opportunity. you can always go to a bigger company but never move the needle. You can always found your own startup. The thing that attracts people to us is to go work at a $24 billion public company where still the company has only realized a small fraction of what it can be. And there are many examples of individuals coming in and changing the trajectory of the company in a short period of time, that’s a compelling story.
TC: What’s your best story about Dorsey?
AR: He just started doing this, he does this at Square — we send out the raw unedited notes from our multi-hour staff meeting to the entire company. It’s interesting, it’s hyper transparent, they’re not summarized, you see the raw bullets. It exposes all the interesting things we’re working on, it has sections like what the challenges or what were missing. I think it’s really interesting idea he’s pioneered for us around making sure everyone around the company understands. We’ve just done it one week so far, I’ve heard it gets really good results anecdotally.
TC: How do you guys view the problem of advertising relevance right now?
“We send out the raw unedited notes from our multi-hour staff meeting to the entire company… It’s hyper transparent, they’re not summarized, you see the raw bullets.” Alex Roetter
AR: I was one of the earliest engineers on the ads team. I was involved in ads for almost five years. It’s a ranking problem, it’s an information problem. Our philosophy is, we want to show ads to users and marketers that add value, and in the process Twitter makes money. We’re always computing over all the content based on your interests and activities — What I should show you and how I show you that. We learn correction factors so we can move things up and down depending on quality and what kind of user you are. It’s similar to the sorts of things we’re taking and applying to the timeline. It’s an iterative improvement process. The first day Google started, search wasn’t as good as it is today. You iterate forever. We’ve improved the ad ranking algorithm constantly for years, and we’re doing the same thing on the organic side as well.
TC: So what happens when I mark an ad as offensive?
AR: We take all that and it goes back into the ranking model. We’ll make different predictions based on all the things users do, what you mark as offensive, what you engage with, what you do organically. there are no hard-coded rules about it.
TC: What’s are some of the technical problems associated with ads?
AR: It’s a real-time, very high velocity large-scale ranking problem. We always want to be able to do a better job — and I can do a better job for you if I have more data, being able to optimally match you to content. Ads done well can be really great, ads done poorly can be really annoying. Now include all the real-time components of this, one thing we’ve learned from tradition ad systems, the real-time signal is really important. When we did that our ranking got much better, as measured by real world user behavior.
TC: Anything else you want to add?
AR: Stuff’s starting to pay off. You see the site is up more, the pace of product development has accelerated. The thing that I think is exciting about twitter and the thing that helps us attract is the realization that the opportunity of Twitter is so much bigger if we rejigger the product a little. There’s this unlimited upside for people to come in and work on small, minimal teams and try new things.