Inside The Spotify – Echo Nest Skunkworks

They call it Truffle Pig, and it’s ProTools for playlists. Punch in parameters like “danceability”, date ranges, or emotions and Truffle Pig spits out a set of top jams that would fit your “Lovesick 90s Party Starters” playlist.

Truffle Pig is just one of the new musical inventions dreamed up at the sonic skunkworks born from Spotify’s $100 million acquisition of The Echo Nest. Deep inside Spotify’s New York headquarters, the team gave me a peek how the combined company plans to nail recommendations, hook other apps up with legal music, and meld human DJs with algorithms to surface the best songs from the history of recorded sound.

Sptoify Science Type

 

 

 

 

 

 

 

 

 

 

 

It’s taken a long time for Spotify to get the basics sorted so it could get to this experimentation phase. It’s history is chronicled in a sprawling mural in its NYC HQ’s lobby. Each milestone is illustrated, from founders Daniel Ek and Martin Lorentzon giving birth to the company in Stockholm back in 2006, to musictech curmudgeons Metallica joining the service, up to the modern day when 40 million listeners rely on Spotify. The art gives a sense of the struggle to make digital music legal again after Napster established piracy as the norm. The only thing missing from the mural is the huge amount of money, over a half billion dollars, that Spotify has raised to fund this journey.

When Spotify launched in the US in mid-2011 with help from Facebook, it was one of the only on-demand streaming services with collections from all the major record labels. Three years later, music has become a brutal turf war with Apple/Beats, Google /Songza, Rdio, Amazon, Deezer, and more duking it out for ears.

echo-nest

Spotify has one of the largest libraries, with over 20 million songs and 20,000 more added each day. But it’s not the size of your catalogue, it’s what you do with it that counts. Users are overwhelmed with choice, and need curation to sift out the sweetest sounds. So each service is desperately trying to differentiate itself with discovery.

That’s why Spotify acquired The Echo Nest. A fundamental building block of music intelligence, The Echo Nest’s vault of meta data and personalized recommendation algorithms powered discovery on Spotify, Rdio, Beats Music, SiriusXM, and more. So to make sure it knew what people wanted to hear better than any of its competitors, Spotify ponied up $100 million, mostly in stock, to buy the Massachusetts startup.

Taste Profiles

After a few months of mashing the companies together, the combined music juggernaut gave me a look at what its been tinkering with. In an assuming room buried in the office, I meet Spotify’s VP of Product Charlie Hellman, and The Echo Nest CEO Jim Lucchese. A lot of people out there claim to be feverish music fanatics, but few foam at the mouth with excitement quite like these two.

The Echo Nest CEO Jim Lucchese

The Echo Nest CEO Jim Lucchese

“The philosophy here is to create the absolute best listening experience” says Lucchese. “To reach every passionate music fan out there, and get casual fans more engaged and more passionate.” Luckily, Spotify’s business model isn’t about selling hardware or search ads. Its incentives are aligned so everyone benefits when Spotify sounds better. Hellman explains “The longer people stay on the platform, the more likely they are to pay for premium”, which is Spotify’s real money-maker.

The key to unlocking that enthusiasm is understanding each user’s unique music preferences so Spotify can suggest them the right song. So The Echo Nest’s first job was to dive into all of Spotify’s data. “Our most engaged users are coming more than 20 days per month and listening to an hour per session. There’s so much data, more than the listener even realizes” Lucchese says. The goal was “to understand you as a music fan with a level of depth that neither of us could do until we came together.”

The result of all this digging is what Spotify calls “Taste Profiles”, a preference analytics and visualization tool currently available only to employees. By assessing the meta data of what you listen to, Spotify can determine a ton of attributes including:

  • Mainstreamness – How widely popular is what you listen to?
  • Freshness – Do you prefer music that was just released?
  • Diversity – How open are you to new sounds? How often do you add new music to your playlists or collection?
  • Hotness – Amongst tastemakers, how buzzy is what you play?
  • Discovery – How often does what you listen to get popular later?

The Taste Profiles tool measures you on each of these scales and compares you to everyone else, then declares “You are 6.6% above the median freshness.” Beyond being interesting, these stats helps Spotify know if it should suggest you more or less mainstream tracks, stick to classics or serve up the freshest release, and whether it should use your savvy ears to suggest what’s hot to other users.

“It’s important to know If you’re a music savant on the bleeding edge, or if you’re more of a casual listener” Hellman tells me. “Because ultimately, we want those users to have two different experiences.”

jc_clusters

Spotify also assesses the genres and artists you favor to visualize your music “Clusters”. These are the different sides of your listening identity. My music taste broke down into “Indietronica” like Hot Chip, Geographer, and Robert Delong, “Nu Disco/Retro Electro” like Goldroom, Poolside, and Touch Sensitive, and “Shiver Pop” like James Blake, Disclosure, and SBTRKT.

I could infer the icy, detached sound of that last one from its name, but I was totally stumped by what “R-Neg-B” was. At first I thought it was a buggy rendering of the “&” in “R&B”. But Hellman explained the “Neg” refers this genre’s refute of traditionally safe R&B subject matter, and its inclination to explore topics like homosexual relationships and asceticism, as characterized by artists like Frank Ocean and The Cool Kids. Spotify constantly crawls the semantic web trying to pinpoint these obscure genres.

Give my auto-generated Indietronica/Indie Pop cluster mix a spin.

Clusters can also be turned into different playlists, including ones with the most popular tracks in a genre, the top songs by artists of that style who you already listen to, or peppered with new artists you might like.

Taste Profiles and Clusters both rely on a combination of algorithmic gathering and editorial insight into how to lump artists into genres. Hellman says “It’s a really good example of how machines and humans are not at odds at all, but are really working together.” These features are still incubating in the skunkworks, but you can savor the recommendations they power in the Spotify Browse and Radio sections.

Truffle Pig

Pandora prides itself on its music genome’s ability to play what you want to hear next without a flesh-and-blood person behind the wheel. Yet Spotify is trying to embrace a classic discovery method: asking the infinitely knowledgeable record store clerk. Rather than rely entirely on the human brain, though, Spotify’s skunkworks has devised a tool to turn its mere mortal editors into playlist-making cyborgs. Truffle Pig.

“Playlisting is getting to the next stage of its life” says Hellman. “It started with what you had in your collection or could record from the radio” But now thanks to the labels offering access to their entire back catalogues, Spotify’s team that makes playlists for its Browse feature needed a way to sort through its 20 million tracks. Truffle Pig creates an advanced search engine which returns results that can be easily collated into precise playlists.

desktop-browse

Tapping The Echo Nest’s detailed analysis of every Spotify song, editors can punch in parameters corresponding with Taste Profile attributes like hotness, or quantifiable aspects like tempo or recording date. But what’s really cool is its ability to surface tracks depending on characteristics about their actual sound, like “acousticness”, “speechiness”, or “loudness”. Each Truffle Pig search parameter can be set with a threshold, so editors may allow a little bit of speech or require a high loudness score for a song to qualify.

Want to make a wordless playlist to work to, but are sick of techno and classical? An editor could search for songs from the last 60 years with zero speechiness, high acousticness, and high energy to build a playlist of motivating yet non-distracting songs from artists like Spanish guitar virtuosos like Rodrigo Y Gabriela. Pissed off but hate heavy metal? Dial up the danceability, set the emotion to “angry”, and exclude anything from the 1980s. Engagement with the resulting playlists is monitored to determine revisions and see which playlisters have the golden touch.

1378434_10100926967566367_702061081_n

Spotify VP Of Product Charlie Hellman

“There’s no one taking playlisting more seriously than Spotify” Hellman claims. “Editors are giving each other shit over their skip rate, making fun of them for a song that performed poorly but they didn’t take out.” Spotify wants to offer curation at scale by putting playlisting on steroids, and Truffle Pig is the performance-enhancing drug. This human curation combines with social recommendations from what friends listen to and algorithmic charts to make Spotify an omni-jukebox.

Sadly, you won’t find Truffle Pig publicly available just yet, but Hellman says Spotify is looking at how it could offer access to tastemakers and top amateur playlisters in its community over the course of 2015.

The Backbone Of Legal Music

Even with its catalogue and The Echo Nest’s data, Spotify can’t build everything. There are so many ways to sort, discover, and play music. The company recognized this early, and launched a platform for third-parties apps within its desktop software in November 2011. But buried amongst other features, many users never even found apps like MoodAgent for emotionally resonant music discovery or Songkick for nearby concerts of your favorite artists. And since the apps lived entirely within Spotify and weren’t on mobile, they were tough to monetize, developers shrugged off the platform, and Spotify stopped accepting new apps.

Spotify App Finder

Yet there was a massive opportunity. Spotify had done the expensive legwork of getting legal music licenses from the major labels — something extremely tough for small music startups to do. So when Spotify acquired The Echo Nest, I suggested it create an API that would let people log in to third-party music apps with their Spotify account, and bring their rights to on-demand streaming music with them. This would make Spotify the backbone of legal music across the industry, like how Facebook became a backbone of identity across the web by letting you log in to other apps and bring along your social graph.

Spotify Plus DjayInstead of being a siloed one-size-fits-all music app like its competitors, Spotify could become a true music platform with a wide variety of interfaces built atop its library. It seemed to me that with each additional app where you could play your Spotify music, you’d be that much more locked in and likely to pay for a subscription.

Spotify apparently had the same melody in its head. A few months later it launched its first official API partnerships with mobile DJ apps DJay and Pacemaker. By logging in with Spotify, the apps began letting people mix any two songs from Spotify’s library instead of restricting users to music locally stored on their device, like in iTunes. The Echo Nest data can also detect the BPM, danceability, and key of a song currently being played, and suggest songs that would mix well with it.

The impact of this API can’t be overstated. You can suddenly DJ with pretty much any song every recorded, rather than the handful you own. That’s a complete game-changer for music performance. And it’s only available if you have a $9.99 a month Spotify Premium subscription.

“The way we’re going to thrive is if people are finding more and better music to listen to, and they’re getting more value out of this ecosystem” Hellman says. If you had to pick between streaming services to subscribe to, Spotify hopes you’ll choose the one with a whole solar system of apps in orbit.

The Missing Note: Social

Spotify may have 10 million paying subscribers and a skunkworks full of future products to recruit more, but it’s still never quite figured out social.

Spotify US Launch

First, it let Facebook’s ticker share news to friends of every song you played. A lot of people hated this. It felt simultaneously invasive and boring, filling Facebook with occasionally embarrassing and rarely useful stories about how you’ve been listening to Britney Spears.

Then Spotify revamped its own ticker on desktop that let you follow the listening habits of artists and celebrities. Stuffed in a forgotten corner of the app, the feature was rarely used. The only thing anyone really followed was Hipster International, a public playlist of indie gems run by Spotify director and Napster founder Sean Parker. People dug how you could subscribe, and new tracks would be periodically added at the top so you always had something fresh to listen to.

While social feels broken on Spotify, it’s still one of its most important elements. “The strongest thing that’s pulling casual listeners into Spotify is friends” Lucchese tells me.

Screen Shot 2014-10-19 at 2.56.45 PM

That’s why rather than build its own little Twitter for music, I think Spotify should take the best elements of the ticker, its Browse section, and Hipster International to create what I call the PlayFeed.

The idea is to offer a personalized, constantly updated playlist based on what friends are listening to, new releases from artists and genres you like, chart-toppers, and picks by Spotify’s editorial team. Instead of Spotify’s Browse feature which is part blog, part News Feed, and requires your full attention, the PlayFeed takes the familiar music-first, fire-and-forget format of the playlist. But rather than staying static or requiring you to add new tracks manually, they’re automatically injected at the top based on all of The Echo Nest’s data.

At any time you could press play to get a constant, passive stream of new song recommendations, or dig down deeper to return to old suggestions you enjoyed. This would effectively unify Spotify’s social, algorithmic, and editorial curation into a single place for music discovery. Done right, it could become as addictive as our other feeds, help artists grow their audiences, lock in subscribers through their social graph, and give Spotify a differentiation point.

More Than A Loss Leader

Spotify BirthThe big problem for Spotify is that its on-demand streaming competitors aren’t in the music business. Apple uses music software to sell iPhones and MacBooks. Google Music is designed to lock people into its ad-filled ecosystem. Amazon bundles music in for “free” with a $99 a year Prime subscription that gets you to buy more physical goods.

[Update: Spotify just launched a family plan, which lets users buy premium subscriptions for up to four additional family members at 50% off, or $4.99 each. A family of five would pay $29.99 through the family plan to each have their own accounts complete with their own playlists and personalized song recommendations, rather than $50.]

Still, Spotify’s competitors could offer music cheaper than it can because they make money elsewhere. Eventually, on-demand streaming could be something we expect to come free with a phone or mobile OS. And most music listeners aren’t hardcore enough to desperately need or even be able to recognize a better interface or catalogue from one that’s simply “good enough”.

To win, Spotify will need to have the best app, and rely on the easy of trying out its ad-supported version. Lucchese says “I believe in today’s app economy, there’s no such thing as ‘good enough’. It’s so easy for users to try out a new experience that people aren’t got to settle for what comes in their phone.”

So while competitors fuddle with integrations and try to decide how much music is even worth to them, Spotify can just keep fine-tuning its product. Lucchese concludes, “We look at it as ‘we’re never done'”.