This content is part of the Essential Guide: An enterprise guide to big data in cloud computing

Spotify moves to Google cloud with eyes on big data

Spotify surprised industry observers by becoming one of the most high-profile cloud customers for Google, which is trying to shake the image that it can't compete with Amazon and Microsoft.

Google scored a coup with its latest big-name cloud customer -- and for a company that often touts pricing as a key benefit, it was higher-level services that made the difference this time.

Spotify, the popular music streaming service, plans to migrate nearly all of its privately hosted back-end workloads to Google Cloud Platform. The firm grew to where it had to consider building its own data centers but opted against such a move due to the cost and required expertise, instead choosing Google to help expand its big data capabilities.

"Google is also the world champion at pushing little streams of constantly changing data to individual users, so Spotify is picking the perfect infrastructure," said Carl Brooks, an analyst at 451 Research in New York.

Spotify has tens of thousands of machines across four data centers worldwide. It started working with Google about 18 months ago, and the goal is to have a "huge chunk of infrastructure" migrated to Google Cloud Platform over the next 18 months, said Wouter de Bie, big data architect at Spotify. The company serves a small group of users from the platform already.

"The main [reason] why we chose Google was their tooling for data processing and data scientists," de Bie said. "Google is pretty much ahead of the curve when it comes to technology dealing with vast amounts of data."

The big news for me is that this wasn't about Google being a few cents cheaper than AWS or Azure, but it was about the value of the higher-level data services, BigQuery, in particular.
Dave Bartolettiprincipal analyst, Forrester Research

BigQuery, Google's managed data analytics database, was the biggest draw, but Spotify is also experimenting with Google's other big data tools: Dataflow for streaming and batch data processing, and Bigtable, which could potentially replace its need for Cassandra and the underlying management it requires.

"The big news for me is that this wasn't about Google being a few cents cheaper than AWS or Azure, but it was about the value of the higher-level data services, BigQuery, in particular," said Dave Bartoletti, principal analyst at Forrester Research.

As the public cloud market matures, vendors have showcased big-name customers to assuage concerns about the multi-tenant environments and prove their enterprise bona fides. Though Google counts the likes of Coca-Cola, Sony and Macy's among its clients, it's often seen as not having nearly the same presence among enterprises as Amazon Web Services (AWS) and Microsoft Azure -- an image Google is keen to shake. The Spotify deal comes days after Sports Authority said it's using Google Cloud Platform for its e-commerce site.

Google needs to nab major wins with large companies to match the major enterprises making broad commitments to AWS, and the Spotify deal provides a "big company goes all-in in the public cloud" narrative, Bartoletti said.

In fact, Spotify is a prominent customer for Amazon, which promotes the media company as a case study touting the benefits of AWS. Spotify will continue to use Amazon Simple Storage Service and CloudFront. Amazon declined to comment on the deal.

"We use Amazon primarily for content delivery and we'll keep on doing that," de Bie said. "That's not going to move, but we haven't used the rest of the infrastructure that much."

That means the actual storage and serving of audio will remain on AWS, but the main part of the infrastructure, which is dedicated to serving metadata around playlists, recommendations and searching catalogues, will move to Google.

"That is where we want to focus on and that is really what Spotify is about," de Bie said. "It's not just about serving that content."

Some critics, surprised by such a high-profile company going with what is seen as the distant Number 3 in the hyper-scale public cloud market, have suggested that Google extended a sweetheart deal as part of the move. Details of the agreement haven't been disclosed, but Nicholas Harteau, Spotify's vice president of engineering and infrastructure, told The Wall Street Journal that the company "negotiated hard on price."

When large customers move to public cloud platforms, they often receive credits or other financial incentives beyond the standard price. But as public cloud infrastructure increasingly becomes commoditized, pricing only goes so far, de Bie said.

"Who wouldn't press hard on pricing? I think that every enterprise or business owner should press hard for pricing to get the best pricing from vendors," he said. "That was definitely not the primary reason for picking Google."

Spotify needs to experiment to stay nimble and ahead of its competition, so it's participating in early access programs at Google and sees itself as a great test of those services at a very large scale, according to de Bie. The long-term plan is to utilize the platform to actually improve Spotify's overall service -- and, in that sense, moving to Google is more of a partnership than a typical vendor-customer relationship, he added.

"What you hear quite a lot is that [with] Google, some of the technology -- it's ready, but it's not as mature as other players," de Bie said. "That gives us the opportunity to really work with Google and improve their product portfolio."

Price was important, but Google's terrific big data services made the difference with a customer that wants to differentiate with its data tier, Bartoletti said. And that level of input on products shouldn't be surprising for a customer of its size.

"If Google keeps up the cadence of major enterprises making large commitments to [its cloud], it will knock out the 'not ready for enterprise' criticisms," Bartoletti said. "We see that happening through this year." 

Trevor Jones is a news writer with TechTarget's data center and virtualization media group. Contact him at [email protected].

Next Steps

Google takes aim at AWS Spot Instances with new cloud service

A closer look at the Google Cloud Storage Nearline service

Evaluating container services from Azure vs. Google vs. AWS

Dig Deeper on Public cloud and other cloud deployment models