Zynga CTO of infrastructure shares cloud lessonsDate: Jul 20, 2011
Zynga uses private and public cloud technologies in harmony to power customer’ gaming experiences. When first launching a game, the company first runs it on Amazon Web Services. Eventually, the company can decide whether Z-Cloud, Zynga's private cloud, can support its capacity requirements and seamlessly transition the services over. As Allan Leinwand, Zynga CTO of infrastructure engineering explains, Zynga's server farm keeps customers' farms, cafes and word games online. All of that and more on this week's episode of Cloud Cover TV.
This week we discuss:
- Zynga Z-Cloud private cloud
- Moving from private leased space into Amazon Web Services
- Trading Capex for Opex
- Leasing space in the public cloud when we could be building it ourselves
- When Zynga can predict and plot the capacity of its games over time, it moves them in-house to Z-Cloud
- Empires and Allies is a new game, infrastructure needs to meet this game are different than older games
- Use Rightscale for managing and monitoring cloud virtual machines as well as Z-Cloud virtual machines
- Cloud.com acquisition with Citrix
- With Amazon running on open source Xen we see a lot of synergy with Citrix
- Capacity planning one of they key things, built custom tools for this that monitor the infrastructure on a 5 minute granularity
- Enterprise IT talks alot about consolidation as a motivation for cloud, take 10, 20, 30 machines and put them on a single box
- Key learnings from Zynga are don't only view cloud as a consolidation or cost savings technique, embrace it for its flexibility
- Speed of spinning machines up and down, new networking paradigms, machines moving throughout the data center
- Enterprise use cases, real time trading, spinning up Hadoop clusters for analytics
- VMware versus Citrix, going after different markets
- Zynga is not about optimzing a database on a SAN architecture which is more of an enterprize IT focus
- It's focused on next gen servers, memcache, big data, hyper scale and hyper growth
- NoSQL, memcache, PHP, couchbase, next gen flash, HTML5
Read the full transcript from this video below:
Zynga CTO of infrastructure shares cloud lessons
Jo Maitland: Hello, welcome to Cloud Cover TV, our weekly show on all the juiciest news in the cloud computing market. My name is Jo Maitland here in San Francisco. This week I invited Allan Leinwand who's the CTO of infrastructure engineering at Zynga, the online games company.
We specifically got Allan on the show because Zynga is all in on the cloud. The company uses Amazon's web services for launching new games and it's also built its own private cloud internally, which it calls Z cloud. And then hooks up with AWS for additional capacity when they need it and when they're launching new games. The idea being that once they understand the capacity and the workload of a specific game they then bring it in house.
We wanted to understand from Allan, you know, how they came up with that strategy and how this might apply to an enterprise IT environment or not. I think what he has to say there is interesting for this audience. Tune in and check it out. Hi Allan, thank you for being on the show.
Allan Leinwand: Thanks Jo, thanks for having me.
Jo Maitland: Tell us about Z cloud. This is your private cloud infrastructure. And what hybrid cloud computing means to you and sort of, what's on Z cloud, I guess. What do you do in the public cloud space and how does this come together?
Allan Leinwand: Yeah, sure, happy to. Z cloud is our Zynga private cloud. It's an infrastructure that's really evolved over a period of time. Zynga as an infrastructure company originally started in sort of the traditional leased data center space. But as our games and our infrastructure evolved, and the games became more popular, and we brought more play to the internet we actually needed to be a lot more flexible on our infrastructure. We started using a public cloud, Amazon's web services. That's sort of been talked about a lot and how we morphed from a private lease space into Amazon's web cloud.
As we continued to use that service and evolve it, and launch more games on AWS we suddenly realized we were essentially trading off CapEx for OpEx. In other words we were leasing a lot of facilities and leasing space up in the public cloud where we could essentially be building it ourselves. Probably the better part of a year ago we decided that we were going to operationalize that and build our own private cloud. Then begin to build a hybrid cloud between AWS and our private cloud. That's where really Z cloud came from. So Z cloud, at its concept, is our private cloud. Same features and functions as Amazon but used primarily with inside of Zynga.
Jo Maitland: When do you use Z cloud, when do you use the public service?
Allan Leinwand: Yeah, it's been an interesting discussion around capacity planning, which is really the topic I think you're hitting. We like to use public infrastructure when we don't know the work load that's involved. When we're about to launch a game, we can build the game, get excited about the mechanics, get really jazzed about the demographic we're shooting for, but we're not sure exactly how popular it's going to be. Generally we like to use infrastructure that can be very flexible and that can scale to the needs of that game. We tend to want to launch in AWS. Once we want to know the load of that game and once we can, as I like to say once the slope of the line is something less than vertical and something that we can predict, something that we can plot over time.
Jo Maitland: Meaning the capacity.
Allan Leinwand: The capacity of the load of the game. How many people are coming on? How much infrastructure is really needed? What's the growth curve? We then tend to move that back over to Z cloud.
Jo Maitland: Have you seen, is there a pattern there yet, on when that optimum moment is to move it back?
Allan Leinwand: You know, I think the pattern there is really being able to understand the workload. The thing that's interesting about Zynga, from an infrastructure perspective is that it's not one app. It's applications like Zynga poker which have been around for over three years now. Those applications have evolved, they've migrated, they've changed their technology. They've changed the way they stress the infrastructure. And then there's brand new games like Empires and Allies, which is a whole different set of infrastructure that has on a different curve and a different sort of capacity rate. We need to really use both our private Z cloud and our public cloud, in a way that meets the capacity and growth curves of both those technologies.
Jo Maitland: Give us an example of the size of the environments, roughly. What you have going on in Z cloud internally. At any one time what could be the minimum or the maximum you've seen so far on the public?
Allan Leinwand: I think the size is interesting but I don't think it's necessarily material. I think what's most material about it is that we can use them both. That's the really unique thing I think we do at Zynga. We have the ability to view both the public cloud and the private cloud as a unified system and be able to move capacity back and forth between them.
Jo Maitland: But for a lot of people out there, possibly in our audience, who is in enterprise IT. You know, not so much involved or even familiar with the gaming world, possibly not Facebook users. We're talking thousands of servers at least. Just to give them an idea of your, the size of the infrastructure. We're talking thousands of machines, thousands of nodes.
Allan Leinwand: Oh yeah. That's right. I mean, we have previously discussed the fact that we can launch 1,000 physical servers in 24 hours. We've had the ability to add that capability into Z cloud and we've done it a number of times.
Jo Maitland: And Z cloud now is on the East Coast and the West Coast, a data center in each place.
Allan Leinwand: Z cloud deployments are evolving. We don't necessarily disclose exactly where it's at because we're trying to build our capacity in a way that's flexible for our users. But, you know, we definitely are building Z cloud in multiple geographies.
Jo Maitland: Talk to us about some of the technology choices there. You guys have talked about using Cloud.com and RightScale. Can you use those internally and on the public cloud?
Allan Leinwand: I mean, the technologies we've talked about publicly are clearly Amazon web services and the tie is to that. We've also talked about RightScale. RightScale is, again for maybe people who don't know, it's a piece of technology that allows you to provision, monitor, and manage cloud instances and cloud virtual machines. Yes, we actually have RightScale constructed in a way and optimized in a way where we can actually manage things both in the public cloud and in the Z cloud from that single [inaudible] that is RightScale.
Jo Maitland: I guess cloud.com is very public about the fact that Zynga is a customer. Is there value for you in those guys being more on the open source side of software, cloud software, cloud platform software? They definitely seem to be pinning the future of their company on that open stack, you know, open source approach to cloud infrastructure.
Allan Leinwand: I think cloud infrastructure is evolving. You just saw cloud.com just went through at acquisition with Citrix this last week. We're really excited about what that brings to the market. We think that with Amazon running on open source zen and cloud.com using things like zen sources or hypervisor we see a nice synergy there. We look forward to seeing what that brings in the future. There's going to be some really interesting new things evolving on the public cloud and the private cloud. We're going to adopt them as we see fit. Our main goal is really just to scale our infrastructure. Make sure it's as flexible as possible for the different workloads we have. When you go to your empire and you want to attack somebody or you want to go do some farming the game had better be there. It's my job to make sure the infrastructure scales for that.
Jo Maitland: Talk about maybe, Amazon web services famously uses a lot of its own custom built tools, that kind of thing for capacity planning. Did you guys have to do any of the same? Are there open source tools out there that you use? Could you talk about that?
Allan Leinwand: Capacity planning is obviously one of the key things we do. We did build a number of innovative technologies that allow us to plan capacity, monitor the capacity of the infrastructure almost on a five minute granularity sort of view. I think that's really important for us, because one of the things that we don't want to do is get in a position where the infrastructure gets in the way of play. We want people to be able to play at any time. That's a key fundamental learning for us and it's a key fundamental thing that we focus on.
Jo Maitland: Help out the enterprise IT audience here, and you've done it before Zynga. You had a chance to know what it's like to be on that side of the picture as well with a lot of legacy infrastructure, perhaps really heavyweight software. Are there any lessons in what you guys have done that, you know, that the enterprise IT audience can learn from? Maybe any nuggets that you can share?
Allan Leinwand: Sure. I mean when I think about enterprise IT and I think about cloud, I think the clear motivation people have is consolidation. People talk a lot about taking 10, 20, 30 machines and putting them into virtual machines on a single box in enterprise IT data center, and letting that optimization be useful. In our particular case that's not necessarily what we do. We really see cloud as being, and cloud technologies, as being something that gives us flexibility. From our point of view, and I think the key learnings are, don't only view cloud technology as something that is a consolidation or a cost savings technique. Really embrace it for its flexibility. Really embrace it for how quickly you can spin machines up and spin machines down. Really embrace it for a new networking paradigm. Machines moving throughout the data center flexibly and easily. You really need to sort of take your mind out of the traditional architecture; you know wired router, wired switch, server. Into something that's far more flexible and far more dynamic. It's something that I think people are beginning to get their heads around these days.
Jo Maitland: Have you seen any use cases in the enterprise that demand that kind of flexibility yet?
Allan Leinwand: I've seen and I've heard of in my past about people doing things like real time trading, interesting Hadoop clusters that spin up to do business analytics and then spin back down. I think you'll start to see more and more of that sort of as the network and the systems and the cloud all become tightly coupled.
Jo Maitland: The other kind of alternative out there from a technology choice, if you like, for at least the Fortune 1,000 that's really kind of pushing this hybrid cloud model is VMware. They recently changed their licensing and pricing model to be priced around VRAM which is, you know, to sort of, in their words, more kind of enable the utility model. From what I've seen of people buying machines for cloud they tend to be high density. For example, Amazon's strategy is very high density boxes. Stuff them with as many CPUs and as much RAM as you possibly can. At least in that model of cloud it would be a lot more expensive. Maybe the Fortune 1,000 rack machines differently. I'm curious to get your thoughts on that too.
Allan Leinwand: I think that, you know, VMware has clearly carved out to go after Enterprise IT and that virtualization model. I think that what you're seeing in terms of what's going on with now Citrix and other folks in the public cloud computing, they're just going after a different market. We're about scale. We're about flexibility. We're not necessarily about optimizing a database on a SAN architecture which is something that might be an enterprise IT focus. We're about sort of the next generation of application. We're about web servers. We're about things like meme cache and meme base. We're focusing on big data. We're focusing on hyper scale and hyper growth which is a completely different, almost orthogonal topic to I need to run SAP and Oracle financials within an Enterprise IT infrastructure. I think there probably is good space in the market for both.
Jo Maitland: Talk to us about the kind of skill sets you're interested in when you're out there looking. It's really competitive here in the valley. What are the core skills you're looking for from the crop of engineers coming up?
Allan Leinwand: I think the things we look for at Zynga and the things that we really focus on when we're looking for folks is innovators. We always want to find folks who are thinking about two quarters, three quarters, two years ahead of time. We're also looking for people who want to operate incredibly fast. You know, we are growing at a scale and growing at a speed, that I think people haven't seen in quite some time if ever. We need people who are really energized for that. It's your classic sort of work hard, play hard mentality. I think we really embrace that. I think Zynga is about bringing innovation to the marketplace. I think we're doing something very unique with Z cloud and having that hybrid model that scales at a way that I don't think people have seen before. I think we're talking about bringing in folks that really understand the next generation of technologies. You know, sequel database technologies, operating things in memory and meme cache as opposed to doing things in a traditional model, new types of programming languages.
Jo Maitland: What would you throw in there for developers out there?
Allan Leinwand: Well, we're clearly focused on things like PHP. We're looking at databases like Touchbase are very interesting to us. We look at all sorts of interesting new technologies whether it's going up into the next generation of flash or HTML five. These are all sorts of core components of our infrastructure that we spend a lot of time architecting and instrumenting.
Jo Maitland: Is that stuff being taught in schools yet or are the developers that know this doing it sort of in their own? I'm curious to see if the school system has caught up yet in comp sci courses.
Allan Leinwand: I think they're getting there.
Jo Maitland: They're getting there.
Allan Leinwand: I think a lot of people are intrigued by the Apple IOS ecosystem. People are intrigued by doing things on Android. People are intrigued by looking at new programming paradigms and new design patterns. I think that's beginning to become, and we're seeing university recruit sort of bubble up, that have those skill sets. It's been pretty exciting for us.
Jo Maitland: Great. Allan, thank you so much for being on the show.
Allan Leinwand: Thank you very much for having me.
Jo Maitland: Cheers. This has been Cloud Cover TV. Thank you for watching and tune in next week for more insider news on the cloud computing market.