As one of the earliest OpenStack supporters, PayPal also happens to be one of its biggest users, with more than...
300,000 cores using OpenStack and drop storage, as well as thousands of developers at the company working with the platform. And while the company continues to champion the open source technology, it's not without challenges.
Jigar Desai, vice president of cloud and platforms at PayPal Inc., based in San Jose, Calif., spoke with SearchCloudComputing about the state of the company's massive OpenStack deployment, how containers changed its homegrown platform as a service (PaaS) layer, and the challenges of operating the open source cloud platform at scale.
Where is PayPal with its OpenStack deployment?
Jigar Desai: All the front-end and mid-tier workloads work on OpenStack. We also built platform as a service automation, so if you're a developer at PayPal, you generally interact with the platform as a service layer to build an application. The PaaS layer interacts programmatically with the infrastructure as a service layer.
What was the impetus behind building your own PaaS layer?
Desai: We started around the time when we were transforming our technologies from being proprietary software to more open source. When we looked at the problem of developer agility, we figured out developers feel very comfortable working on the technologies that are generally also available outside.
We could have just offered them infrastructure API, so they could secure the hardware and deploy their application, but we also wanted standardization from the architectural pattern for deployment. We thought it would make sense to build one, so if you deploy your Java application in a very patterned way, it doesn't remain a developer's worry on how to get that application in multiple data centers, different failure zones, [or] availability zones.
When we started this journey, there weren't too many PaaS players, so we ended up building our own and that layer is going through a lot of transformation. Now that we have capabilities like Docker, we are very excited about using containers for our PaaS layer. And we are investing into technologies like Mesos, as well as Kubernetes. So, while OpenStack gives us a lot of freedom in terms of our infrastructure and automation around it, we are also trying to add more capabilities in our PaaS layer.
There has been some confusion about what your OpenStack use has meant for your use of VMware. Could you clarify where that stands?
Desai: We still have VMware deployments within our production system somewhere. It's not like we are 100% OpenStack for everything. We also work with VMware on our operating system, so we have a good relationship with them. There is a lot of collaboration there. In fact, we collaborate on even the other layers VMware is looking at, like Docker, and other cluster benefit systems like Kubernetes.
With OpenStack, which hypervisor are you using?
Desai: We use [kernel-based virtual machine] KVM and that has been the story from Day 1.
Do you use supported versions of OpenStack?
Desai: Right now, we are upgrading to Kilo and we run pure OpenStack. We are not using any modification. Mirantis was helping us out a few years back, but at this point of time, we are pretty much on our own.
Wal-Mart [Stores, Inc.] is another major OpenStack user that has said it's behind on upgrades. What's the rationale there?
Desai: We would like to be on the latest and greatest version of OpenStack, and that's what we are aiming toward. We are operating OpenStack in multiple availability zones, multiple regions, so we actually need a good level of automation to do that. And that's why our investment has been going in the PaaS layer, because when we are running OpenStack in five different data centers or availability zones, you have to make sure there is no drift in terms of the integration, in terms of capabilities.
We can't really deconstruct the whole availability zone or data center and then construct a new cloud. That would have been the easy story to do, but that's not a choice we have. Doing upgrades on Nova, on Neutron, it takes a while to figure out how to do it without impacting our site and our customers. I'm sure most companies are facing similar issues.
Do you see this problem being resolved soon?
Desai: I think [the OpenStack Foundation] should look into this. I'm not sure if there is enough focus, but that could be one of the areas the foundation could put some energy behind. As the scale increases, these kinds of problems are pretty apparent. I've seen a few projects where the deployment of cloud automation is a focus and I'm kind of excited about those, but as more companies like PayPal are investing heavily into OpenStack, these capabilities come down as shortcomings that are very obvious.
You mentioned scaling. Has networking been a major challenge?
Desai: Scaling OpenStack has been our journey for the last three years or so. We started with a very small footprint and then we expanded it, and now, I feel we're one of the largest private clouds running on OpenStack. And the scale challenges, in terms of Nova and Neutron, they are coming as we expand our footprint. For example, we are using [software-defined networking], and how that technology plays out with hundreds and thousands of cores and how the networking can actually scale to that is something we are still developing.
Jigar DesaiVP of cloud and platforms, PayPal
One of the big capabilities our developers are looking forward to is the ability to flex up or flex down an application within minutes. When we start doing that at scale, we do find a performance issue, and we are actually fixing those performance issues and contributing it back…This is the scale challenge we face almost every day.
But the community is picking up. Every version is more focused on solving some of the performance issues. It's like any open source software; it is a responsibility both ways. We cannot just rely on OpenStack to solve the problem. As we are finding scale problems, we are also putting the fixes back to the OpenStack community.
Where do you come down on this split in the OpenStack community over whether the focus should be on the core projects or if there should be energy committed to the expanding list of ancillary projects?
Desai: It needs to be a combination of the two. The core projects, they need to be scaled up to really meet the demands of the customers, and I think that investment has to continue. But I'm also pretty excited about the new projects that are coming in, because those are the projects that actually enable integration with newer technologies like containers. If it's just investments in core projects, I think OpenStack projects will become obsolete. But the ecosystem around it is also evolving very rapidly, so you can't just focus on the new projects because the foundation of OpenStack needs to be matured even further.
Anything you would have done differently or advice for a company just starting out with OpenStack?
Desai: OpenStack is as much a people journey, an organizational journey, as it is a technology journey. There are lots of pointers around technology and what should migrate to infrastructure as a service, and what does it take to go from a very small proof of concept to full-scale like PayPal. But there is also this organizational part of it, where we really need to rally the teams around the vision of it. It has to be your engineering team, your product team, your projects team, your infrastructure team -- all of them coming together.
Trevor Jones is a news writer with TechTarget's data center and virtualization media group. Contact him at firstname.lastname@example.org.
Evaluating OpenStack vs. VMware for private cloud
Comparing open source cloud platforms
OpenStack gains traction in the enterprise