Another Xen bug generated some headlines this week, but it’s much ado about nothing for most cloud consumers.
A bug in the Xen open-source hypervisor, popular among cloud infrastructure vendors from Amazon Web Services (AWS) to Rackspace to IBM Softlayer, was publicly identified this week that allows malicious para-virtualized guests to escalate their privilege to that of the host. But unlike past Xen vulnerabilities, this one involved minimal reboots, with most vendors avoiding it altogether.
Two reasons this bug should cause minimal problems for cloud providers are that the required access level isn’t trivial and the patch is “absurdly simple,” said Carl Brooks, an analyst with 451 Research. In truth, enterprise IT shops face a much bigger threat from someone opening a bad email or clicking on the wrong link.
“Some providers can be affected and some may well be … but if this is a problem for you as a provider, you have more systemic issues around security than this particular exploit,” Brooks said.
Cloud vendors also are generally given advanced notice in these situations so they can address it internally before the vulnerability is made public. And they’ve had plenty of chances to learn how to deal with Xen problems — there have been more than a dozen publicized patches for it in 2016 alone.
Cloud providers have learned to dodge bullets
Xen bugs have dogged AWS in the past, requiring roughly 10% of its instances to be rebooted as recently as October 2014. The last time a major Xen bug came to light in February 2015, however, AWS limited the impact to only 0.1% of EC2 instances.
This time, AWS managed to avoid a reboot entirely, telling customers that data and instances are not affected and no customer action is required. Amazon declined to comment on how it addressed this patch and managed to avoid the reboot.
It’s possible that Amazon is so far off the trunk of the open-source project that this incident didn’t cause any major impact to AWS in the first place, Brooks said. But for customers that are still worried about the bug, the simplest solution is to switch from para-virtualization to full virtualization.
Other Xen-dependent cloud platforms appear to be taking a cue from Amazon’s improvements in addressing these patches to avoid downtime. Details are scarce, but it’s believed some vendors have relied on live migration, while AWS is among those suspected of using hot patching.
Rackspace, which also declined to comment, issued a statement telling customers it had no plans to perform any reboots and that there were no actions needed by its customers. That’s a reverse from 2015, when it needed to do a Xen-related reboot but even then it communicated with customers, giving them advanced notice when their instances would be shut down.
The only major cloud provider that appeared to have significant downtime from this Xen bug was IBM. CloudHarmony, which tracks uptime for a wide range of cloud providers, noticed all 20 of its IBM SoftLayer status VMs were rebooted over the weekend with five to 15 minutes of downtime for each.
IBM, which did not return requests for comment, does not appear to have a public-facing status page for maintenance notifications — its Twitter feed for SoftLayer notifications has been updated once in the past year, to proclaim it is “only used if there is an emergency situation.”
Cloud hosting provider Linode has had to perform maintenance on its servers that rely on Xen. Since the last major bug in early 2015, Linode has moved its newer servers to the KVM hypervisor. In its security notice, it told customers that anyone wanting to move from the legacy Xen virtual machines to the KVM virtual machines would avoid the reboot.
It’s unclear if Linode’s move to KVM had anything to do with Xen’s spat of patches over the past 18 months. The company declined to comment.