Hybrid cloud management tools: An overview of functions and products
A collection of articles that takes you from defining technology needs to purchasing options
One of the biggest IT hotspots today is the hybrid cloud. Larger IT operations teams are either experimenting with a hybrid structure or actively moving into production use. As a relatively new IT approach, the hybrid cloud still presents deployment and management pitfalls that slow adoption and make smooth, ongoing operations harder to achieve.
A hybrid cloud management platform consists of two or more segments. One is a private computer cluster that typically handles mission-critical and core enterprise workloads, and the other is a segment -- or multiple segments -- of public cloud infrastructure that handles workload demand spikes and other tasks, such as backup and archiving, web-serving and media delivery.
In an ideal configuration, workloads can migrate back and forth across hybrid cloud boundaries dynamically. This allows a modern data center team to plan and build out for average workloads, rather than peak loads, and to configure for nonstop operation in the event of a failure.
There is a price for such agility, though: fast-changing operational dynamics.
The case for automated hybrid cloud management
Ultimately, hybrid cloud requires management tasks well beyond the traditional knife-and-fork administration approach with command-line interfaces and PERL scripts. At one level, it's just too complicated. A server farm with about 100 servers could today host 300,000 containers, for example, while all instances have vLANs and virtual drives to control. Moreover, the public cloud has brought the concept of tenant control to clouds, where the instance renters do much of their own configuration within their cluster of instances.
The migration toward containers increases the stress on any management approach. Containers raise the instance count dramatically, and the ease and speed with which you can create and delete containers points to a much more dynamic environment than traditional hypervisor instances.
If this were not enough, software-defined infrastructure (SDI) is set to take over the platform layer in clusters and virtualize both data services and the underlying hardware resources. SDI allows for more agile configuration, making transient microservices the norm for many operations and homogenizing the apps, platforms and virtual infrastructure into a fluid mass of changing relationships.
In all of this, it is easy to lose sight of some overarching objectives in IT. Admins need to maintain governance across the system, together with compliance with the law. This brings up issues such as encryption policies and key management. Likewise, it is essential to carry security mechanisms into hybrid cloud and to enhance and harden them as necessary. Both of these issues -- governance and security -- require policy-based controls, automated monitoring and threat identification.
The hybrid cloud approach accelerates the move toward reusable code segments, often from online public libraries. While this is a good way to improve time to market and reduce development cost, there are inherent risks in third-party code. Taking this a step further, the cloud is making software as a service (SaaS), and other as a service offerings, attractive; most future IT operation flows will likely be mashups of in-house code, third-party modules and SaaS packages. With cloud tenants having a good deal of freedom as to where they source code, maintaining a high standard of code quality is a continuous challenge to manage.
During the early phases of hybrid cloud deployment, companies got by with manual methods that were mainly prototyping setups in a sandbox. But with OpenStack now reaching production and Azure entering the private cloud space, the question of automated management methods is high on the priority list. The good news is that the industry is responding to the need, though not yet at a mature-enough level. In the process of automating the clusters, we will see a radical move away from traditional methods, with a realignment of data center admin teams as a result.
For all these reasons -- user empowerment, comprehensive governance, security and efficient, agile operations -- an automated management tool that can handle growth in scale well beyond the sandbox prototype level is essential to bring a hybrid cloud into production.
In an ideal world, all these features would be part of a single, turnkey management package, but we are nowhere near that level of sophistication. Today, we use multiple software modules to achieve a hybrid cloud management platform. However, as tool sets continue to evolve, the gaps should close and it will be easier to build an adequate control system.
Public cloud providers such as Amazon Web Services (AWS), Azure and Google have already resolved these problems within their proprietary systems. The mega-cloud service providers will likely move to bring their products to the private segment of a hybrid cloud, as AWS has done with the U.S. government cloud. As an alternative, companies such as Rackspace have moved to private cloud management as a service. In both cases, the value proposition is to remove the management system integration complexity from production deployment.
Let's look at the key areas for automated hybrid cloud management in more detail.
Hybrid cloud deployment
Using commercial off-the-shelf systems for cloud deployments has simplified integration, but some vendors continue to include proprietary hooks in their products that constrain everything from storage drives to NIC cards or even box-level appliances and network modules. This was understandable in the early days of hybrid clouds, where KISS (Keep it simple, stupid) helped everyone get a working tool up faster, but today it creates vendor lock-in that buyers should avoid.
An automated hybrid cloud management platform starts with deployment. It should be possible to merge new equipment into the resource pool automatically and seamlessly, irrespective of its feature set. This allows admins to expand the hybrid cloud in stages, with the latest gear, at any time. In an industry with fast-evolving technologies, using the most up-to-date SSD or NVDIMM or server chipset will impact cost and performance substantially.
Automated deployment eliminates much of the need to build hardware expertise into a data center team, and existing hardware experts can focus instead on performance tuning and bottleneck mitigation.
Likewise, the automated management of updates and upgrades removes a major chore in large cluster operations. It also radically reduces the risk of uneven update practices, manual errors and complexity with multi-module software stacks such as OpenStack.
Automatic software updates spill over into application and image management. We are seeing automated image validation, along with certified image libraries, which help resolve potential quality or malware issues.
Hybrid cloud monitoring
To make an informed decision about hybrid cloud performance or failures, admins must know what has happened and how it happened. A monitoring system is essential within any large-scale management suite, and should monitor events, performance and system response time. Newer packages can even detect slowing activity in an app segment, which can enable root-cause determination.
The easiest way to give cloud tenants agility and freedom is to apply a policy control system, defined by central IT, which constrains the choices tenants can make, but still within wide boundaries. By providing templates, this also simplifies the admins' work and reduces manual errors. A good hybrid cloud management platform controls and publishes policy templates.
With software-defined infrastructure arriving as the next IT wave, policy-based control is critical to manage fast-moving configuration changes; there's no time for a paper-based request system. A good deal of progress has been made in networking to bring policy-based software-defined networking and network functions virtualization to realization, and early success stories point to improved ease-of-use and rapid response to demand.
Despite early concerns about public cloud security, evidence suggests that mega-clouds are actually more secure than typical data centers. The challenge in hybrid cloud is to apply those public cloud security practices to cover all of a hybrid environment seamlessly.
Modern tool sets for hybrid cloud security are somewhat fragmented, with point software for firewalls, access controls and more. We can, however, expect disparate modules to merge into security suites over time to ease integration challenges.
Encryption, of all of the security features, remains a major thorn in cloud operations, although this is mostly true of traditional data center operations, too. Most users do not encrypt critical data and we often see businesses pay the price for that. Performance in encryption and key management are both challenges that the industry is still working through, but it takes a serious level of user demand, which seems to be lacking.
Intrusion detection, which spots invaders or even accidental incursions, is relatively new, but is another way to protect hybrid clouds' attack surface. This method can handle an agile environment where configurations and external accesses are fluid, and where there are many tenants operating quasi-independently. An intrusion detection tool would have thwarted many recent prominent attacks, such as at Target, Experian, Anthem and JP Morgan.
Admins will want to run their cloud just like AWS -- on a pay-for-play basis -- so they will need a billing tool. Billing tools may offer the ability to control spending policies and limit tenant operations, but all will tally up the charges at the end of the month. A good model for billing is provided by any mega-cloud provider.
The importance of good billing is apparent in the scientific community, where fine-grained, pay-as-you-go access to supercomputing clouds has led to more projects running with simulations and powerful data analysis.
The case for hybrid cloud automation is strong, and vendors are moving to meet this need. The next segments in this series will look in more detail at what is available and how to achieve a first-rate hybrid cloud management for both cloud IT teams and tenants.
Read how to stay up to date with hybrid cloud management tool options
Learn more about the software and strategies available for your hybrid cloud management platform
Read about the ongoing progress of hybrid cloud management software