Mathias Rosenthal - Fotolia


HPC market opening arms to public cloud

While only a small percentage of HPC applications run in the public cloud today, new services from AWS and Google could soon increase that number.

Like many businesses, organizations that have entered the high-performance computing market are showing an increased interest in public cloud. In fact, public cloud usage for high-performance computing is growing at double-digit rates, according to Intersect360 Research, a market research firm focused on high-performance computing. But despite this increase in adoption, only a miniscule number of high-performance computing systems today use public cloud, as their applications require a high degree of software and hardware customization.

High-performance computing (HPC) is a niche computing model, and the HPC market has unique application and infrastructure requirements. HPC organizations do not frequently run typical ERP, CRM or email systems on HPC servers. Instead, HPC infrastructure supports computer-aided engineering, chemical engineering, geosciences, mechanical engineering, weather applications and other resource-intensive apps.

What's more, HPC applications demand top-end computing throughput; performance is measured in floating-point operations per second rather than the millions of instructions per second metric seen with commercial systems.

Capacity is also different in most HPC environments. While the typical enterprise server has 10% to 15% of unused capacity, HPC systems strive to run at 100% capacity, said Chris Willard, chief research officer at Intersect360. To reach that goal, HPC systems use complex parallel processing techniques.

In addition, many organizations in the HPC market build special-purpose applications that take months and even years to design. Many of the applications then run for hours or days on special-purpose systems. A scientist, for instance, might follow this model to test a mathematical theorem. Consequently, unlike most business applications, many HPC applications do not run continuously.

Customers in the HPC industry are atypical, as well. They complete much of their work in university and government research laboratories, and few HPC applications come from the traditional enterprise market. These specific requirements present challenges to public cloud vendors.

"Building a business case that requires your system to sit idly for long periods of time and requires a great deal of customization is difficult," said Willard, who estimates that only 1% of the HPC sector today uses public cloud.

Cloud vendors eye HPC market gains

Despite challenges, HPC use cases are emerging in public cloud. Some organizations, for example, are migrating "low-hanging fruit" HPC applications -- or parallel applications that don't require specific architectures -- to the public cloud, said Bob Sorensen, research vice president of IDC's High Performance Computing group.

Building a business case that requires your system to sit idly for long periods of time and requires a great deal of customization is difficult.
Chris Willardchief research officer at Intersect360

Recent industry changes may also boost public cloud acceptance in the HPC industry. Commercial businesses are becoming more interested in HPC capabilities. Big data is pushing the boundaries of traditional processing and blurring the line between commercial and HPC applications. In markets such as pharmacology and healthcare, the need for high-end, compute-intensive applications is growing.

This interest is creating ripple effects, and HPC hardware is becoming more standardized. For instance, Nvidia, a leader in graphic processors, is building clustered GPUs with the aim of making its system more attractive to the HPC community.

Recently, Amazon Web Services (AWS) and Google Compute Engine developed special public cloud services to better meet HPC needs. AWS Spot Instance VMs and Google's Preemptible VMs are low-cost processing cycles designed to operate for a short period of time. A user could provision a cluster of 1,000 AWS Spot Instances and theoretically reduce its computing costs for HPC applications, said Willard.

While public cloud today seems to only suit a small subset of HPC applications, the industry's shift toward standardized hardware and new, specialized services from vendors such as AWS and Google have the potential to boost HPC use cases in public cloud.

Next Steps

HPC applications inching toward enterprise

Using HPC for big data analysis

Public cloud adoption trends in the enterprise

HPC on demand makes supercomputing a possibility for any company

Dig Deeper on Public cloud and other cloud deployment models