HPC systems: Out of the lab and into the enterprise

HPC was once sequestered in research labs, but enterprise IT is beginning to see its value for addressing some of the toughest business challenges.


The need for high-performance computing has long been limited to scientists and engineers, but the advent of big data and the availability of cloud computing is bringing HPC systems into the enterprise.

High-performance computing was designed to address computationally intensive problems -- such as weather prediction, oil exploration and problems in biochemistry -- but that is not the extent of its uses. HPC has a wide range of business applications, from analyzing increasingly large volumes of customer data and monitoring real-time network and application logs to modeling complex workflows and simulating product designs. HPC techniques can help address some of the toughest challenges in business and IT, such as predicting which customers will churn, identifying best practices for cross-selling, understanding why products fail and monitoring security events across a corporate network. To benefit from HPC, it helps to understand what hardware, software and methodologies are available and which are appropriate for the enterprise.

Determining enterprise HPC hardware options

High-performance computing is typically executed either on supercomputers, such as IBM Blue Gene, or on clusters of computers connected with high-speed network interconnects. Assuming you don't have millions of dollars to spend on a supercomputer, the cluster option is best suited to the enterprise, because commodity hardware and open source HPC cluster software is readily available.

HPC techniques can help address some of the toughest challenges in business and IT.

When there is consistent and long-term demand for HPC applications, the most reasonable option is to build a cluster on-premises. Vendors such as Amax, Aspen Systems and PCCS Labs offer preconfigured HPC clusters designed to support large numbers of core processors, high-speed networking and high-performance storage. Alternatively, you could use commodity servers and a freely available cluster operating system, such as MOSIX, to build your own cluster on-premises.

Another option is to provision clusters from a public cloud provider. Amazon Web Services (AWS) provides cluster compute instances configured with 2 Intel Xeon E5-2670 processors, 60.5 GB of memory and 10 GB Ethernet. AWS has machine images designed for HPC that provide improved performance with the use of hardware virtualization and, in some cases, support graphics processing units. Virtual servers within a cluster can be kept in a single placement group to minimize latency.

If you'd rather spend more time analyzing your data than configuring clusters, consider using the StarCluster toolkit from MIT; it's designed to streamline the deployment and management of clusters of virtual machines on Amazon Elastic Compute Cloud (EC2). However, if you are developing your own HPC application, Google App Engine, a Platform as a Service selection, may be a better option than Amazon EC2. With Google App Engine, you build your application with the Google App Engine software developer's kit, in one of several supported languages.

Using enterprise methodologies to determine HPC software

Enterprise developers have several options for HPC software, including MapReduce platforms, stream-processing software, graph databases and high-performance, agent-based modeling systems. Each of these is well suited for particular types of HPC use cases and methodologies.

MapReduce is a processing model popularized by Google and implemented by Hadoop. Business problems that fit well with the MapReduce model can be broken down into smaller problems that can be solved simultaneously and have their results combined to produce an output. For example, a marketing manager might want to understand which characteristics of customers are associated with time spent online shopping. Data from Web application logs, which collect details about the geographic location of clients' IP addresses, start and stop times of sessions, and webpages visited can be combined with demographics data from third-party vendors to produce a multifeatured view of customers. Since each customer's interaction with the Web application is independent of other customers' behaviors, this data set is easily divided into subsets and distributed across a cluster.

Hadoop is available in several implementations, including Apache Hadoop, Cloudera Hadoop and MapR Hadoop.

However, not all business problems are easily broken down into smaller problems. For example, after the marketing manager collects information on individual customers and their shopping patterns, she may want insight into connections among customers. Are there clusters of customers linked by social relationships that might influence product purchasing choices? This type of social network analysis is well suited for a graph database, such as Neo4j. Graph databases allow enterprises to easily model entities, such as customers and products, and relationships among those entities, such as customers' social connections or the likelihood of products being sold together. The number of entities and relationships can be quite large in realistic problems and require high-performance computing hardware.

HPC can also help with real-time event detection in the enterprise. Security information management systems, for example, can collect streaming data about events on a network, such as login failures, changes to administrative privileges or access violations. Ideally, these events can be analyzed and correlated with other events as they occur. Stream-processing systems, such as Storm and S4, allow users to analyze continuous streams of data. Splunk, a commercially supported big-machine data analysis tool, is used for a number of IT event management use cases, including application management, IT operations, security and compliance.

HPC outside the data center

Though it's often aligned with big data analysis, HPC is useful for other business processes. Enterprises are using modeling and simulation to help businesses improve plant design, optimize performance of mechanical systems and analyze component failures. Some simulations, such as modeling airflow over a car, require computationally intensive models based on differential equations. Designers can take advantage of modeling and HPC to better understand how products function under a range of circumstances. Commercial tools, such as Matlab, and open source tools, such as NumPy and SciPy packages for Python, are widely used for these kinds of HPC problems.

Other types of simulations are best modeled as systems of interacting agents, such as people changing their shopping behavior during a flu epidemic or businesses adjusting prices in response to changes in market conditions.

Repast is a widely used simulation toolkit with a number of business applications, including consumer market modeling. Repast is available in a Java version for workstations and small clusters, or a C++ version for HPC.

About the author
Dan Sullivan, M.Sc., is an author, systems architect and consultant with more than 20 years of IT experience. He has had engagements in advanced analytics, systems architecture, database design, enterprise security and business intelligence. He has worked in a broad range of industries, including financial services, manufacturing, pharmaceuticals, software development, government, retail and education. Dan has written extensively about topics that range from data warehousing, cloud computing and advanced analytics to security management, collaboration and text mining.

Dig Deeper on Big data, machine learning and AI