Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Is Google's PerfKit a reliable cloud benchmarking tool?

As the IaaS market grows, choosing the right vendor becomes challenging. Google's PerfKit Benchmarker can help, but will it provide unbiased results?

In a growing pool of infrastructure as a service options, choosing the right vendor can be a challenge. With that in mind, Google released PerfKit Benchmarker, an open source tool that evaluates virtual machine performance in the three major IaaS providers: Amazon Web Services, Microsoft Azure and Google Compute Engine.

However, potential conflict of interest may cause some users to question whether PerfKit Benchmarker provides accurate and objective results.

The best way to assess any benchmark is to dig into the details. And, since all the PerfKit code is available on GitHub, this is easy to do.

Written in Python, the PerfKit benchmarking tool provides classes and methods to implement common IaaS operations. For example, there is code to implement VM objects, create scratch disks and execute commands on VMs. Additionally, specialized code manages IaaS-specific object features, such as Azure disks and AWS firewalls.

PerfKit implements benchmark tests for basic operations, such as block I/O, as well as many commonly used applications, including Cassandra, MongoDB and Redis. The benchmarker uses the Yahoo Cloud Serving Benchmark (YCSB) framework to run some of the loads, such as the MongoDB benchmark. Other tests are implemented using reasonable choices for operations. For example, the block I/O test involves tasks such as writing through the file system instead of using direct I/O, sequentially writing data, randomly reading 10% of bytes written and sequentially reading all bytes written.

Documentation states that PerfKit Benchmarker uses default configurations in the benchmark tests, which mitigates the risk of tests favoring one IaaS provider over another -- in terms of their configuration settings, at least. And because many cloud users accept default configurations when spinning up VMs, it seems reasonable for benchmark comparisons to use cloud providers' defaults. In addition, this practice isn't likely to produce biased benchmark results in ways that don't reflect typical use cases.

PerfKit Benchmarker builds on open source tools, such as the boto Python package and YCSB. The benchmarker code is straightforward and accessible to developers with Python and cloud admin experience. While there's always the risk of biased benchmark results, transparency is the best way to mitigate that risk.

About the author:
Dan Sullivan holds a master of science degree and is an author, systems architect and consultant with more than 20 years of IT experience. He has had engagements in advanced analytics, systems architecture, database design, enterprise security and business intelligence. He has worked in a broad range of industries, including financial services, manufacturing, pharmaceuticals, software development, government, retail and education. Dan has written extensively about topics that range from data warehousing, cloud computing and advanced analytics to security management, collaboration and text mining.

Next Steps

Getting to know Google's cloud platform

Five links to familiarize yourself with Google's cloud

Comparing and contrasting Google cloud and AWS

Dig Deeper on Managing cloud infrastructure

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

As the article points out, PerKit Benchmarker is open source, so it would be pretty risky for Google to stack the deck and provide unbiased results. It would also be inconsistent with the image that Goolge wants to project and have far-reaching impacts, not just for their IaaS offerings and GCE, but for all things Google.
@Dan - Thanks for the thoughtful post. We are working hard to be transparent and more complete. Just this week we added DigitalOcean. Rackspace is coming in the next couple of days.

@mcorum - We would rather have the right methodology and be last in the results than have the wrong methodology be first. We really look to the community to make sure we get this right.

*** Disclose - I am the technical lead for the PerfKitBenchmaker project at Google
@Dan - Thanks for the post. We are working hard to create a solid and consistent methodology for analyzing Cloud Performance.
@mcorum - OpenSource is one of the way we work to be really transparent. We would rather have the right comparison and the lowest score than the wrong methodology and the best score. We really look to the community to help ensure its correct.

* Disclosure I am the tech lead and manager for the PerfKitBenchMarker project at Google.