Although cloud services abstract away and automate much of the underlying IT infrastructure, users still need an...
objective way to measure service performance and the effect it has on their business.
To perform this kind of measurement, users need to define cloud performance metrics. Without these metrics, there is no assurance that businesses get what they pay for, no way to assess cloud benefits and no foundation upon which to build improvements.
Here's some guidance to determine and define the key performance indicators (KPIs) that matter most in your cloud deployment.
Select the right cloud KPIs
There is no one-size-fits-all KPI list, so users must narrow the field of possibilities to find their most relevant technical and business parameters.
Organizations typically choose cloud KPIs based on user needs. Identify any operational or business criteria that users currently seek to improve. For example, a business unit might be perplexed by slow IT response times or the storage costs involved with a critical application.
Next, know what your goals are. In the example above, you'd want to determine much time that business unit expects an IT response to take or how much they want to save on cloud storage costs. These anticipated goals are called critical success factors (CSFs), and you should correlate them to costs, where appropriate.
After you define your users' requirements and overall goals, you can create measurable KPIs. Perform initial KPI measurement to establish an objective behavioral or performance baseline. Then, track those KPIs over time, using collected data, and hold providers accountable for their services in compliance with any service-level agreement (SLA). And be sure to frequently review cloud KPIs to evaluate provider performance and cater to changing business needs.
Now, let's consider four common types of cloud KPIs.
Whether public, private or hybrid cloud, businesses often start with KPIs that reflect the integrity, quality and availability of those cloud services. Such service-oriented KPIs usually track the cloud availability and problem-resolution criteria.
Service KPI examples include:
- Availability: Track the percentage of time that cloud services run normally, and verify that availability meets the provider's SLA. Remember, an availability of 99.9% still means that an organization cannot use the cloud to do business an average of 42 minutes per month.
- Reliability: Reliability typically involves metrics such as mean time to failure (MTTF) and mean time to repair (MTTR). Falling MTTF or increasing MTTR could indicate cloud provider service problems.
- New or resolved incidents: Cloud providers typically use a ticketing system to track problems, changes and help requests. Watch the number of new or resolved incidents per billing cycle, and see how incidents track over time. Increases in new incidents -- or decreases in resolved incidents -- could suggest problems with the provider.
- Average time to resolve incidents: Service quality is frequently measured in resolution response time. Average response times from your cloud provider should remain short and well within the SLA. Increasing response times reduces the perceived quality of service.
If users are not satisfied with a cloud service, they might not use it, or they might seek alternative options. To prevent this, public, private and hybrid cloud deployments can include customer-focused KPIs, such as:
- Time to provision: Speed is a major cloud benefit, and self-service enables non-IT users to engage resources on demand. Provisioning metrics help you assess how quickly cloud infrastructure responds. Longer provisioning times decrease user satisfaction and create issues when users require frequent and rapid workload scaling.
- Workloads deployed: Organizations can indirectly gauge user satisfaction based on how heavily a cloud service is used. Satisfied users tend to move more workloads to the cloud.
- Services in use: You can also measure cloud engagement based on the breadth of services in use. A business might start off only using a few different services, but the scope of those services likely increases over time. Broader usage is an indirect measure of user satisfaction with the cloud provider.
- Customer satisfaction: Combine workload performance metrics with several other cloud KPIs, such as time to provision, number of workloads and MTTF, to gauge if, overall, users get expected results. You can couple these metrics with other tools, such as user satisfaction surveys or focus groups, to know whether the cloud experience is satisfactory.
Organizations also implement cloud KPIs to report on costs. Variability in cloud usage can result in unpredictable and unexpected costs, which makes analysis critical to determine whether a cloud deployment actually benefits the business.
Cost KPIs include:
- Operational costs: This is basically your monthly cloud computing bill. Use this metric to compare actual vs. predicted cloud usage. Since cloud billing is often broken down by user or business group, determine which departments spend the most on cloud services. An upward trend in monthly cloud costs might suggest an increasing number of applications, more complex applications or scaling more resources.
- Infrastructure costs: This metric is typically the cost of remaining local data center resources and services compared to public or private cloud usage. Present this KPI as a fixed monthly cost, and compare it to the cloud's operational costs. Then, use this comparison to gauge another dimension of cloud adoption or user satisfaction.
- Chargeback billing: Private cloud deployments might deduct local IT expenses from departmental budgets. While this is largely a matter of bookkeeping, viewing private cloud usage as a budget line item raises awareness of overall use and helps you rein in any excess or unnecessary use. An increase in chargeback usage can signal greater private cloud adoption.
- Cost of services: SaaS offerings, such as Salesforce or Office 365, make it easy to establish costs for cloud-based services. Compare those recurring expenses against the fixed or average cost of similar in-house services. Ideally, the cost of cloud services should be competitive with -- or lower than -- traditional local deployments.
Infrastructure cloud KPIs, mostly used for private cloud, include granular operational details of the servers, storage, network and other data center elements that underpin that cloud environment.
- Server hardware: This KPI reports on the availability and utilization of servers, storage and networks. For example, hardware KPIs might report the number of physical or virtual CPUs, memory and utilization for each physical server. The might also track the total, available and utilized storage capacity.
- Virtual environment: Virtualization KPIs track the number of VMs and containers in the overall environment and on each physical server, as well as the memory remaining on each server.
- Network infrastructure: This KPI monitors network performance attributes, such as throughput, latency and bandwidth, at strategic points in the physical network. This can reveal possible network bottlenecks that hurt private cloud workload performance.
Although the list of infrastructure metrics can be extensive, they won't generally guide cloud business decisions. In addition, it's extremely difficult to gather infrastructure KPIs from public cloud providers, as that data is not readily exposed to users.