CPU Benchmarks for Cloud Providers: Intel vs. AMD vs. Amazon’s Arm-Based Graviton2
Effective benchmarks provide valuable, reproducible insights into performance for various types of systems. Based on 12 open source benchmarks, the 2021 Cloud Report, produced by Cockroach Labs, evaluated how the machines of cloud providers perform in ways that matter most to CockroachDB workloads. We measured performance in terms of CPU, Network, Storage I/O, and an application-level OLTP benchmark. While the report focuses on cloud providers, we also found some interesting takeaways among the three major CPU processors: Intel, AMD and Amazon’s Arm-based Graviton2.
The 2021 Cloud Report is our third annual report looking at the performance of the top three cloud providers: Amazon Web Services (AWS), Google Cloud Platform (GCP) and Microsoft Azure (Azure). The report is based on benchmarks that reflect real-world applications and workloads, and is designed to help application builders understand where each cloud performs best and which best suits their use case.
Intel Excels in Single-Core but Does Not Scale
The CPU benchmark looked at how each cloud provider (AWS, Azure, or Google) performs on single-core and 16-core CPU benchmarks. When we tested dozens of machines across the three clouds, it soon became apparent that what’s inside each machine matters. Intel swept the board in single-core processing, but when more processing units were added, it faltered.
For the single-core CoreMark runs, all the winning machines ran Intel processors. When we looked at the 16-core CoreMark benchmark performance, none of the winning machines ran Intel processors. Instead, the AWS custom-built Graviton2 Processor, which uses a 64-bit ARM architecture, came in just ahead of GCP and Azure’s winning machines — both of which ran AMD processors.
This is bad news for Intel, as the industry gravitates toward the idea that more is better. While single-core performance is important for some types of applications, complex applications — particularly cloud-based ones — benefit from multicore CPUs. This is especially valuable as applications become more feature-rich, which creates added demand. Combined with the fact that AMD (and ARM) CPUs use less power, it’s not surprising that major cloud vendors invest significant resources into their AMD offerings.
Benchmarking AWS, Azure and GCP with CoreMark
We evaluated each cloud’s CPU performance using the CoreMark version 1.0 benchmark. Designed for CPU benchmarking, CoreMark produces a single-number score — the higher the score, the better the performance.
We arrived at CoreMark because it is open source, cloud-agnostic, and more representative of general CPU performance than stress-ng. For example, CoreMark tests against various real-world workloads, like list sort and search.
The CoreMark benchmark can be limited to run on a single vCPU, or execute workloads on multiple vCPUs in parallel. We ran CoreMark in both modes ten times and reported the average number of iterations per second for both the single-core and 16-core results.
CPU Performance Overall
GCP Produces Highest Score on Single-Core, while AWS Scales with Graviton2 on Multiple Cores
We found that CoreMark performance on a single core served as an effective indicator for raw throughput performance. Here GCP led the way, achieving the highest CoreMark score by 10% over the best performing compute-optimized machines in AWS and Azure.
As a less indicative metric of raw throughput than single performance, 16-core CoreMark highlighted each machine’s ability to scale to multiple cores. Amazon’s Graviton2 led the way here.
The AWS Graviton2 Processor, which runs on ARM, enabled the top-performing m6g.4xlarge machine to achieve better multi-CPU scaling than the other machines. AWS’s m6g.4xlarge hit 308,852 iterations per second, while GCP and Azure’s machines produced 5% and 7% lower scores, respectively.
#AWS CPU Performance
Note: Where AWS machines demonstrated minimal variability, results have been grouped by type and processor for improved readability.
While machines with Intel processors outperformed the others on single-core CPU performance, with the notable exception of the c5a series running AMD EPYC 7002 processors, AWS machines with Graviton2 processors consistently outperformed all other AWS machines on 16-core (and, indeed, all other machines in the field). These machines are also AWS’ cheapest.
#GCP CPU Performance
GCP’s compute-optimized machine (c2-standard) demonstrated the best raw CPU performance among GCP machines. Beyond the c2-standard machine, there was minimal variation in the single-core benchmark among machines tested, because the N2 machines have the same processors and only differ in their vCPU-to-memory ratios.
The same applies to the N2D machines. On the 16-core benchmark, the N2D machines with AMD EPYC 7002 Processors and the c2-standard-16 machines led all other GCP machines.
#Azure CPU Performance
Like the single-core benchmark on AWS, Azure machines with AMD processors performed on par with Azure machines with Intel processors. Consistent with both AWS and GCP, Azure machines with AMD processors outperformed Azure machines with Intel processors on the 16-core benchmark.
Notably, the memory-optimized Standard_M16ms machine and the storage-optimized Standard_L16s_v2 machine were the bottom performers among Azure’s machines on both metrics.
For the full results — including how each cloud performed in CPU, Network, Storage I/O and OLTP benchmarks — download the 2021 Cloud Report.