Our link checker heavily relies on AWS and EC2 instances in particular. One of the more difficult decisions when dealing with EC2 is choosing the right instance type. Will burstable and (potentially) cheap T3 instances do the job, or should you pay more for general purpose M5 instances? In this blog post, I will try to shed some light on this and provide answers to the following questions:
Without further ado, let me dive directly into some actual testing. I create a t3.large instance (without Unlimited mode) in the us-east-2 region and select Ubuntu 18.04 as the operating system. The first thing I check is the processor the machine is running on:
$ cat /proc/cpuinfo | grep -E 'processor|model name|cpu MHz'
processor : 0
model name : Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
cpu MHz : 2500.000
processor : 1
model name : Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
cpu MHz : 2500.000
As you can see from the output, the T3 instance has access to two vCPUs (= hyperthreads) of an Intel Xeon Platinum 8175M processor.
In order to test the performance of the processor, I install sysbench …
$ sudo apt-get install sysbench
… and start the benchmark (which stresses the CPU for ten seconds by calculating prime numbers):
$ sysbench --threads=2 cpu run
sysbench 1.0.11 (using system LuaJIT 2.1.0-beta3)
Running the test with following options:
Number of threads: 2
Initializing random number generator from current time
Prime numbers limit: 10000
Initializing worker threads...
Threads started!
CPU speed:
events per second: 1623.24
General statistics:
total time: 10.0012s
total number of events: 16237
Latency (ms):
min: 1.16
avg: 1.23
max: 9.89
95th percentile: 1.27
sum: 19993.46
Threads fairness:
events (avg/stddev): 8118.5000/0.50
execution time (avg/stddev): 9.9967/0.00
The events per second (1623 in this case) is the number you should care about. The higher this number, the higher the performance of the CPU. As points of reference, here are the results from the same test on several other cloud machines as well as some desktop and mobile processors:
| Instance Type | Processor | Threads | Burst | Baseline |
|---|---|---|---|---|
| t3.micro | Intel Xeon Platinum 8175M | 2 | 1654 | 175 |
| t3.medium | Intel Xeon Platinum 8175M | 2 | 1653 | 343 |
| t3.large | Intel Xeon Platinum 8175M | 2 | 1623 | 510 |
| m5.large | Intel Xeon Platinum 8175M | 2 | 1647 | 1647 |
| t3a.nano | AMD EPYC 7571 | 2 | 1510 | 75 |
| t2.large | Intel Xeon E5-2686 v4 | 2 | 1830 | 538 |
| a1.medium | AWS Graviton | 1 | 2281 | 2281 |
| t4g.micro | AWS Graviton2 | 2 | 5922 | 592 |
| m4.large | Intel Xeon E5-2686 v4 | 2 | 1393 | 1393 |
| c4.large | Intel Xeon E5-2666 v3 | 2 | 1583 | 1583 |
| GCP f1-micro | Intel Xeon (Skylake) | 1 | 956 | 190 |
| GCP e2-medium | Intel Xeon | 2 | 1434 | 717 |
| GCP n1-standard-2 | Intel Xeon (Skylake) | 2 | 1435 | 1435 |
| Linode 2GB | AMD EPYC 7501 | 1 | 1236 | 1170 |
| Hetzner CX11 | Intel Xeon (Skylake) | 1 | 970 | 860 |
| Dedicated Server | Intel Xeon E3-1271 v3 | 8 | 7530 | 7530 |
| Desktop | Intel Core i9-9900K | 16 | 19274 | 19274 |
| Laptop | Intel Core i7-8565U | 8 | 8372 | 8372 |
While bursting, T3 instances provide the same CPU performance as equally sized M5 instances, which answers my first question from above.
For the next test, I let the t3.large instance sit unused for 24 hours until it has accrued the maximum number of CPU credits.
T3 instances always start with zero credits and earn credits at a rate determined by the size of the instance. A t3.large instance earns 36 CPU credits per hour with a maximum of 864 credits. According to the AWS documentation, one CPU credit is equal to one vCPU running at 100% for one minute. So how long should my t3.large instance be able to burst to 100% CPU utilization?
If the instance has 864 CPU credits, uses 2 credits per minute, and refills its credits at a rate of 0.6 (= 36/60) per minute, it should have the capacity to burst for 864 / (2 - 0.6) = 617 minutes = 10.3 hours. Let me put that to a test.
sysbench --time=0 --threads=2 --report-interval=60 cpu run
Just as expected, the performance drops to about 30% (the baseline performance for t3.large instances) after about ten and a half hours.
T3 instances with activated Unlimited Mode are allowed to burst even if no CPU credits are available. This comes at a price: a T3 instance that continuously bursts at 100% CPU costs approximately 1.5 times the price of an equally sized M5 instance. However, how reliable is Unlimited Mode? My worry is that AWS puts too many instances on a single physical machine, so not enough spare burst capacity is available. To answer this question, I launch a t3.nano instance with Unlimited Mode and let it run at full steam for about four days.
As promised, there is no drop in CPU performance. The t3.nano instance delivers the full capacity of 2 vCPUs (almost) all the time. Quite impressive!
Instead of running my own network performance tests, I rely on the results that Andreas Wittig published on the cloudonaut blog. He used iperf3 to determine the baseline and burst network throughput for different EC2 instance types. Here are the values for different T3 instances and an m5.large instance:
| Instance Type | Burst (Gbit/s) | Baseline (Gbit/s) |
|---|---|---|
| t3.nano | 5.06 | 0.03 |
| t3.micro | 5.09 | 0.06 |
| t3.small | 5.11 | 0.13 |
| t3.medium | 4.98 | 0.25 |
| t3.large | 5.11 | 0.51 |
| m5.large | 10.04 | 0.74 |
Although an m5.large instance costs only about 15% more than a t3.large instance, it provides a 50% higher baseline throughput and more than double the burst capacity.
When determining how much network throughput you need, consider that EBS volumes are network-attached.
T3 instances are great! Even a t3.nano instance at a monthly on-demand price of less than $4 gives you access to the full power of two hyperthreads on an Intel Xeon processor and at burst runs as fast as a $70 m5.large instance. By activating Unlimited Mode, you can easily insure yourself against running out of CPU credits and being throttled.
If you don’t need the 8 GiB of memory (RAM) that an m5.large instance provides and can live with the lower network throughput, one of the smaller T3 instances with activated Unlimited Mode might be the much more cost-effective choice. In the end, it depends on how high the average CPU usage of your instance is. The table below lists the CPU utilizations up to which bursting T3 instances remain cheaper than an m5.large instance. Please note that the calculations are based on the on-demand prices and might be different when using reserved instances.
| Instance Type | Memory | Cost-effective if average CPU usage is less than |
|---|---|---|
| t3.nano | 0.5 GiB | 95.8% |
| t3.micro | 1 GiB | 95.6% |
| t3.small | 2 GiB | 95.2% |
| t3.medium | 4 GiB | 74.4% |
| t3.large | 8 GiB | 42.5% |
Last but not least, let me state that you shouldn’t rely solely on the benchmarks and comparisons in this post. You are welcome to use my finding as a first guide when choosing the right instance type, but don’t forget to run your own tests and make your own calculations.
