Workloads on Arm-based AWS instances

May 2023

---

Not too long ago I suggested using Arm-based EC2 instances on AWS as a way to further cost savings.

Depending on the applications and types of workloads, that might not be worth the trouble. As usual, one has to measure. And measure the results again.

However, I went ahead and decided I should check how much of a non-starter that could be. I did that by evaluation one of the most common use cases, a simple REST-based endpoint.

Here’s what I found.

Test setup

The test service exposes a single endpoint to accept POST requests, with a JSON payload, and respond with the exact same that was received.

I did that using 4 different and (I believe) commonly used Web frameworks:

• Django (Python)
• Spring Boot (Java)
• Actix-Web (Rust)
• Gin (Go)

The services were then tested under high request rate using hey with a JSON payload of 6k. Each combination of framework and architecture processed 1M requests.

The EC2 instances used were

(instance types and their configuration as of May 2023)

• for x64, a t3a.micro with 2 vCPUs and 1G of memory
• for Arm. a t4g.micro with 2 vCPUs and 1G of memory

I chose these instance types based on number of cores and memory size, though the architectures are very different and to for certain applications, these may not be the best attributes for comparison.

Indicators and expectations

I am looking for throughput and latency for the same service and framework on both architectures.

For that I primarily collected service response time (round trip) at various percentiles.

It’s important to note that network and instance jitter can affect the results and I would sometimes have a single request, in 10M, to take 10x the expected time.

Repeating the test many times over under different instances was necessary to eliminate the outliers.

Note on payload content

The JSON payload consisted of various data types, long and short arrays of single and complex objects.

I also tested much smaller payloads and the results were the same, so I decided to use 6k as a large enough payload size.

The results

Before diving into the results, I want to mention this is not a comparison between frameworks (though waiting 10-20x for the python app was a little annoying) rather, I was interested in seeing how the same framework (and language compiler/interpreter) behaves on both architectures.

Python + Django

Response time in milliseconds

%	x86_64	Arm
99.50	55.6	51.8
99.90	59.7	57.0
99.99999	92.3	95.8
Average	47.2	47.4
Min	5.8	5.8
Max	92.4	95.0

Distribution x86_64

Distribution arm64

On the surface it appears that the x86 architecture performed a 2-3% better at the high percentile but up to 99.5, Arm was actually 8% faster.

Java + Spring Boot

Response time in milliseconds

%	x86_64	Arm
99.50	8.1	7.3
99.90	10.9	9.0
99.99999	17.2	23.2
Average	1.7	1.6
Min	1.3	1.3
Max	17.3	23.3

Distribution x86_64

Distribution arm64

Again we see the same pattern, where the Arm test performs better up until the 99.5 percentile but not so much after that.

The histograms show a higher number of requests in the lower response time bands.

Go + Gin

Response time in milliseconds

%	x86_64	Arm
99.50	7.2	7.2
99.90	8.9	8.8
99.99999	15.5	16.9
Average	1.6	1.5
Min	1.3	1.3
Max	15.6	17.0

Distribution x86_64

Distribution arm64

The same pattern again with a 8% slower response time for Arm at the 99.99998th percentile.

Rust + Actix-web

Response time in milliseconds

%	x86_64	Arm
99.50	2.7	3.0
99.90	7.2	7.2
99.99999	9.6	10.9
Average	1.4	1.4
Min	1.2	1.2
Max	9.7	11.0

Distribution x86_64

Distribution arm64

Much like the results for go, thought the histograms suggest a much smaller right tail for Arm.

Final notes

It does appear that the Arm-based instances can’t consistently maintain the same performance at high request rates.

However, I wouldn’t take these results as discouraging, quite the opposite. It’s very possible that most services won’t need to keep those levels of performance at “7 nines” percentiles and the differences these tests show would be immaterial.

Arm instances on AWS are around 10% cheaper than x86(10.6% to be more precise, at the time of this writing). So even if more instances are needed to keep up with load, it’s possible the total cost would still be lower.

If the workloads you’re running don’t depend on specific instruction sets, I’d suggest giving Arm a try.

---

from Hacker News https://ift.tt/OsmGX13