Your CPU May Have Slowed Down on Wednesday

A Strange Performance Effect

The plot below shows the throughput of filling a region of the given size (varying on the x-axis) with zeros on Skylake (and Ice Lake in the second tab).

The two series were generated under apparently identical conditions: the same binary on the same machine. Only the date the benchmark was run varies. That is, on Tuesday (June 7th) filling with zeros is substantially faster than the same benchmark on Wednesday, at least when the region no longer fits in the L2 cache.

Hump Day Strikes Back

What’s going on here? Are my Skylake and Ice Lake hosts simply work-weary by Wednesday and don’t put in as much effort? Is there a new crypto-coin based on who can store the most zeros and this is a countermeasure to avoid ballooning CPU prices in the face of this new workload?

Believe it or not, it is none of the above!

These hosts run Ubuntu 20.04 and on Wednesday June 8th an update to the intel-microcode OS package was released. After a reboot, this loads the CPU with new microcode that causes the behavior shown above. Specifically, this microcode disables the hardware zero store optimization we discussed in a previous post. It was disabled to mitigate CVE-2020-24512 further described in Intel security advisory INTEL-SA-00464.

To be clear, I don’t know for sure that the microcode disables the zero store optimization – but the evidence is rather overwhelming. After the update, performance is the same when filling zeros as for any other value, and the performance counters tracking L2 evictions suggestion that substantially all evictions are now non-silent (recall from the previous posts that silent evictions were a hallmark of the optimization).

Although I suspect the performance impact will be minuscule on average, this surprise still serves as a reminder that raw CPU performance can silently change due to microcode updates and most Linux distributions and modern Windows have these updates enabled by default. We’ve seen this before. If you are trying to run reproducible benchmarks, you should always re-run your entire suite in order to make accurate comparisons, even on the same hardware, rather than just running the stuff you think has changed.

Mea Culpa and an Unsustainable Path

In writing the earlier blog entries on this topic, I was interested in the performance aspects of this optimization, not its potential as an attack vector. However, merely by observing (and publishing) the results, the optimization was affected: the system under measurement changed as a result of the observation. I can’t be sure that the optimization wouldn’t have eventually been disabled anyway, but it does seem that the proximate cause this change to the microcode was my earlier post.

I am not convinced that removing any optimization which can be used in a timing-based side channel is sustainable. I am not sure this is a thread you want to keep pulling on: practically every aspect of a modern CPU can vary in performance and timing based on internal state. Trying to draw the security boundaries tightly around co-located entities (e.g., processes on the same CPU, especially on the same core), without allowing any leaks seems destined to fail without a complete overhaul of CPU design, likely at the cost of a large amount of performance. There are just too many holes to plug.

I hope that once the wave of vulnerabilities and disclosures that started with Meltdown and Spectre beings to recede, we can start to work on a measured approach to classifying and mitigating timing and other side-channel attacks. This could start by enumerating which performance characteristics are reasonable guaranteed to hold, and which aren’t. For example, it could be specified whether memory access timing may vary based on the value accessed. If it is allowed to vary, the zero store optimization would be allowed.

In any case, I still plan to write about performance-related microarchitectural details. I just hope this outcome does not repeat itself.

Thanks

Stone photo by Colin Watts on Unsplash.

Discussion and Feedback

You can join the discussion on Twitter, Hacker News or r/Intel.

If you have a question or any type of feedback, you can leave a comment below.

If you liked this post, check out the homepage for others you might enjoy.

---

from Hacker News https://ift.tt/3wZwjnf