On Thu, Oct 21, 2021, at 10:49 AM, Kashyap Chamarthy wrote:
Summary -------
RHEL-9 / CentOS-9 (but not Fedora) has switched[1] to a new baseline microarchitecture called "x86-64-v2". This is to bring in support for additioal low-level CPU instructions, among other reasons. Now, if you've explicitly configured "cpu_mode=none" in `nova.conf` on your compute nodes — which results in the guest getting the extremely undesirable "qemu64" CPU model — it will refuse to boot RHEL-9 or CentOS-9 guests.
To fix this, please update the CPU model to "Nehalem". It is the oldest CPU model that is compatible with CentOS-9/RHEL-9 "x86-64-v2". Further, Nehalem also works with `virt_type=kvm|qemu`, _and_ on both Intel and AMD hardware. So this is a good alternative.
Thank you for looking into this and providing such detailed information. It has been really helpful.
Details -------
Nova has three config attributes to setup various aspect of a guest CPU: `cpu_mode`, `cpu_model[s]`, and `cpu_model_extra_flags`. Examples of how to use these are in the documentation[2]. If you're using `cpu_mode = none` (e.g. upstream DevStack defalts to it for understandable reasons, mainly live-migration compatiblity):
[libvirt] cpu_mode = none
... and want to boot CentOS-9, replace the above with the custom model, "Nehalem", which is the oldest CPU model that's compatible with the new x86-64-v2 baseline:
[libvirt] cpu_mode = custom cpu_model = Nehalem
The same applies if you're using "qemu64" or "kvm64" with, or without any custom CPU flags — i.e. use Nehalem. (Also, please refer to[3] for more fine-grained recommendations of guest CPU configuration. It's a long document, but a patient reader will be rewarded.)
Why is "qemu64" model undesirable for production? -------------------------------------------------
For those wondering about it, a few reasons why `qemu64` CPU model is not at all desirable:
(1) It is vulnerable to many of the Spectre and other side-channel security flaws. To see this in "action", you can launch a guest with 'qemu64' CPU model, and then run the below:
$ cd /sys/devices/system/cpu/vulnerabilities/ $ grep . * l1tf:Mitigation: PTE Inversion mds:Vulnerable: ... no microcode; SMT Host state unknown meltdown:Mitigation: PTI spec_store_bypass:Vulnerable spectre_v1:Mitigation: usercopy/swapgs barriers ... spectre_v2:Mitigation: Full generic retpoline ...
Notice the "Vulnerable" entries.
(2) "qemu64" does not support several critical CPU features:
(a) AES (Advanced Encryption Standard) instruction, which is important for imporved TLS performance and encryption.
(b) RDRAND instruction: without this, guests can get starved for entropy.
(c) PCID flag: an obscure-but-important flag that'll lower the performance degradation that you incur from the "Meltdown" security fixes.
Probably there are more reasons that I don't know of.
An understandable reason why CI systems running in a cloud environment go with 'qemu64' is convenience: with 'qemu64', you can live-migrate a guest regardless of its underlying hardware (whether it's Intel or AMD). That's one main reason why upstream DevStack defaults to it.
I've got a change up to Devstack to convert it over to Nehalem by default [5]. So far it looks good, but we will want to recheck it a few times and make sure we have good test coverage across the clouds we run testing on just to be sure that the CPUs we get from those clouds are able to support this CPU type. Good news is that we successfully built a centos-9-stream image and booted it with the Nehalem change in place [6].
* * *
Overall, the thumb-rule here is to either always explicitly specify a "sane" CPU model, based on the recommendations here[3]. Or to use Nova/libvirt's default ("host-model").
Devstack is currently setting cpu_mode to none. Should Nova be updated to make this result in a better behavior? Is this literally not passing a cpu mode to libvirt/qemu and allowing them to choose a default? If so maybe libvirt/qemu need to update their defaults?
[1] https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-li... [2] https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.cpu... [3] https://www.qemu.org/docs/master/system/i386/cpu.html#recommendations-for-kv... [4] https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/.zuu...
[5] https://review.opendev.org/c/openstack/devstack/+/815020 [6] https://zuul.opendev.org/t/openstack/build/b5841d4d264c4c8f93d2368500d6221d
-- /kashyap