Hi, If we create pinned vms, is there a chance of cpu steal in this scenario ? If so what could be the reason since we have dedicated cpu assigned to the guest ? What would happen if it is a physical cpu, directly assigned to the vm ? Does it experience cpu steal in this case as well ? Thanks GK
Hi,
If we create pinned vms, is there a chance of cpu steal in this scenario ? yes, libvirt and by extention nova does not isolate kernel or operating system
On Wed, 2024-05-22 at 15:41 +0530, Gk Gk wrote: processes from running on the cpu_dedicated_set cpus.
If so what could be the reason since we have dedicated cpu assigned to the guest ? kernel threads, os processes or irqs can all "steal" time from the vm to prevent this you should adjust the default core mask in systemd for all operating system userland process using
What would happen if it is a physical cpu, directly assigned to the vm ? Does it experience cpu steal in this case as well ? yes it would be the same. the other souce of cpu steal is the qemu emulator thread
[Manager] CPUAffinity= in /etc/systemd/system.conf addtionally you shoudl ensure that if irq balance is isntalled that it is confirured to not balance irqs to the cpu_dedicated_set i hesitate to mention the third option as i strongly recommend agaisnt using it but if you want to be extra sure that host process dont steal you can us the kernel isolcpus parmater. however that should reallly only be done on a realtime host and should never contain any cores form the cpu_shared_set. if you add cores to isolcpus form thet cpu_dedicated_set you need to ensure that cpu_shared_set is also defiend and your flavor have hw:emulator_thread_policy=shared if you dont do that your vms may take 10 of minutes to boot and be unreliable and we will be talkign about that problem in a week or so :) i mention that the kernel parmater exists but unless you do your reasearch and understand what it does you shoudl not use it. that is why we recommend using hw:emulator_thread_policy=shared for all vms using cpu pinning and have 1-2 cores in cpu_shared_set even if the host is going to used exclustively for pinned vms if you are going to use it exlucsively for pinned vms you should also adjust the host reserved cpu config option so that no other vms can boot on the cpu_shared_set and they are only used for the qemu emulator thread the emulator thread is used for disk io and networking emulation as well as the monitor process that is used ot recives commands form nova/libivrt. it is often mostly idel but it does run on the same cores as the vms cpus so it steals tiem form the vms cpus by default.cpu_shared_set + hw:emulator_thread_policy=shared moves the emulator thread ot other cores to aoivd that stealing.
Thanks GK
Thanks for the response. Appreciate it. Let me know or please send me your views on "vms may take 10 of minutes to boot and be unreliable " . Also just a reminder to point me to any link or document for the previous question regarding disk capacity check by placement service before scheduling the vm on a hypervisor ? Thanks GK On Wed, May 22, 2024 at 5:14 PM <smooney@redhat.com> wrote:
Hi,
If we create pinned vms, is there a chance of cpu steal in this scenario ? yes, libvirt and by extention nova does not isolate kernel or operating system
If so what could be the reason since we have dedicated cpu assigned to
On Wed, 2024-05-22 at 15:41 +0530, Gk Gk wrote: processes from running on the cpu_dedicated_set cpus. the
guest ? kernel threads, os processes or irqs can all "steal" time from the vm to prevent this you should adjust the default core mask in systemd for all operating system userland process using
[Manager] CPUAffinity=
in /etc/systemd/system.conf
addtionally you shoudl ensure that if irq balance is isntalled that it is confirured to not balance irqs to the cpu_dedicated_set
i hesitate to mention the third option as i strongly recommend agaisnt using it but if you want to be extra sure that host process dont steal you can us the kernel isolcpus parmater. however that should reallly only be done on a realtime host and should never contain any cores form the cpu_shared_set. if you add cores to isolcpus form thet cpu_dedicated_set you need to ensure that cpu_shared_set is also defiend and your flavor have hw:emulator_thread_policy=shared if you dont do that your vms may take 10 of minutes to boot and be unreliable and we will be talkign about that problem in a week or so :)
What would happen if it is a physical cpu, directly assigned to the vm ? Does it experience cpu steal in this case as well ? yes it would be the same. the other souce of cpu steal is the qemu emulator thread
i mention that the kernel parmater exists but unless you do your reasearch and understand what it does you shoudl not use it. that is why we recommend using hw:emulator_thread_policy=shared for all vms using cpu pinning and have 1-2 cores in cpu_shared_set even if the host is going to used exclustively for pinned vms if you are going to use it exlucsively for pinned vms you should also adjust the host reserved cpu config option so that no other vms can boot on the cpu_shared_set and they are only used for the qemu emulator thread the emulator thread is used for disk io and networking emulation as well as the monitor process that is used ot recives commands form nova/libivrt. it is often mostly idel but it does run on the same cores as the vms cpus so it steals tiem form the vms cpus by default.cpu_shared_set + hw:emulator_thread_policy=shared moves the emulator thread ot other cores to aoivd that stealing.
Thanks GK
participants (2)
-
Gk Gk
-
smooney＠redhat.com