[ops] [nova] "invalid argument: shares xxx must be in range [1, 10000]" after 1:25.2.0 to 1.25.2.1. update
Dear all We have recently updated openstack nova on some AlmaLinux9 compute nodes running Yoga from 1:25.2.0 to 1.25.2.1. After this operation some VMs don't start anymore. In the log it is reported: libvirt.libvirtError: invalid argument: shares \'57344\' must be in range [1, 10000]\n'} libvirt version is 9.0.0-10.3 A quick google search suggests that it is something related to cgroups and it is fixed in libvirt >= 9.1 (which is not yet in the almalinux9 repos). Did I get it right ? Thanks, Massimo
On Thu, 2023-10-05 at 16:53 +0200, Massimo Sgaravatto wrote:
Dear all
We have recently updated openstack nova on some AlmaLinux9 compute nodes running Yoga from 1:25.2.0 to 1.25.2.1. After this operation some VMs don't start anymore. In the log it is reported:
libvirt.libvirtError: invalid argument: shares \'57344\' must be in range [1, 10000]\n'}
libvirt version is 9.0.0-10.3
A quick google search suggests that it is something related to cgroups and it is fixed in libvirt >= 9.1 (which is not yet in the almalinux9 repos). Did I get it right ? not quite
it is reated to cgroups but the cause is that in cgroups_v1 the maxvlaue of shares i.e. cpu_shares changed form make int to 10000 in cgroups_v2 so the issue is teh vm requested a cpu share value of 57344 which is not vlaid on an OS that is useing cgroups_v2 libvirt will not clamp the value nor will nova. you have to change the volue in your flavor and resize the vm.
Thanks, Massimo
Hi Massimo, We are using Ubuntu for our environments and we experienced the same issue during upgrade from Yoga/Focal to Yoga/Jammy. On Yoga/Focal cgroups_v1 were used, and cpu_shares parameter value was cpu count * 1024. From Jammy cgroups_v2 have been implemented, and cpu_shares value has been set by default to 100. It has hard limit of 10000, so flavors with more than 9vCPUs won't fit. If you need to fix this issue without stopping VMs, you can set cpu_shares with libvirt command: virsh schedinfo $domain --live cpu_shares=100 for more details about virsh schedinfo visit: https://libvirt.org/manpages/virsh.html#schedinfo BR, Franciszek
On 5 Oct 2023, at 21:17, smooney@redhat.com wrote:
On Thu, 2023-10-05 at 16:53 +0200, Massimo Sgaravatto wrote:
Dear all
We have recently updated openstack nova on some AlmaLinux9 compute nodes running Yoga from 1:25.2.0 to 1.25.2.1. After this operation some VMs don't start anymore. In the log it is reported:
libvirt.libvirtError: invalid argument: shares \'57344\' must be in range [1, 10000]\n'}
libvirt version is 9.0.0-10.3
A quick google search suggests that it is something related to cgroups and it is fixed in libvirt >= 9.1 (which is not yet in the almalinux9 repos). Did I get it right ? not quite
it is reated to cgroups but the cause is that in cgroups_v1 the maxvlaue of shares i.e. cpu_shares changed form make int to 10000 in cgroups_v2 so the issue is teh vm requested a cpu share value of 57344 which is not vlaid on an OS that is useing cgroups_v2 libvirt will not clamp the value nor will nova. you have to change the volue in your flavor and resize the vm.
Thanks, Massimo
Thanks a lot Franciszek ! I was indeed seeing the problem with a VM big 56 vcpus while I didn't see the issue with a tiny instance Thanks again ! Cheers, Massimo On Fri, Oct 6, 2023 at 8:58 AM Franciszek Przewoźny <fprzewozny@opera.com> wrote:
Hi Massimo,
We are using Ubuntu for our environments and we experienced the same issue during upgrade from Yoga/Focal to Yoga/Jammy. On Yoga/Focal cgroups_v1 were used, and cpu_shares parameter value was cpu count * 1024. From Jammy cgroups_v2 have been implemented, and cpu_shares value has been set by default to 100. It has hard limit of 10000, so flavors with more than 9vCPUs won't fit. If you need to fix this issue without stopping VMs, you can set cpu_shares with libvirt command: virsh schedinfo $domain --live cpu_shares=100 for more details about virsh schedinfo visit: https://libvirt.org/manpages/virsh.html#schedinfo
BR, Franciszek
On 5 Oct 2023, at 21:17, smooney@redhat.com wrote:
On Thu, 2023-10-05 at 16:53 +0200, Massimo Sgaravatto wrote:
Dear all
We have recently updated openstack nova on some AlmaLinux9 compute nodes running Yoga from 1:25.2.0 to 1.25.2.1. After this operation some VMs don't start anymore. In the log it is reported:
libvirt.libvirtError: invalid argument: shares \'57344\' must be in range [1, 10000]\n'}
libvirt version is 9.0.0-10.3
A quick google search suggests that it is something related to cgroups and it is fixed in libvirt >= 9.1 (which is not yet in the almalinux9 repos). Did I get it right ?
not quite
it is reated to cgroups but the cause is that in cgroups_v1 the maxvlaue of shares i.e. cpu_shares changed form make int to 10000 in cgroups_v2 so the issue is teh vm requested a cpu share value of 57344 which is not vlaid on an OS that is useing cgroups_v2 libvirt will not clamp the value nor will nova. you have to change the volue in your flavor and resize the vm.
Thanks, Massimo
Thanks for your answer I guess you are referring to the cpu_shares in the flavor quota [*] Actually I never explicitly set a value of cpu_shares in the flavor ... [*] https://docs.openstack.org/nova/yoga/admin/resource-limits.html On Thu, Oct 5, 2023 at 9:17 PM <smooney@redhat.com> wrote:
On Thu, 2023-10-05 at 16:53 +0200, Massimo Sgaravatto wrote:
Dear all
We have recently updated openstack nova on some AlmaLinux9 compute nodes running Yoga from 1:25.2.0 to 1.25.2.1. After this operation some VMs don't start anymore. In the log it is reported:
libvirt.libvirtError: invalid argument: shares \'57344\' must be in range [1, 10000]\n'}
libvirt version is 9.0.0-10.3
A quick google search suggests that it is something related to cgroups and it is fixed in libvirt >= 9.1 (which is not yet in the almalinux9 repos). Did I get it right ? not quite
it is reated to cgroups but the cause is that in cgroups_v1 the maxvlaue of shares i.e. cpu_shares changed form make int to 10000 in cgroups_v2 so the issue is teh vm requested a cpu share value of 57344 which is not vlaid on an OS that is useing cgroups_v2 libvirt will not clamp the value nor will nova. you have to change the volue in your flavor and resize the vm.
Thanks, Massimo
Thanks for your answer
I guess you are referring to the cpu_shares in the flavor quota [*]
Actually I never explicitly set a value of cpu_shares in the flavor ... if you are seeing this for instance where its not explictly in the falvor then
On Fri, 2023-10-06 at 09:04 +0200, Massimo Sgaravatto wrote: this is form the old behavior where we set a default share value as flavor.vcpus *1000 in newer version of openstack we never set an implict value https://github.com/openstack/nova/commit/f77a9fee5b736899ecc39d33e4f4e4012ce... change is aviable in zed+ i know this has been backported downstream to wallaby so its possibel to do i belive there was consern with upstream backporting do to cpu starvation if you were hevialy using this funtionatlity. that is why it was not backported upstream in the past but you could apply it too your own cloud.
[*] https://docs.openstack.org/nova/yoga/admin/resource-limits.html
On Thu, Oct 5, 2023 at 9:17 PM <smooney@redhat.com> wrote:
On Thu, 2023-10-05 at 16:53 +0200, Massimo Sgaravatto wrote:
Dear all
We have recently updated openstack nova on some AlmaLinux9 compute nodes running Yoga from 1:25.2.0 to 1.25.2.1. After this operation some VMs don't start anymore. In the log it is reported:
libvirt.libvirtError: invalid argument: shares \'57344\' must be in range [1, 10000]\n'}
libvirt version is 9.0.0-10.3
A quick google search suggests that it is something related to cgroups and it is fixed in libvirt >= 9.1 (which is not yet in the almalinux9 repos). Did I get it right ? not quite
it is reated to cgroups but the cause is that in cgroups_v1 the maxvlaue of shares i.e. cpu_shares changed form make int to 10000 in cgroups_v2 so the issue is teh vm requested a cpu share value of 57344 which is not vlaid on an OS that is useing cgroups_v2 libvirt will not clamp the value nor will nova. you have to change the volue in your flavor and resize the vm.
Thanks, Massimo
Ok thanks I still do not fully understand why I see the problem with nova 1.25.2:1 while it works with v. 1.25.2:0 (without changing any other packages). According to the release notes only this change [*] was introduced [*] https://bugs.launchpad.net/nova/+bug/1941005 Cheers, Massimo On Fri, Oct 6, 2023 at 12:25 PM <smooney@redhat.com> wrote:
Thanks for your answer
I guess you are referring to the cpu_shares in the flavor quota [*]
Actually I never explicitly set a value of cpu_shares in the flavor ... if you are seeing this for instance where its not explictly in the falvor
On Fri, 2023-10-06 at 09:04 +0200, Massimo Sgaravatto wrote: then this is form the old behavior where we set a default share value as flavor.vcpus *1000 in newer version of openstack we never set an implict value
https://github.com/openstack/nova/commit/f77a9fee5b736899ecc39d33e4f4e4012ce... change is aviable in zed+ i know this has been backported downstream to wallaby so its possibel to do i belive there was consern with upstream backporting do to cpu starvation if you were hevialy using this funtionatlity. that is why it was not backported upstream in the past but you could apply it too your own cloud.
[*] https://docs.openstack.org/nova/yoga/admin/resource-limits.html
On Thu, Oct 5, 2023 at 9:17 PM <smooney@redhat.com> wrote:
On Thu, 2023-10-05 at 16:53 +0200, Massimo Sgaravatto wrote:
Dear all
We have recently updated openstack nova on some AlmaLinux9 compute
nodes
running Yoga from 1:25.2.0 to 1.25.2.1. After this operation some VMs don't start anymore. In the log it is reported:
libvirt.libvirtError: invalid argument: shares \'57344\' must be in range [1, 10000]\n'}
libvirt version is 9.0.0-10.3
A quick google search suggests that it is something related to cgroups and it is fixed in libvirt >= 9.1 (which is not yet in the almalinux9 repos). Did I get it right ? not quite
it is reated to cgroups but the cause is that in cgroups_v1 the maxvlaue of shares i.e. cpu_shares changed form make int to 10000 in cgroups_v2 so the issue is teh vm requested a cpu share value of 57344 which is not vlaid on an OS that is useing cgroups_v2 libvirt will not clamp the value nor will nova. you have to change the volue in your flavor and resize the vm.
Thanks, Massimo
participants (3)
-
Franciszek Przewoźny
-
Massimo Sgaravatto
-
smooney＠redhat.com