[nova][placement] Openstack only building one VM per machine in cluster, then runs out of resources
Hi all,
I would like to follow up on a discussion from last year [1] since the issue is still present. In a fresh Victoria deployment we faced the same issue and my colleague tracked it down to a reoccuring issue within MariaDB [2]. Quoting from his response to a different thread [3]:
---snip--- MariaDB is version 10.6.5. When running placement API in debug mode, its log reveals a message like
"Over capacity for VCPU on resource provider . Needed: 1, Used: 4118, Capacity: 768.0"
but checking the entries in the placement db / allocations table shows that "4118" is the sum of all resources for the resource provided, not only for the CPU class.
The problem results from errors in the DBMS handling the "outer join" with a subquery while retrieving the currently allocated resources. ---snip---
We found a workaround for this environment, but I wanted to bring some more attention to this, hoping that this will be fixed in MariaDB.
Regards, Eugen
[1] https://lists.openstack.org/pipermail/openstack-discuss/2021-July/023430.htm... [2] https://jira.mariadb.org/browse/MDEV-25714 [3] https://serverfault.com/questions/1064579/openstack-only-building-one-vm-per...
Hello,
We were hit by this issue October last year as well, was introduced when we upgraded MariaDB to 10.4.19 and was potentially fixed in 10.4.20 (but we never verified), we downgraded back to 10.4.18 to workaround the regression.
Best regards
On 31 Jan 2022, at 13:20, Eugen Block eblock@nde.ag wrote:
Hi all,
I would like to follow up on a discussion from last year [1] since the issue is still present. In a fresh Victoria deployment we faced the same issue and my colleague tracked it down to a reoccuring issue within MariaDB [2]. Quoting from his response to a different thread [3]:
---snip--- MariaDB is version 10.6.5. When running placement API in debug mode, its log reveals a message like
"Over capacity for VCPU on resource provider . Needed: 1, Used: 4118, Capacity: 768.0"
but checking the entries in the placement db / allocations table shows that "4118" is the sum of all resources for the resource provided, not only for the CPU class.
The problem results from errors in the DBMS handling the "outer join" with a subquery while retrieving the currently allocated resources. ---snip---
We found a workaround for this environment, but I wanted to bring some more attention to this, hoping that this will be fixed in MariaDB.
Regards, Eugen
[1] https://lists.openstack.org/pipermail/openstack-discuss/2021-July/023430.htm... [2] https://jira.mariadb.org/browse/MDEV-25714 [3] https://serverfault.com/questions/1064579/openstack-only-building-one-vm-per...
participants (2)
-
Eugen Block
-
Tobias Urdin