Scheduler sends VM to HV that lacks resources
If I create 20 VMs at once, at least one of them fails with "Exceeded maximum number of retries." When I look at the logs I see that the scheduler sent the VM to a host that doesn't have enough CPU "Free vcpu 14.00 VCPU < requested 16 VCPU."
https://paste.fedoraproject.org/paste/6N3wcDzlbNQgj6hRApHiDQ
I thought that this must be caused by a race condition, so I stopped the scheduler and conductor on 2 controllers, and then created 20 more VMs. Now I see the logs only on controller 3, and some of the failures are now saying "Unable to establish connection to <LB>" but I still see the single scheduler sending VMs to a host that lacks resources "Free vcpu 14.00 VCPU < requested 16 VCPU."
https://paste.fedoraproject.org/paste/lGlVpfbB9C19mMzrWQcHCQ
I'm looking at my nova.conf but don't see anything misconfigured. My filters are pretty standard:
enabled_filters = RetryFilter,AvailabilityZoneFilter,CoreFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,DifferentHostFilter,SameHostFilter
What should I be looking for here? Why would a single scheduler send a VM to a host that is too full? We have lots of compute hosts that are not full:
https://paste.fedoraproject.org/paste/6SX9pQ4V1KnWfQkVnfoHOw
This is the command line I used:
openstack server create --flavor s1.16cx120g --image QSC-P-CentOS6.6-19P1-v4 --network vg-network --max 20 alberttestB
On Tue, 2019-11-12 at 19:42 +0000, Albert Braden wrote:
If I create 20 VMs at once, at least one of them fails with "Exceeded maximum number of retries." When I look at the logs I see that the scheduler sent the VM to a host that doesn't have enough CPU "Free vcpu 14.00 VCPU < requested 16 VCPU."
https://paste.fedoraproject.org/paste/6N3wcDzlbNQgj6hRApHiDQ
I thought that this must be caused by a race condition, so I stopped the scheduler and conductor on 2 controllers, and then created 20 more VMs. Now I see the logs only on controller 3, and some of the failures are now saying "Unable to establish connection to <LB>" but I still see the single scheduler sending VMs to a host that lacks resources "Free vcpu 14.00 VCPU < requested 16 VCPU."
https://paste.fedoraproject.org/paste/lGlVpfbB9C19mMzrWQcHCQ
I'm looking at my nova.conf but don't see anything misconfigured. My filters are pretty standard:
enabled_filters = RetryFilter,AvailabilityZoneFilter,CoreFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter, ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,DifferentHostFilter,SameHostFilter
What should I be looking for here? Why would a single scheduler send a VM to a host that is too full? We have lots of compute hosts that are not full:
https://paste.fedoraproject.org/paste/6SX9pQ4V1KnWfQkVnfoHOw
This is the command line I used:
openstack server create --flavor s1.16cx120g --image QSC-P-CentOS6.6-19P1-v4 --network vg-network --max 20 alberttestB
what version of openstack are you running? if its not using placement then this behaviour is expected as the resources are not claimed untill the vm is booted on the node so there is and interval where the scudler is selecting hosts where you can race with other vm boot.
if you are using placement and you are not using numa or pci pass though, which you do not appear to be based on your enabled filters, then this should not happen and we should dig deeper as there is likely a bug either in your configuration or in nova.
We are running placement under apache:
https://paste.fedoraproject.org/paste/mZviLVe5xONPsXfLqdxI6A
The placement error logs show a lot of GETs but no errors:
https://paste.fedoraproject.org/paste/xDVGaXEdoQ5Z3wHv17Lezg
We are planning to use NUMA but haven't started yet. It's probably a config error. Where should I be looking? This is our nova config on the controllers:
https://paste.fedoraproject.org/paste/kNe1eRimk4ifrAuuN790bg
-----Original Message----- From: Sean Mooney smooney@redhat.com Sent: Tuesday, November 12, 2019 12:22 PM To: Albert Braden albertb@synopsys.com; openstack-discuss@lists.openstack.org Subject: Re: Scheduler sends VM to HV that lacks resources
On Tue, 2019-11-12 at 19:42 +0000, Albert Braden wrote:
If I create 20 VMs at once, at least one of them fails with "Exceeded maximum number of retries." When I look at the logs I see that the scheduler sent the VM to a host that doesn't have enough CPU "Free vcpu 14.00 VCPU < requested 16 VCPU."
https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.fedoraproject.org...
I thought that this must be caused by a race condition, so I stopped the scheduler and conductor on 2 controllers, and then created 20 more VMs. Now I see the logs only on controller 3, and some of the failures are now saying "Unable to establish connection to <LB>" but I still see the single scheduler sending VMs to a host that lacks resources "Free vcpu 14.00 VCPU < requested 16 VCPU."
https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.fedoraproject.org...
I'm looking at my nova.conf but don't see anything misconfigured. My filters are pretty standard:
enabled_filters = RetryFilter,AvailabilityZoneFilter,CoreFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter, ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,DifferentHostFilter,SameHostFilter
What should I be looking for here? Why would a single scheduler send a VM to a host that is too full? We have lots of compute hosts that are not full:
https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.fedoraproject.org...
This is the command line I used:
openstack server create --flavor s1.16cx120g --image QSC-P-CentOS6.6-19P1-v4 --network vg-network --max 20 alberttestB
what version of openstack are you running? if its not using placement then this behaviour is expected as the resources are not claimed untill the vm is booted on the node so there is and interval where the scudler is selecting hosts where you can race with other vm boot.
if you are using placement and you are not using numa or pci pass though, which you do not appear to be based on your enabled filters, then this should not happen and we should dig deeper as there is likely a bug either in your configuration or in nova.
On 11/12/2019 2:47 PM, Albert Braden wrote:
It's probably a config error. Where should I be looking? This is our nova config on the controllers:
https://paste.fedoraproject.org/paste/kNe1eRimk4ifrAuuN790bg
If your deployment is pike or newer (I'm guessing rocky because your other email says rocky), then you don't need these filters:
RetryFilter - alternate hosts bp in queens release makes this moot CoreFilter - placement filters on VCPU RamFilter - placement filters on MEMORY_MB
On 11/12/2019 3:13 PM, Matt Riedemann wrote:
If your deployment is pike or newer (I'm guessing rocky because your other email says rocky), then you don't need these filters:
RetryFilter - alternate hosts bp in queens release makes this moot CoreFilter - placement filters on VCPU RamFilter - placement filters on MEMORY_MB
Sorry, I should have said:
If your deployment is pike or newer then you don't need the CoreFilter or RamFilter.
If your deployment is queens or newer then you don't need the RetryFilter.
Removing these 3 obsolete filters appears to have fixed the problem. Thank you for your advice!
-----Original Message----- From: Matt Riedemann mriedemos@gmail.com Sent: Tuesday, November 12, 2019 1:14 PM To: openstack-discuss@lists.openstack.org Subject: Re: Scheduler sends VM to HV that lacks resources
On 11/12/2019 2:47 PM, Albert Braden wrote:
It's probably a config error. Where should I be looking? This is our nova config on the controllers:
https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.fedoraproject.org...
If your deployment is pike or newer (I'm guessing rocky because your other email says rocky), then you don't need these filters:
RetryFilter - alternate hosts bp in queens release makes this moot CoreFilter - placement filters on VCPU RamFilter - placement filters on MEMORY_MB
I am running stable Queens with hundreds of ironic baremetal nodes. Things are mostly stable but occasionally some baremetal node provisions are failing. These failures have been tracked to nova placement failure leading to 409 errors.My nova and baremetal filters do NOT have the 3 filters you mention. [root@sc-control03 objects]# grep filter /etc/nova/nova.conf | grep filters # * enabled_filters #enabled_filters=RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter#use_baremetal_filters=false#baremetal_enabled_filters=RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ExactRamFilter,ExactDiskFilter,ExactCoreFilter
The baremetal nodes are all using resource class. My image does NOT have the changes for https://review.opendev.org/#/c/565841
Ultimately, nova-conductor is reported "NoValidHost: No valid host was found. There are not enough hosts available"This has been traced to nova-placement-api "Allocation for CUSTOM_RRR430 on resource provider 3cacac3f-9af0-4e39-9bc8-d1f362bdb730 violates min_unit, max_unit, or step_size. Requested: 2, min_unit: 1, max_unit: 1, step_size: 1" Any pointers on what next steps I should be looking at ? thanks,Fred.
Relevant logs: nova-conductor.log2019-11-12 10:26:02.593 1666486 ERROR nova.conductor.manager [req-fa1bfb2e-c765-432d-aa66-e16db8329312 - - - - -] Failed to schedule instances: NoValidHost_Remote: No valid host was found. There are not enough hosts available.Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 154, in select_destinations allocation_request_version, return_alternates) File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 91, in select_destinations allocation_request_version, return_alternates) File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 243, in _schedule claimed_instance_uuids) File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 280, in _ensure_sufficient_hosts raise exception.NoValidHost(reason=reason) NoValidHost: No valid host was found. There are not enough hosts available.
nova-placement-api.log 3cacac3f-9af0-4e39-9bc8-d1f362bdb730 = resource ID of baremetal node 84ea2b90-06b2-489e-92ea-24b859b3c997 = instance ID
2019-11-12 10:26:02.427 4161131 INFO nova.api.openstack.placement.requestlog [req-66a6dc45-8326-4e24-9216-fc77099303ba 1ee9f9bf77294e8e8bf50bb35c581689 acf8cd411e5e4751a61d1ed54e8e874d - default default] 10.33.24.13 "GET /allocations/84ea2b90-06b2-489e-92ea-24b859b3c997" status: 200 len: 111 microversion: 1.0
2019-11-12 10:26:02.461 4161129 WARNING nova.objects.resource_provider [req-6d79841e-6abe-490e-b79b-8d88b04215af 1ee9f9bf77294e8e8bf50bb35c581689 acf8cd411e5e4751a61d1ed54e8e874d - default default] Allocation for CUSTOM_Z370_A on resource provider 3cacac3f-9af0-4e39-9bc8-d1f362bdb730 violates min_unit, max_unit, or step_size. Requested: 2, min_unit: 1, max_unit: 1, step_size: 1 2019-11-12 10:26:02.568 4161129 INFO nova.api.openstack.placement.requestlog [req-6d79841e-6abe-490e-b79b-8d88b04215af 1ee9f9bf77294e8e8bf50bb35c581689 acf8cd411e5e4751a61d1ed54e8e874d - default default] 10.33.24.13 "PUT /allocations/84ea2b90-06b2-489e-92ea-24b859b3c997" status: 409 len: 383 microversion: 1.17
http_access_log10.33.24.13 - - [12/Nov/2019:10:26:02 -0800] "GET /allocations/84ea2b90-06b2-489e-92ea-24b859b3c997 HTTP/1.1" 200 111 "-" "nova-scheduler keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5"10.33.24.13 - - [12/Nov/2019:10:26:02 -0800] "PUT /allocations/84ea2b90-06b2-489e-92ea-24b859b3c997 HTTP/1.1" 409 383 "-" "nova-scheduler keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5"
On Wednesday, November 13, 2019, 11:36:35 AM PST, Albert Braden albert.braden@synopsys.com wrote:
Removing these 3 obsolete filters appears to have fixed the problem. Thank you for your advice!
-----Original Message----- From: Matt Riedemann mriedemos@gmail.com Sent: Tuesday, November 12, 2019 1:14 PM To: openstack-discuss@lists.openstack.org Subject: Re: Scheduler sends VM to HV that lacks resources
On 11/12/2019 2:47 PM, Albert Braden wrote:
It's probably a config error. Where should I be looking? This is our nova config on the controllers:
https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.fedoraproject.org...
If your deployment is pike or newer (I'm guessing rocky because your other email says rocky), then you don't need these filters:
RetryFilter - alternate hosts bp in queens release makes this moot CoreFilter - placement filters on VCPU RamFilter - placement filters on MEMORY_MB
On Thu, 14 Nov 2019, fsbiz@yahoo.com wrote:
Ultimately, nova-conductor is reported "NoValidHost: No valid host was found. There are not enough hosts available"This has been traced to nova-placement-api "Allocation for CUSTOM_RRR430 on resource provider 3cacac3f-9af0-4e39-9bc8-d1f362bdb730 violates min_unit, max_unit, or step_size. Requested: 2, min_unit: 1, max_unit: 1, step_size: 1" Any pointers on what next steps I should be looking at ?
Your request, is asking for CUSTOM_RRR430 will a value of 2, but it is only available as 1. Have a look at your server create request, there's something, probably your flavor, which is unexpected.
Placement and nova scheduler are working correctly with the data they have, the problem is with how inventory is being reported or requested. This could either be with how your ironic nodes are being reported, or with flavors.
2019-11-12 10:26:02.461 4161129 WARNING nova.objects.resource_provider [req-6d79841e-6abe-490e-b79b-8d88b04215af 1ee9f9bf77294e8e8bf50bb35c581689 acf8cd411e5e4751a61d1ed54e8e874d - default default] Allocation for CUSTOM_Z370_A on resource provider 3cacac3f-9af0-4e39-9bc8-d1f362bdb730 violates min_unit, max_unit, or step_size. Requested: 2, min_unit: 1, max_unit: 1, step_size: 1
This is the same issue, but with a different class of inventory
On 11/14/2019 3:12 AM, Chris Dent wrote:
Your request, is asking for CUSTOM_RRR430 will a value of 2, but it is only available as 1. Have a look at your server create request, there's something, probably your flavor, which is unexpected.
https://review.opendev.org/#/c/620111/ comes to mind, I'm not sure if that helps you workaround the problem or not.
Be sure to go through this doc as well:
https://docs.openstack.org/ironic/queens/install/configure-nova-flavors.html...
Specifically the part about overriding the VCPU/MEMORY_MB/DISK_GB values in the baremetal flavors. My guess is maybe you haven't done that and the scheduler is selecting a node based on vcpu/ram/disk that is already fully consumed by another node with the same resource class?
Failing all that, it might be an issue due to https://review.opendev.org/#/c/637217/ which I abandoned because I just didn't have the time or will to push on it any further.
If nothing else the bugs linked to those patches might be helpful with workarounds that CERN did when they were doing their baremetal flavor migration to custom resource classes. There were definitely bumps along the way.
Hi Chris, Thanks for the response.
Your request, is asking for CUSTOM_RRR430 will a value of 2, but it is only available as 1. Have a look at your server create request, there's something, probably your flavor, which is unexpected.
The requests coming in are "forced host" requests. The PaaS layer maintains an inventory of actual bare-metal available nodes and a user has to explicitly selecta baremetal node. The PaaS layer then makes a nova api call for an instance to be createdon that specific baremetal node.
Placement and nova scheduler are working correctly with the data they have, the problem is with how inventory is being reported or requested. This could either be with how your ironic nodes are being reported, or with flavors.As far as I can recall, we've started seeing this particular error only recently after we added another 200 nodes to our flat infrastructure.
Thanks,Fred.
On Thursday, November 14, 2019, 01:18:40 AM PST, Chris Dent cdent+os@anticdent.org wrote:
On Thu, 14 Nov 2019, fsbiz@yahoo.com wrote:
Ultimately, nova-conductor is reported "NoValidHost: No valid host was found. There are not enough hosts available"This has been traced to nova-placement-api "Allocation for CUSTOM_RRR430 on resource provider 3cacac3f-9af0-4e39-9bc8-d1f362bdb730 violates min_unit, max_unit, or step_size. Requested: 2, min_unit: 1, max_unit: 1, step_size: 1" Any pointers on what next steps I should be looking at ?
Your request, is asking for CUSTOM_RRR430 will a value of 2, but it is only available as 1. Have a look at your server create request, there's something, probably your flavor, which is unexpected.
Placement and nova scheduler are working correctly with the data they have, the problem is with how inventory is being reported or requested. This could either be with how your ironic nodes are being reported, or with flavors.
2019-11-12 10:26:02.461 4161129 WARNING nova.objects.resource_provider [req-6d79841e-6abe-490e-b79b-8d88b04215af 1ee9f9bf77294e8e8bf50bb35c581689 acf8cd411e5e4751a61d1ed54e8e874d - default default] Allocation for CUSTOM_Z370_A on resource provider 3cacac3f-9af0-4e39-9bc8-d1f362bdb730 violates min_unit, max_unit, or step_size. Requested: 2, min_unit: 1, max_unit: 1, step_size: 1
This is the same issue, but with a different class of inventory
On 11/14/2019 10:09 AM, fsbiz@yahoo.com wrote:
The requests coming in are "forced host" requests. The PaaS layer maintains an inventory of actual bare-metal available nodes and a user has to explicitly select a baremetal node. The PaaS layer then makes a nova api call for an instance to be created on that specific baremetal node.
To be clear, by forced host you mean creating the server with an availability zone in the format ZONE:HOST:NODE or ZONE:NODE where NODE is the ironic node UUID, correct?
https://docs.openstack.org/nova/latest/admin/availability-zones.html#using-a...
Yeah that's a problem because then the scheduler filters aren't run.
A potential alternative is to create the server using a hypervisor_hostname query hint that will run through the JsonFilter:
https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#j...
Then at least you're not forcing the node and run the scheduler filters.
I forget exactly how the scheduler code works in Queens with respect to forced hosts/nodes on server create but the scheduler still has to allocate resources in placement. It looks like we work around that in Queens by disabling the limit we place on getting allocation candidates from placement:
https://review.opendev.org/#/c/584616/
My guess is your PaaS layer has bugs in it since it's allowing users to select hosts that are already consumed, or it's just racy. Anyway, this is why nova uses placement since Pike for atomic consumption of resources during scheduling.
Thanks Matt for the excellent suggestions in this email and the prior one.I am currently trying to eliminate them one by one and will update. Yes, by forced host I do mean creating the server with an availability zone in the ZONE:NODE format. Yes, I understand the scheduler filters aren't run but why should that bean issue? For now, I am tracing all the logs from the PaaS layer all the way to Openstack nova placement API tosee if there is anything unusual. Thanks,Fred. On Thursday, November 14, 2019, 10:07:15 AM PST, Matt Riedemann mriedemos@gmail.com wrote:
On 11/14/2019 10:09 AM, fsbiz@yahoo.com wrote:
The requests coming in are "forced host" requests. The PaaS layer maintains an inventory of actual bare-metal available nodes and a user has to explicitly select a baremetal node. The PaaS layer then makes a nova api call for an instance to be created on that specific baremetal node.
To be clear, by forced host you mean creating the server with an availability zone in the format ZONE:HOST:NODE or ZONE:NODE where NODE is the ironic node UUID, correct?
https://docs.openstack.org/nova/latest/admin/availability-zones.html#using-a...
Yeah that's a problem because then the scheduler filters aren't run.
A potential alternative is to create the server using a hypervisor_hostname query hint that will run through the JsonFilter:
https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#j...
Then at least you're not forcing the node and run the scheduler filters.
I forget exactly how the scheduler code works in Queens with respect to forced hosts/nodes on server create but the scheduler still has to allocate resources in placement. It looks like we work around that in Queens by disabling the limit we place on getting allocation candidates from placement:
https://review.opendev.org/#/c/584616/
My guess is your PaaS layer has bugs in it since it's allowing users to select hosts that are already consumed, or it's just racy. Anyway, this is why nova uses placement since Pike for atomic consumption of resources during scheduling.
am what version of openstack have you deployed. i did not see that in your email. is it ocata or newer http://lists.openstack.org/pipermail/openstack-dev/2018-January/126283.html i see you have the CoreFilter and RamFilter filters enabled. form octa on they shoudl be disabled as we claim those in placement but it should not break anything on older releases. we have removed them in train after we removed the caching scheduler.
On Tue, 2019-11-12 at 20:47 +0000, Albert Braden wrote:
We are running placement under apache:
https://paste.fedoraproject.org/paste/mZviLVe5xONPsXfLqdxI6A
The placement error logs show a lot of GETs but no errors:
https://paste.fedoraproject.org/paste/xDVGaXEdoQ5Z3wHv17Lezg
We are planning to use NUMA but haven't started yet. It's probably a config error. Where should I be looking? This is our nova config on the controllers:
https://paste.fedoraproject.org/paste/kNe1eRimk4ifrAuuN790bg
-----Original Message----- From: Sean Mooney smooney@redhat.com Sent: Tuesday, November 12, 2019 12:22 PM To: Albert Braden albertb@synopsys.com; openstack-discuss@lists.openstack.org Subject: Re: Scheduler sends VM to HV that lacks resources
On Tue, 2019-11-12 at 19:42 +0000, Albert Braden wrote:
If I create 20 VMs at once, at least one of them fails with "Exceeded maximum number of retries." When I look at the logs I see that the scheduler sent the VM to a host that doesn't have enough CPU "Free vcpu 14.00 VCPU < requested 16 VCPU."
https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.fedoraproject.org...
I thought that this must be caused by a race condition, so I stopped the scheduler and conductor on 2 controllers, and then created 20 more VMs. Now I see the logs only on controller 3, and some of the failures are now saying "Unable to establish connection to <LB>" but I still see the single scheduler sending VMs to a host that lacks resources "Free vcpu 14.00 VCPU < requested 16 VCPU."
https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.fedoraproject.org...
I'm looking at my nova.conf but don't see anything misconfigured. My filters are pretty standard:
enabled_filters = RetryFilter,AvailabilityZoneFilter,CoreFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilte r, ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,DifferentHostFilter,SameHostFilter
What should I be looking for here? Why would a single scheduler send a VM to a host that is too full? We have lots of compute hosts that are not full:
https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.fedoraproject.org...
This is the command line I used:
openstack server create --flavor s1.16cx120g --image QSC-P-CentOS6.6-19P1-v4 --network vg-network --max 20 alberttestB
what version of openstack are you running? if its not using placement then this behaviour is expected as the resources are not claimed untill the vm is booted on the node so there is and interval where the scudler is selecting hosts where you can race with other vm boot.
if you are using placement and you are not using numa or pci pass though, which you do not appear to be based on your enabled filters, then this should not happen and we should dig deeper as there is likely a bug either in your configuration or in nova.
We're on Rocky
-----Original Message----- From: Sean Mooney smooney@redhat.com Sent: Tuesday, November 12, 2019 1:23 PM To: Albert Braden albertb@synopsys.com; openstack-discuss@lists.openstack.org Subject: Re: Scheduler sends VM to HV that lacks resources
am what version of openstack have you deployed. i did not see that in your email. is it ocata or newer https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_pipe... i see you have the CoreFilter and RamFilter filters enabled. form octa on they shoudl be disabled as we claim those in placement but it should not break anything on older releases. we have removed them in train after we removed the caching scheduler.
On Tue, 2019-11-12 at 20:47 +0000, Albert Braden wrote:
We are running placement under apache:
https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.fedoraproject.org...
The placement error logs show a lot of GETs but no errors:
https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.fedoraproject.org...
We are planning to use NUMA but haven't started yet. It's probably a config error. Where should I be looking? This is our nova config on the controllers:
https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.fedoraproject.org...
-----Original Message----- From: Sean Mooney smooney@redhat.com Sent: Tuesday, November 12, 2019 12:22 PM To: Albert Braden albertb@synopsys.com; openstack-discuss@lists.openstack.org Subject: Re: Scheduler sends VM to HV that lacks resources
On Tue, 2019-11-12 at 19:42 +0000, Albert Braden wrote:
If I create 20 VMs at once, at least one of them fails with "Exceeded maximum number of retries." When I look at the logs I see that the scheduler sent the VM to a host that doesn't have enough CPU "Free vcpu 14.00 VCPU < requested 16 VCPU."
https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.fedoraproject.org...
I thought that this must be caused by a race condition, so I stopped the scheduler and conductor on 2 controllers, and then created 20 more VMs. Now I see the logs only on controller 3, and some of the failures are now saying "Unable to establish connection to <LB>" but I still see the single scheduler sending VMs to a host that lacks resources "Free vcpu 14.00 VCPU < requested 16 VCPU."
https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.fedoraproject.org...
I'm looking at my nova.conf but don't see anything misconfigured. My filters are pretty standard:
enabled_filters = RetryFilter,AvailabilityZoneFilter,CoreFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilte r, ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,DifferentHostFilter,SameHostFilter
What should I be looking for here? Why would a single scheduler send a VM to a host that is too full? We have lots of compute hosts that are not full:
https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.fedoraproject.org...
This is the command line I used:
openstack server create --flavor s1.16cx120g --image QSC-P-CentOS6.6-19P1-v4 --network vg-network --max 20 alberttestB
what version of openstack are you running? if its not using placement then this behaviour is expected as the resources are not claimed untill the vm is booted on the node so there is and interval where the scudler is selecting hosts where you can race with other vm boot.
if you are using placement and you are not using numa or pci pass though, which you do not appear to be based on your enabled filters, then this should not happen and we should dig deeper as there is likely a bug either in your configuration or in nova.
participants (5)
-
Albert Braden
-
Chris Dent
-
fsbiz＠yahoo.com
-
Matt Riedemann
-
Sean Mooney