RE: [ops][nova][victoria] Unable to Migrate: Host filter ignoring hosts
All; Thank you for reading my previous email, despite its poor Subject. I have attached a subset of the Nova Scheduler log, with debugging enabled. I have added line numbers to the beginning, in order to facilitate referencing. At line 18 it appears to select the requested server for the migration, but then at line 38, it rejects it. Am I missing something here? Is it unable to assign the resources through placement? Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: melanie witt [mailto:melwittt@gmail.com] Sent: Wednesday, July 28, 2021 1:24 PM To: Dominic Hilsbos; openstack-discuss@lists.openstack.org Subject: Re: [ops][nova] On Wed, 28 Jul 2021, 20:10 +0000, <DHilsbos@performair.com> wrote:
All;
I'm having a problem, though I'm not certain of the full scope.
The most obvious issue is that I can't live migrate instances. The server I want to send the instance to gets filtered out, though I don't see any logs that tell me why it's being filtered out.
It might be related that all of my new instances seems to land on a single host.
An pointers on where to look for logs for Host Filters?
You'll want to set [DEFAULT]debug = True in the nova.conf for your nova-scheduler service and restart or HUP the service. And then the logging per scheduler filter (each one will show the starting and ending number of hosts) will be included at DEBUG log level in the nova-scheduler log. -melwitt
Do you see anything in the placement.log? Are there enough resources on that node? If you have it installed you could check the current allocations with the osc-placement command: openstack resource provider list and then openstack resource provider show --allocations <UUID> Zitat von DHilsbos@performair.com:
All;
Thank you for reading my previous email, despite its poor Subject.
I have attached a subset of the Nova Scheduler log, with debugging enabled. I have added line numbers to the beginning, in order to facilitate referencing.
At line 18 it appears to select the requested server for the migration, but then at line 38, it rejects it.
Am I missing something here? Is it unable to assign the resources through placement?
Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: melanie witt [mailto:melwittt@gmail.com] Sent: Wednesday, July 28, 2021 1:24 PM To: Dominic Hilsbos; openstack-discuss@lists.openstack.org Subject: Re: [ops][nova]
On Wed, 28 Jul 2021, 20:10 +0000, <DHilsbos@performair.com> wrote:
All;
I'm having a problem, though I'm not certain of the full scope.
The most obvious issue is that I can't live migrate instances. The server I want to send the instance to gets filtered out, though I don't see any logs that tell me why it's being filtered out.
It might be related that all of my new instances seems to land on a single host.
An pointers on where to look for logs for Host Filters?
You'll want to set [DEFAULT]debug = True in the nova.conf for your nova-scheduler service and restart or HUP the service. And then the logging per scheduler filter (each one will show the starting and ending number of hosts) will be included at DEBUG log level in the nova-scheduler log.
-melwitt
On Thu, 2021-07-29 at 08:32 +0000, Eugen Block wrote:
Do you see anything in the placement.log? Are there enough resources on that node? If you have it installed you could check the current allocations with the osc-placement command:
openstack resource provider list
and then
openstack resource provider show --allocations <UUID>
if we got to the filters the hosts passed placement resouce check initally but it could fail to claim if there was a concurrent boot or migration. the host is being rejected by the host filter looking at the it looks like you have 2 schduling reuests. in the first request we claim the placment resouces and select host chd0p-oscp1 17 2021-07-28 13:43:32.488 661142 DEBUG nova.scheduler.utils [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Attempting to claim resources in the placement API for instance c44f86ec-8e21-472e-9200-94c21d52904f claim_resources /usr/lib/python3.6/site-packages/nova/scheduler/utils.py:1215 18 2021-07-28 13:43:32.972 661142 DEBUG nova.scheduler.filter_scheduler [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] [instance: c44f86ec-8e21-472e-9200-94c21d52904f] Selected host: (chd0p-oscp1, chd0p-oscp1) ram: 257162MB disk: 6846464MB io_ops: 0 instances: 0 _consume_selected_host /usr/lib/python3.6/site- packages/nova/scheduler/filter_scheduler.py:354 then there is a seond phase where the host filter eliminates the host 37 2021-07-28 13:43:34.641 661143 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter only checking host chd0p-oscp1 and node chd0p-oscp1 38 2021-07-28 13:43:34.641 661143 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter ignoring hosts: chd0p-oscp1 in this second phase chd0p-oscp1is on the ignored list and its the only one that is in the lsit. so there are no host left. i suspect you specified chd0p-oscp1 as the destination of the live migration correct rather then allowing any host as the destination. i would use the request id req-df136ad7-5c7c-4099-97c4-caf9853e9a48 to check the nova comppute log on chd0p-oscp1. i suspect the migration was attempted and failed or the calim of the resouse on line 17 failed so it went to the next host. we can see in the inital query the ingnored hosts was empty. it is only populated in this case if the host was eliminated because the claim faild or the migration failed. 06 2021-07-28 13:43:32.477 661142 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter only checking host chd0p-oscp1 and node chd0p-oscp1 07 2021-07-28 13:43:32.478 661142 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter ignoring hosts: so i suspect Eugen is right that its the placment claim that is failing or the migration failed. if the request id show up in the nova comppute log on chd0p-oscp1 then the allocation claim in placment suceeded if not then we know it was the placment claim that failed. you could confirm this by also enabling debug in the placment api log and use teh request id do corralate the attempt to claim the allocation.
Zitat von DHilsbos@performair.com:
All;
Thank you for reading my previous email, despite its poor Subject.
I have attached a subset of the Nova Scheduler log, with debugging enabled. I have added line numbers to the beginning, in order to facilitate referencing.
At line 18 it appears to select the requested server for the migration, but then at line 38, it rejects it.
Am I missing something here? Is it unable to assign the resources through placement?
Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: melanie witt [mailto:melwittt@gmail.com] Sent: Wednesday, July 28, 2021 1:24 PM To: Dominic Hilsbos; openstack-discuss@lists.openstack.org Subject: Re: [ops][nova]
On Wed, 28 Jul 2021, 20:10 +0000, <DHilsbos@performair.com> wrote:
All;
I'm having a problem, though I'm not certain of the full scope.
The most obvious issue is that I can't live migrate instances. The server I want to send the instance to gets filtered out, though I don't see any logs that tell me why it's being filtered out.
It might be related that all of my new instances seems to land on a single host.
An pointers on where to look for logs for Host Filters?
You'll want to set [DEFAULT]debug = True in the nova.conf for your nova-scheduler service and restart or HUP the service. And then the logging per scheduler filter (each one will show the starting and ending number of hosts) will be included at DEBUG log level in the nova-scheduler log.
-melwitt
Yes, I specified the host, as I'm trying to test live-migration to a new host, running a new operating system (Ubuntu 20.04 vs CentOS 8.4). I'm seeing the following in my logs during startup: No provider configs found in /etc/nova/provider_config/ Could this have anything to do with this problem? I haven't found a placement.log anywhere yet. Where would I expect to find that? Alongside nova-api, or alongside nova-compute? Thank you, Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Sean Mooney [mailto:smooney@redhat.com] Sent: Thursday, July 29, 2021 5:54 AM To: Eugen Block; openstack-discuss@lists.openstack.org Subject: Re: [ops][nova][victoria] Unable to Migrate: Host filter ignoring hosts On Thu, 2021-07-29 at 08:32 +0000, Eugen Block wrote:
Do you see anything in the placement.log? Are there enough resources on that node? If you have it installed you could check the current allocations with the osc-placement command:
openstack resource provider list
and then
openstack resource provider show --allocations <UUID>
if we got to the filters the hosts passed placement resouce check initally but it could fail to claim if there was a concurrent boot or migration. the host is being rejected by the host filter looking at the it looks like you have 2 schduling reuests. in the first request we claim the placment resouces and select host chd0p-oscp1 17 2021-07-28 13:43:32.488 661142 DEBUG nova.scheduler.utils [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Attempting to claim resources in the placement API for instance c44f86ec-8e21-472e-9200-94c21d52904f claim_resources /usr/lib/python3.6/site-packages/nova/scheduler/utils.py:1215 18 2021-07-28 13:43:32.972 661142 DEBUG nova.scheduler.filter_scheduler [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] [instance: c44f86ec-8e21-472e-9200-94c21d52904f] Selected host: (chd0p-oscp1, chd0p-oscp1) ram: 257162MB disk: 6846464MB io_ops: 0 instances: 0 _consume_selected_host /usr/lib/python3.6/site- packages/nova/scheduler/filter_scheduler.py:354 then there is a seond phase where the host filter eliminates the host 37 2021-07-28 13:43:34.641 661143 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter only checking host chd0p-oscp1 and node chd0p-oscp1 38 2021-07-28 13:43:34.641 661143 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter ignoring hosts: chd0p-oscp1 in this second phase chd0p-oscp1is on the ignored list and its the only one that is in the lsit. so there are no host left. i suspect you specified chd0p-oscp1 as the destination of the live migration correct rather then allowing any host as the destination. i would use the request id req-df136ad7-5c7c-4099-97c4-caf9853e9a48 to check the nova comppute log on chd0p-oscp1. i suspect the migration was attempted and failed or the calim of the resouse on line 17 failed so it went to the next host. we can see in the inital query the ingnored hosts was empty. it is only populated in this case if the host was eliminated because the claim faild or the migration failed. 06 2021-07-28 13:43:32.477 661142 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter only checking host chd0p-oscp1 and node chd0p-oscp1 07 2021-07-28 13:43:32.478 661142 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter ignoring hosts: so i suspect Eugen is right that its the placment claim that is failing or the migration failed. if the request id show up in the nova comppute log on chd0p-oscp1 then the allocation claim in placment suceeded if not then we know it was the placment claim that failed. you could confirm this by also enabling debug in the placment api log and use teh request id do corralate the attempt to claim the allocation.
Zitat von DHilsbos@performair.com:
All;
Thank you for reading my previous email, despite its poor Subject.
I have attached a subset of the Nova Scheduler log, with debugging enabled. I have added line numbers to the beginning, in order to facilitate referencing.
At line 18 it appears to select the requested server for the migration, but then at line 38, it rejects it.
Am I missing something here? Is it unable to assign the resources through placement?
Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: melanie witt [mailto:melwittt@gmail.com] Sent: Wednesday, July 28, 2021 1:24 PM To: Dominic Hilsbos; openstack-discuss@lists.openstack.org Subject: Re: [ops][nova]
On Wed, 28 Jul 2021, 20:10 +0000, <DHilsbos@performair.com> wrote:
All;
I'm having a problem, though I'm not certain of the full scope.
The most obvious issue is that I can't live migrate instances. The server I want to send the instance to gets filtered out, though I don't see any logs that tell me why it's being filtered out.
It might be related that all of my new instances seems to land on a single host.
An pointers on where to look for logs for Host Filters?
You'll want to set [DEFAULT]debug = True in the nova.conf for your nova-scheduler service and restart or HUP the service. And then the logging per scheduler filter (each one will show the starting and ending number of hosts) will be included at DEBUG log level in the nova-scheduler log.
-melwitt
Placement is a also a service, check out /var/log/placement. Zitat von DHilsbos@performair.com:
Yes, I specified the host, as I'm trying to test live-migration to a new host, running a new operating system (Ubuntu 20.04 vs CentOS 8.4).
I'm seeing the following in my logs during startup: No provider configs found in /etc/nova/provider_config/ Could this have anything to do with this problem?
I haven't found a placement.log anywhere yet. Where would I expect to find that? Alongside nova-api, or alongside nova-compute?
Thank you,
Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: Sean Mooney [mailto:smooney@redhat.com] Sent: Thursday, July 29, 2021 5:54 AM To: Eugen Block; openstack-discuss@lists.openstack.org Subject: Re: [ops][nova][victoria] Unable to Migrate: Host filter ignoring hosts
On Thu, 2021-07-29 at 08:32 +0000, Eugen Block wrote:
Do you see anything in the placement.log? Are there enough resources on that node? If you have it installed you could check the current allocations with the osc-placement command:
openstack resource provider list
and then
openstack resource provider show --allocations <UUID>
if we got to the filters the hosts passed placement resouce check initally but it could fail to claim if there was a concurrent boot or migration. the host is being rejected by the host filter
looking at the it looks like you have 2 schduling reuests. in the first request we claim the placment resouces and select host chd0p-oscp1
17 2021-07-28 13:43:32.488 661142 DEBUG nova.scheduler.utils [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Attempting to claim resources in the placement API for instance c44f86ec-8e21-472e-9200-94c21d52904f claim_resources /usr/lib/python3.6/site-packages/nova/scheduler/utils.py:1215 18 2021-07-28 13:43:32.972 661142 DEBUG nova.scheduler.filter_scheduler [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] [instance: c44f86ec-8e21-472e-9200-94c21d52904f] Selected host: (chd0p-oscp1, chd0p-oscp1) ram: 257162MB disk: 6846464MB io_ops: 0 instances: 0 _consume_selected_host /usr/lib/python3.6/site- packages/nova/scheduler/filter_scheduler.py:354
then there is a seond phase where the host filter eliminates the host
37 2021-07-28 13:43:34.641 661143 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter only checking host chd0p-oscp1 and node chd0p-oscp1 38 2021-07-28 13:43:34.641 661143 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter ignoring hosts: chd0p-oscp1
in this second phase chd0p-oscp1is on the ignored list and its the only one that is in the lsit. so there are no host left.
i suspect you specified chd0p-oscp1 as the destination of the live migration correct rather then allowing any host as the destination.
i would use the request id req-df136ad7-5c7c-4099-97c4-caf9853e9a48 to check the nova comppute log on chd0p-oscp1. i suspect the migration was attempted and failed or the calim of the resouse on line 17 failed so it went to the next host.
we can see in the inital query the ingnored hosts was empty. it is only populated in this case if the host was eliminated because the claim faild or the migration failed.
06 2021-07-28 13:43:32.477 661142 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter only checking host chd0p-oscp1 and node chd0p-oscp1 07 2021-07-28 13:43:32.478 661142 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter ignoring hosts:
so i suspect Eugen is right that its the placment claim that is failing or the migration failed.
if the request id show up in the nova comppute log on chd0p-oscp1 then the allocation claim in placment suceeded if not then we know it was the placment claim that failed.
you could confirm this by also enabling debug in the placment api log and use teh request id do corralate the attempt to claim the allocation.
Zitat von DHilsbos@performair.com:
All;
Thank you for reading my previous email, despite its poor Subject.
I have attached a subset of the Nova Scheduler log, with debugging enabled. I have added line numbers to the beginning, in order to facilitate referencing.
At line 18 it appears to select the requested server for the migration, but then at line 38, it rejects it.
Am I missing something here? Is it unable to assign the resources through placement?
Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: melanie witt [mailto:melwittt@gmail.com] Sent: Wednesday, July 28, 2021 1:24 PM To: Dominic Hilsbos; openstack-discuss@lists.openstack.org Subject: Re: [ops][nova]
On Wed, 28 Jul 2021, 20:10 +0000, <DHilsbos@performair.com> wrote:
All;
I'm having a problem, though I'm not certain of the full scope.
The most obvious issue is that I can't live migrate instances. The server I want to send the instance to gets filtered out, though I don't see any logs that tell me why it's being filtered out.
It might be related that all of my new instances seems to land on a single host.
An pointers on where to look for logs for Host Filters?
You'll want to set [DEFAULT]debug = True in the nova.conf for your nova-scheduler service and restart or HUP the service. And then the logging per scheduler filter (each one will show the starting and ending number of hosts) will be included at DEBUG log level in the nova-scheduler log.
-melwitt
Placement is a also a service, check out /var/log/placement. yes placmenet is its own project/service like nova, neutron, keystone ectra some older deployemnt tools deployed it under nova becuase it was incubated in the nova poject but its should have its own log directory becide nova,neturon and the other openstakc services.
Zitat von DHilsbos@performair.com:
Yes, I specified the host, as I'm trying to test live-migration to a new host, running a new operating system (Ubuntu 20.04 vs CentOS 8.4).
I'm seeing the following in my logs during startup: No provider configs found in /etc/nova/provider_config/ Could this have anything to do with this problem? /etc/nova/provider_config/ is for the provider.yaml feature and is unrealted to your current issue.
On Thu, 2021-07-29 at 16:09 +0000, Eugen Block wrote: this is the spec for reference https://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/provi... but its a way to advertise avaiabel resouce on a plathform tha tare not managed by openstack. for example you cna use it to implement curded power budgets. e.g. on each host adverties an inventory of CUSTOM_WATTS then then you can request CUSTOM_WATTS in the flaovr another example would be for cache aviableity, you can advertise invetoy fo l3 cache and use an external service to enforce the usage fo it on the host.
I haven't found a placement.log anywhere yet. Where would I expect to find that? Alongside nova-api, or alongside nova-compute?
Thank you,
Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: Sean Mooney [mailto:smooney@redhat.com] Sent: Thursday, July 29, 2021 5:54 AM To: Eugen Block; openstack-discuss@lists.openstack.org Subject: Re: [ops][nova][victoria] Unable to Migrate: Host filter ignoring hosts
On Thu, 2021-07-29 at 08:32 +0000, Eugen Block wrote:
Do you see anything in the placement.log? Are there enough resources on that node? If you have it installed you could check the current allocations with the osc-placement command:
openstack resource provider list
and then
openstack resource provider show --allocations <UUID>
if we got to the filters the hosts passed placement resouce check initally but it could fail to claim if there was a concurrent boot or migration. the host is being rejected by the host filter
looking at the it looks like you have 2 schduling reuests. in the first request we claim the placment resouces and select host chd0p-oscp1
17 2021-07-28 13:43:32.488 661142 DEBUG nova.scheduler.utils [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Attempting to claim resources in the placement API for instance c44f86ec-8e21-472e-9200-94c21d52904f claim_resources /usr/lib/python3.6/site-packages/nova/scheduler/utils.py:1215 18 2021-07-28 13:43:32.972 661142 DEBUG nova.scheduler.filter_scheduler [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] [instance: c44f86ec-8e21-472e-9200-94c21d52904f] Selected host: (chd0p-oscp1, chd0p-oscp1) ram: 257162MB disk: 6846464MB io_ops: 0 instances: 0 _consume_selected_host /usr/lib/python3.6/site- packages/nova/scheduler/filter_scheduler.py:354
then there is a seond phase where the host filter eliminates the host
37 2021-07-28 13:43:34.641 661143 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter only checking host chd0p-oscp1 and node chd0p-oscp1 38 2021-07-28 13:43:34.641 661143 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter ignoring hosts: chd0p-oscp1
in this second phase chd0p-oscp1is on the ignored list and its the only one that is in the lsit. so there are no host left.
i suspect you specified chd0p-oscp1 as the destination of the live migration correct rather then allowing any host as the destination.
i would use the request id req-df136ad7-5c7c-4099-97c4-caf9853e9a48 to check the nova comppute log on chd0p-oscp1. i suspect the migration was attempted and failed or the calim of the resouse on line 17 failed so it went to the next host.
we can see in the inital query the ingnored hosts was empty. it is only populated in this case if the host was eliminated because the claim faild or the migration failed.
06 2021-07-28 13:43:32.477 661142 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter only checking host chd0p-oscp1 and node chd0p-oscp1 07 2021-07-28 13:43:32.478 661142 INFO nova.scheduler.host_manager [req-df136ad7-5c7c-4099-97c4-caf9853e9a48 d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 - default default] Host filter ignoring hosts:
so i suspect Eugen is right that its the placment claim that is failing or the migration failed.
if the request id show up in the nova comppute log on chd0p-oscp1 then the allocation claim in placment suceeded if not then we know it was the placment claim that failed.
you could confirm this by also enabling debug in the placment api log and use teh request id do corralate the attempt to claim the allocation.
Zitat von DHilsbos@performair.com:
All;
Thank you for reading my previous email, despite its poor Subject.
I have attached a subset of the Nova Scheduler log, with debugging enabled. I have added line numbers to the beginning, in order to facilitate referencing.
At line 18 it appears to select the requested server for the migration, but then at line 38, it rejects it.
Am I missing something here? Is it unable to assign the resources through placement?
Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: melanie witt [mailto:melwittt@gmail.com] Sent: Wednesday, July 28, 2021 1:24 PM To: Dominic Hilsbos; openstack-discuss@lists.openstack.org Subject: Re: [ops][nova]
On Wed, 28 Jul 2021, 20:10 +0000, <DHilsbos@performair.com> wrote:
All;
I'm having a problem, though I'm not certain of the full scope.
The most obvious issue is that I can't live migrate instances. The server I want to send the instance to gets filtered out, though I don't see any logs that tell me why it's being filtered out.
It might be related that all of my new instances seems to land on a single host.
An pointers on where to look for logs for Host Filters?
You'll want to set [DEFAULT]debug = True in the nova.conf for your nova-scheduler service and restart or HUP the service. And then the logging per scheduler filter (each one will show the starting and ending number of hosts) will be included at DEBUG log level in the nova-scheduler log.
-melwitt
participants (3)
-
DHilsbos＠performair.com
-
Eugen Block
-
Sean Mooney