[ops][nova][victoria] Unable to Migrate: Host filter ignoring hosts

Sean Mooney smooney at redhat.com
Thu Jul 29 18:30:31 UTC 2021


On Thu, 2021-07-29 at 16:09 +0000, Eugen Block wrote:
> Placement is a also a service, check out /var/log/placement.
yes placmenet is its own project/service like nova, neutron, keystone ectra
some older deployemnt tools deployed it under nova becuase it was incubated in the nova poject but its should have its own log directory becide nova,neturon and the other openstakc services.
> 
> Zitat von DHilsbos at performair.com:
> 
> > Yes, I specified the host, as I'm trying to test live-migration to a  
> > new host, running a new operating system (Ubuntu 20.04 vs CentOS 8.4).
> > 
> > I'm seeing the following in my logs during startup:
> > No provider configs found in /etc/nova/provider_config/
> > Could this have anything to do with this problem?
/etc/nova/provider_config/ is for the provider.yaml feature and is unrealted to your current issue.
this is the spec for reference https://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/provider-config-file.html
but its a way to advertise avaiabel resouce on a plathform tha tare not managed by openstack.

for example you cna use it to implement curded power budgets.
e.g. on each host adverties an inventory of CUSTOM_WATTS then then you can request CUSTOM_WATTS in the flaovr 
another example would be for cache aviableity, you can advertise invetoy fo l3 cache and use an external service to enforce the usage fo it on
the host.

 


> > 
> > I haven't found a placement.log anywhere yet.  Where would I expect  
> > to find that?  Alongside nova-api, or alongside nova-compute?
> > 
> > Thank you,
> > 
> > Dominic L. Hilsbos, MBA
> > Vice President – Information Technology
> > Perform Air International Inc.
> > DHilsbos at PerformAir.com
> > www.PerformAir.com
> > 
> > -----Original Message-----
> > From: Sean Mooney [mailto:smooney at redhat.com]
> > Sent: Thursday, July 29, 2021 5:54 AM
> > To: Eugen Block; openstack-discuss at lists.openstack.org
> > Subject: Re: [ops][nova][victoria] Unable to Migrate: Host filter  
> > ignoring hosts
> > 
> > On Thu, 2021-07-29 at 08:32 +0000, Eugen Block wrote:
> > > Do you see anything in the placement.log? Are there enough resources
> > > on that node? If you have it installed you could check the current
> > > allocations with the osc-placement command:
> > > 
> > > openstack resource provider list
> > > 
> > > and then
> > > 
> > > openstack resource provider show --allocations <UUID>
> > 
> > if we got to the filters the hosts passed placement resouce check  
> > initally but it could fail to claim if there was a concurrent boot  
> > or migration.
> > the host is being rejected by the host filter
> > 
> > looking at the it looks like you have 2 schduling reuests.
> > in the first request we claim the placment resouces and select host  
> > chd0p-oscp1
> > 
> > 17 2021-07-28 13:43:32.488 661142 DEBUG nova.scheduler.utils  
> > [req-df136ad7-5c7c-4099-97c4-caf9853e9a48  
> > d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 -  
> > default default] Attempting
> > to claim resources in the placement API for instance  
> > c44f86ec-8e21-472e-9200-94c21d52904f claim_resources  
> > /usr/lib/python3.6/site-packages/nova/scheduler/utils.py:1215
> > 18 2021-07-28 13:43:32.972 661142 DEBUG  
> > nova.scheduler.filter_scheduler  
> > [req-df136ad7-5c7c-4099-97c4-caf9853e9a48  
> > d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 -  
> > default default]
> > [instance: c44f86ec-8e21-472e-9200-94c21d52904f] Selected host:  
> > (chd0p-oscp1, chd0p-oscp1) ram: 257162MB disk: 6846464MB io_ops: 0  
> > instances: 0 _consume_selected_host /usr/lib/python3.6/site-
> > packages/nova/scheduler/filter_scheduler.py:354
> > 
> > then there is a seond phase where the host filter eliminates the host
> > 
> > 37 2021-07-28 13:43:34.641 661143 INFO nova.scheduler.host_manager  
> > [req-df136ad7-5c7c-4099-97c4-caf9853e9a48  
> > d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 -  
> > default default] Host
> > filter only checking host chd0p-oscp1 and node chd0p-oscp1
> > 38 2021-07-28 13:43:34.641 661143 INFO nova.scheduler.host_manager  
> > [req-df136ad7-5c7c-4099-97c4-caf9853e9a48  
> > d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 -  
> > default default] Host
> > filter ignoring hosts: chd0p-oscp1
> > 
> > in this second phase chd0p-oscp1is on the ignored list and its the  
> > only one that is in the lsit. so there are no host left.
> > 
> > i suspect you specified chd0p-oscp1 as the destination of the live  
> > migration correct rather then allowing any host as the destination.
> > 
> > i would use the request id req-df136ad7-5c7c-4099-97c4-caf9853e9a48  
> > to check the nova comppute log on chd0p-oscp1.
> > i suspect the migration was attempted and failed or the calim of the  
> > resouse on line 17 failed so it went to the next host.
> > 
> > 
> > we can see in the inital query the ingnored hosts was empty. it is  
> > only populated in this case if the host was eliminated because the  
> > claim faild or the migration failed.
> > 
> > 06 2021-07-28 13:43:32.477 661142 INFO nova.scheduler.host_manager  
> > [req-df136ad7-5c7c-4099-97c4-caf9853e9a48  
> > d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 -  
> > default default] Host
> > filter only checking host chd0p-oscp1 and node chd0p-oscp1
> > 07 2021-07-28 13:43:32.478 661142 INFO nova.scheduler.host_manager  
> > [req-df136ad7-5c7c-4099-97c4-caf9853e9a48  
> > d7c514813e5d4fe6815f5f59e8e35f2f a008ad02d16f436a9e320882ca497055 -  
> > default default] Host
> > filter ignoring hosts:
> > 
> > so i suspect Eugen is right that its the placment claim that is  
> > failing or the migration failed.
> > 
> > if the request id show up in the nova comppute log on chd0p-oscp1  
> > then the allocation claim in placment suceeded if not then we know  
> > it was the placment claim that failed.
> > 
> > you could confirm this by also enabling debug in the placment api  
> > log and use teh request id do corralate the attempt to claim the  
> > allocation.
> > 
> > > 
> > > 
> > > Zitat von DHilsbos at performair.com:
> > > 
> > > > All;
> > > > 
> > > > Thank you for reading my previous email, despite its poor Subject.
> > > > 
> > > > I have attached a subset of the Nova Scheduler log, with debugging
> > > > enabled.  I have added line numbers to the beginning, in order to
> > > > facilitate referencing.
> > > > 
> > > > At line 18 it appears to select the requested server for the
> > > > migration, but then at line 38, it rejects it.
> > > > 
> > > > Am I missing something here?  Is it unable to assign the resources
> > > > through placement?
> > > > 
> > > > Dominic L. Hilsbos, MBA
> > > > Vice President – Information Technology
> > > > Perform Air International Inc.
> > > > DHilsbos at PerformAir.com
> > > > www.PerformAir.com
> > > > 
> > > > 
> > > > -----Original Message-----
> > > > From: melanie witt [mailto:melwittt at gmail.com]
> > > > Sent: Wednesday, July 28, 2021 1:24 PM
> > > > To: Dominic Hilsbos; openstack-discuss at lists.openstack.org
> > > > Subject: Re: [ops][nova]
> > > > 
> > > > On Wed, 28 Jul 2021, 20:10 +0000, <DHilsbos at performair.com> wrote:
> > > > > All;
> > > > > 
> > > > > I'm having a problem, though I'm not certain of the full scope.
> > > > > 
> > > > > The most obvious issue is that I can't live migrate instances.  The
> > > > > server I want to send the instance to gets filtered out, though I
> > > > > don't see any logs that tell me why it's being filtered out.
> > > > > 
> > > > > It might be related that all of my new instances seems to land on a
> > > > > single host.
> > > > > 
> > > > > An pointers on where to look for logs for Host Filters?
> > > > 
> > > > You'll want to set [DEFAULT]debug = True in the nova.conf for your
> > > > nova-scheduler service and restart or HUP the service. And then the
> > > > logging per scheduler filter (each one will show the starting and ending
> > > > number of hosts) will be included at DEBUG log level in the
> > > > nova-scheduler log.
> > > > 
> > > > -melwitt
> > > 
> > > 
> > > 
> > > 
> 
> 
> 





More information about the openstack-discuss mailing list