Open Stack

On Wed, Nov 15, 2017 at 9:59 AM, Song Sam <samsong8610 at gmail.com> wrote:
> Hi, forks:
> 
> I have a openstack o version cluster with two compute nodes. The 
> resources are almost used up.
> So, I add a new compute node to a running cluster and try to cold 
> migrate a vm to the new compute node. At the first time, the 
> migration failed with 'No valid host found' for my mistake. I forgot 
> to mount the data disk to extend local available storage.
> 
> After I extend the storage, I confirmed the hypervisor have 
> recognized the free disk space, as shown in the output of 'openstack 
> hypervisor show'. But when I try to cold migrate again, It failed 
> with 'No valid host found' again. After some dig in the scheduler 
> log, I find the RetryFilter reject the new added host.

It seems to me that you hit this bug: 
https://bugs.launchpad.net/nova/+bug/1718512
> 
> My question are:
> 1. Where does the RetryFilter store the host availability status? In 
> memory or in DB? Which table?

The RequestSpec persists the already tried hosts and RetryFilter checks 
that.

> 2. How to convince the RetryFilter that the host is available again? 
> I tried to restart nova-api, nova-scheduler, memcached services, but 
> it doesn't work.

There is a request_spec table in the nova_api db where the RequestSpec 
objects are stored. Unfortunately the content of the object is json 
serialized into the spec field of that table so I don't suggest to try 
to fix it manually.

cheers,
gibi

> 
> The related log lines as following:
> 
> 2017-11-15 14:24:37.906 7 DEBUG oslo_messaging._drivers.amqpdriver 
> [-] received message msg_id: fa38b97af9504ebaa1d3118d91157f45 reply 
> to reply_2c3ce56f29774ed19c504407625c632c __call__ 
> /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:194
> 2017-11-15 14:24:37.999 7 DEBUG nova.filters 
> [req-25d6619e-c611-4ecd-b696-f5cc8a39e93d 
> a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - - 
> -] Starting with 0 host(s) get_filtered_objects 
> /var/lib/kolla/venv/lib/python2.7/site-packages/nova/filters.py:70
> 2017-11-15 14:24:38.000 7 INFO nova.filters 
> [req-25d6619e-c611-4ecd-b696-f5cc8a39e93d 
> a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - - 
> -] Filter RetryFilter returned 0 hosts
> 2017-11-15 14:24:38.001 7 DEBUG nova.filters 
> [req-25d6619e-c611-4ecd-b696-f5cc8a39e93d 
> a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - - 
> -] Filtering removed all hosts for the request with instance ID 
> '14139777-351e-4fe9-9554-3c5729bc88cf'. Filter results: 
> [('RetryFilter', None)] get_filtered_objects 
> /var/lib/kolla/venv/lib/python2.7/site-packages/nova/filters.py:129
> 2017-11-15 14:24:38.001 7 INFO nova.filters 
> [req-25d6619e-c611-4ecd-b696-f5cc8a39e93d 
> a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - - 
> -] Filtering removed all hosts for the request with instance ID 
> '14139777-351e-4fe9-9554-3c5729bc88cf'. Filter results: 
> ['RetryFilter: (start: 0, end: 0)']
> 2017-11-15 14:24:38.002 7 DEBUG nova.scheduler.filter_scheduler 
> [req-25d6619e-c611-4ecd-b696-f5cc8a39e93d 
> a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - - 
> -] There are 0 hosts available but 1 instances requested to build. 
> select_destinations 
> /var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:76
> 
> 
> Any help are appreciated.
> Thanks
> 
> Sam Song