Open Stack

Hi, forks:

I have a openstack o version cluster with two compute nodes. The resources
are almost used up.
So, I add a new compute node to a running cluster and try to cold migrate a
vm to the new compute node. At the first time, the migration failed with
'No valid host found' for my mistake. I forgot to mount the data disk to
extend local available storage.

After I extend the storage, I confirmed the hypervisor have recognized the
free disk space, as shown in the output of 'openstack hypervisor show'. But
when I try to cold migrate again, It failed with 'No valid host found'
again. After some dig in the scheduler log, I find the RetryFilter reject
the new added host.

My question are:
1. Where does the RetryFilter store the host availability status? In memory
or in DB? Which table?
2. How to convince the RetryFilter that the host is available again? I
tried to restart nova-api, nova-scheduler, memcached services, but it
doesn't work.

The related log lines as following:

2017-11-15 14:24:37.906 7 DEBUG oslo_messaging._drivers.amqpdriver [-]
received message msg_id: fa38b97af9504ebaa1d3118d91157f45 reply to
reply_2c3ce56f29774ed19c504407625c632c __call__
/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:194
2017-11-15 14:24:37.999 7 DEBUG nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d a5efca4a04654a829ee19229df707e08
2e13a26ead3c4da493c5c234233308df - - -] Starting with 0 host(s)
get_filtered_objects
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/filters.py:70
2017-11-15 14:24:38.000 7 INFO nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d a5efca4a04654a829ee19229df707e08
2e13a26ead3c4da493c5c234233308df - - -] Filter RetryFilter returned 0 hosts
2017-11-15 14:24:38.001 7 DEBUG nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d a5efca4a04654a829ee19229df707e08
2e13a26ead3c4da493c5c234233308df - - -] Filtering removed all hosts for the
request with instance ID '14139777-351e-4fe9-9554-3c5729bc88cf'. Filter
results: [('RetryFilter', None)] get_filtered_objects
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/filters.py:129
2017-11-15 14:24:38.001 7 INFO nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d a5efca4a04654a829ee19229df707e08
2e13a26ead3c4da493c5c234233308df - - -] Filtering removed all hosts for the
request with instance ID '14139777-351e-4fe9-9554-3c5729bc88cf'. Filter
results: ['RetryFilter: (start: 0, end: 0)']
2017-11-15 14:24:38.002 7 DEBUG nova.scheduler.filter_scheduler
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d a5efca4a04654a829ee19229df707e08
2e13a26ead3c4da493c5c234233308df - - -] There are 0 hosts available but 1
instances requested to build. select_destinations
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:76


Any help are appreciated.
Thanks

Sam Song
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171115/ae304b6d/attachment.html>