[Openstack] Pike NOVA Disable and Live Migrate all instances.
Steven D. Searles
SSearles at zimcom.net
Wed Sep 20 02:00:32 UTC 2017
That would be fine if the behavior was to move one at a time and just queue the jobs up but it is failing outright. Thanks for the reply David! I still think I am missing something in my config. Do you have a pike setup to see if you can replicate the behavior by chance?
-Steve
On Sep 19, 2017, at 8:32 PM, David Medberry <openstack at medberry.net<mailto:openstack at medberry.net>> wrote:
Best practice is to never migrate more than one at a time. Maybe that has been encoded. I.e. no more than one off of a compute at a time and no more than one onto a compute at a time.
On Sep 19, 2017 5:37 PM, "Steven D. Searles" <SSearles at zimcom.net<mailto:SSearles at zimcom.net>> wrote:
Hello everyone and thanks in advance. I have Openstack Pike (KVM,FC-SAN/Cinder) installed in our lab for testing before upgrade and am seeing a possible issue with disabling a host and live migrating the instances off via the horizon interface. I can migrate the instances individually via the Openstack client without issue. It looks like I might be missing something relating to concurrent jobs in my nova config? Interestingly enough when a migrate host is attempted via horizon they all fail. Migrating a single instance through the horizon interface does function. Below is what I am seeing in my scheduler log on the controller when trying to live migrate all instances from a disabled host. I believe the last line to be the obvious issue but I can not find a nova variable that seems to relate to this. Can anyone point me in the right direction?
-----------------------------------
2017-09-19 19:02:30.529 19741 INFO nova.scheduler.host_manager [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Host filter ignoring hosts: kvm02
2017-09-19 19:02:30.530 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Starting with 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:70
2017-09-19 19:02:30.530 19741 DEBUG nova.scheduler.filters.retry_filter [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Re-scheduling is disabled host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:34
2017-09-19 19:02:30.530 19741 DEBUG nova.scheduler.filters.retry_filter [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Re-scheduling is disabled host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:34
2017-09-19 19:02:30.531 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter RetryFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104
2017-09-19 19:02:30.531 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter AvailabilityZoneFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104
2017-09-19 19:02:30.532 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter ComputeFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104
2017-09-19 19:02:30.532 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter ComputeCapabilitiesFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104
2017-09-19 19:02:30.532 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter ImagePropertiesFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104
2017-09-19 19:02:30.533 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter ServerGroupAntiAffinityFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104
2017-09-19 19:02:30.533 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter ServerGroupAffinityFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104
2017-09-19 19:02:30.533 19741 DEBUG nova.scheduler.filter_scheduler [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filtered [(kvm01, kvm01.c1.us-east-dtw.os.libertycenterone.com<http://kvm01.c1.us-east-dtw.os.libertycenterone.com>) ram: 257403MB disk: 199680MB io_ops: 0 instances: 0, (kvm03, kvm03.c1.us-east-dtw.os.libertycenterone.com<http://kvm03.c1.us-east-dtw.os.libertycenterone.com>) ram: 257403MB disk: 199680MB io_ops: 0 instances: 0] _get_sorted_hosts /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:298
2017-09-19 19:02:30.534 19741 DEBUG nova.scheduler.filter_scheduler [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Weighed [(kvm01, kvm01.c1.us-east-dtw.os.libertycenterone.com<http://kvm01.c1.us-east-dtw.os.libertycenterone.com>) ram: 257403MB disk: 199680MB io_ops: 0 instances: 0, (kvm03, kvm03.c1.us-east-dtw.os.libertycenterone.com<http://kvm03.c1.us-east-dtw.os.libertycenterone.com>) ram: 257403MB disk: 199680MB io_ops: 0 instances: 0] _get_sorted_hosts /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:308
2017-09-19 19:02:30.534 19741 DEBUG nova.scheduler.filter_scheduler [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Attempting to claim resources in the placement API for instance a16f866b-9d82-44b7-bc66-414f919ba067 _claim_resources /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:270
2017-09-19 19:02:30.547 19741 DEBUG nova.scheduler.client.report [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Doubling-up allocation request for move operation. _move_operation_alloc_request /usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py:162
2017-09-19 19:02:30.547 19741 DEBUG nova.scheduler.client.report [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] New allocation request containing both source and destination hosts in move operation: {'allocations': [{'resource_provider': {'uuid': u'fb1b6ab4-df2a-4b0c-9578-6f7dee019e36'}, 'resources': {u'VCPU': 4, u'MEMORY_MB': 8192, u'DISK_GB': 80}}, {'resource_provider': {'uuid': u'04fe593a-5af9-4878-a635-e9981a7d4dcd'}, 'resources': {u'VCPU': 2, u'MEMORY_MB': 4096, u'DISK_GB': 40}}]} _move_operation_alloc_request /usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py:202
2017-09-19 19:02:30.583 19741 DEBUG nova.scheduler.filter_scheduler [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Selected host: (kvm01, kvm01.c1.us-east-dtw.os.libertycenterone.com<http://kvm01.c1.us-east-dtw.os.libertycenterone.com>) ram: 257403MB disk: 199680MB io_ops: 0 instances: 0 _schedule /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:227
2017-09-19 19:02:30.584 19741 DEBUG oslo_concurrency.lockutils [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Lock "(u'kvm01', u'kvm01.c1.us-east-dtw.os.libertycenterone.com<http://kvm01.c1.us-east-dtw.os.libertycenterone.com>')" acquired by "nova.scheduler.host_manager._locked" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:273
2017-09-19 19:02:30.585 19741 DEBUG nova.virt.hardware [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Require both a host and instance NUMA topology to fit instance on host. numa_fit_instance_to_host /usr/lib/python2.7/dist-packages/nova/virt/hardware.py:1467
2017-09-19 19:02:30.588 19741 DEBUG oslo_concurrency.lockutils [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Lock "(u'kvm01', u'kvm01.c1.us-east-dtw.os.libertycenterone.com<http://kvm01.c1.us-east-dtw.os.libertycenterone.com>')" released by "nova.scheduler.host_manager._locked" :: held 0.004s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285
2017-09-19 19:02:30.588 19741 DEBUG nova.scheduler.filter_scheduler [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] There are 1 hosts available but 10 instances requested to build. select_destinations /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:101
root at controller01:/var/log/nova#
---------------------------------------------
Migrating from the openstack console.
root at controller01:~# openstack server list --long |grep Linux-Test
| 1d2866c3-ffc1-43a1-82d7-8bb92f396836 | Linux-Test-10 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.24 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| f5e17cb0-9679-4322-a480-7a9bd21e4d5b | Linux-Test-9 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.19 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 38a84ba2-a410-4903-983b-6d1631ad1f0b | Linux-Test-8 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.17 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 2598ed75-9c35-4871-864d-924215eeb5d7 | Linux-Test-7 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.23 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 04218d15-6bbd-4687-900d-00ad8260c867 | Linux-Test-6 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.11 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 5e28d8ef-a577-4982-8799-adb87ec3359a | Linux-Test-5 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.9 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| daee8c1b-5c69-4600-86d6-b10adbbe6e21 | Linux-Test-4 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.15 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 33c348a7-60a1-4887-8761-e5ae17ea9357 | Linux-Test-3 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.18 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| f7d50f42-8f55-47a5-8701-fe8c47b62bcc | Linux-Test-2 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.7 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| f9242792-80c3-4dcf-bdb9-f820156925f7 | Linux-Test-1 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.14 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
root at controller01:~# openstack server migrate --live kvm03 Linux-Test-1
root at controller01:~# openstack server migrate --live kvm03 Linux-Test-2
root at controller01:~# openstack server migrate --live kvm03 Linux-Test-3
root at controller01:~# openstack server migrate --live kvm03 Linux-Test-4
root at controller01:~# openstack server migrate --live kvm03 Linux-Test-5
root at controller01:~# openstack server migrate --live kvm03 Linux-Test-6
root at controller01:~# openstack server migrate --live kvm03 Linux-Test-7
root at controller01:~# openstack server migrate --live kvm03 Linux-Test-8
root at controller01:~# openstack server migrate --live kvm03 Linux-Test-9
root at controller01:~# openstack server migrate --live kvm03 Linux-Test-10
root at controller01:~# openstack server list --long |grep Linux-Test
| 1d2866c3-ffc1-43a1-82d7-8bb92f396836 | Linux-Test-10 | MIGRATING | migrating | Running | Admin-RFC1918=10.0.0.24 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| f5e17cb0-9679-4322-a480-7a9bd21e4d5b | Linux-Test-9 | MIGRATING | migrating | Running | Admin-RFC1918=10.0.0.19 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 38a84ba2-a410-4903-983b-6d1631ad1f0b | Linux-Test-8 | MIGRATING | migrating | Running | Admin-RFC1918=10.0.0.17 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 2598ed75-9c35-4871-864d-924215eeb5d7 | Linux-Test-7 | MIGRATING | migrating | Running | Admin-RFC1918=10.0.0.23 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 04218d15-6bbd-4687-900d-00ad8260c867 | Linux-Test-6 | MIGRATING | migrating | Running | Admin-RFC1918=10.0.0.11 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 5e28d8ef-a577-4982-8799-adb87ec3359a | Linux-Test-5 | MIGRATING | migrating | Running | Admin-RFC1918=10.0.0.9 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| daee8c1b-5c69-4600-86d6-b10adbbe6e21 | Linux-Test-4 | MIGRATING | migrating | Running | Admin-RFC1918=10.0.0.15 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 33c348a7-60a1-4887-8761-e5ae17ea9357 | Linux-Test-3 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.18 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| f7d50f42-8f55-47a5-8701-fe8c47b62bcc | Linux-Test-2 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.7 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| f9242792-80c3-4dcf-bdb9-f820156925f7 | Linux-Test-1 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.14 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
root at controller01:~#
root at controller01:~# openstack server list --long |grep Linux-Test
| 1d2866c3-ffc1-43a1-82d7-8bb92f396836 | Linux-Test-10 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.24 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| f5e17cb0-9679-4322-a480-7a9bd21e4d5b | Linux-Test-9 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.19 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| 38a84ba2-a410-4903-983b-6d1631ad1f0b | Linux-Test-8 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.17 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| 2598ed75-9c35-4871-864d-924215eeb5d7 | Linux-Test-7 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.23 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| 04218d15-6bbd-4687-900d-00ad8260c867 | Linux-Test-6 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.11 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| 5e28d8ef-a577-4982-8799-adb87ec3359a | Linux-Test-5 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.9 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| daee8c1b-5c69-4600-86d6-b10adbbe6e21 | Linux-Test-4 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.15 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| 33c348a7-60a1-4887-8761-e5ae17ea9357 | Linux-Test-3 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.18 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| f7d50f42-8f55-47a5-8701-fe8c47b62bcc | Linux-Test-2 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.7 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| f9242792-80c3-4dcf-bdb9-f820156925f7 | Linux-Test-1 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.14 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
root at controller01:~#
Thanks!
Steven Searles
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20170920/516135ab/attachment.html>
More information about the Openstack
mailing list