[Openstack] Pike NOVA Disable and Live Migrate all instances.
David Medberry
openstack at medberry.net
Wed Sep 20 00:32:42 UTC 2017
Best practice is to never migrate more than one at a time. Maybe that has
been encoded. I.e. no more than one off of a compute at a time and no more
than one onto a compute at a time.
On Sep 19, 2017 5:37 PM, "Steven D. Searles" <SSearles at zimcom.net> wrote:
> Hello everyone and thanks in advance. I have Openstack Pike
> (KVM,FC-SAN/Cinder) installed in our lab for testing before upgrade and am
> seeing a possible issue with disabling a host and live migrating the
> instances off via the horizon interface. I can migrate the instances
> individually via the Openstack client without issue. It looks like I might
> be missing something relating to concurrent jobs in my nova config?
> Interestingly enough when a migrate host is attempted via horizon they all
> fail. Migrating a single instance through the horizon interface does
> function. Below is what I am seeing in my scheduler log on the controller
> when trying to live migrate all instances from a disabled host. I believe
> the last line to be the obvious issue but I can not find a nova variable
> that seems to relate to this. Can anyone point me in the right direction?
>
> -----------------------------------
> 2017-09-19 19:02:30.529 19741 INFO nova.scheduler.host_manager
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Host filter ignoring
> hosts: kvm02
> 2017-09-19 19:02:30.530 19741 DEBUG nova.filters
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Starting with 2
> host(s) get_filtered_objects /usr/lib/python2.7/dist-
> packages/nova/filters.py:70
> 2017-09-19 19:02:30.530 19741 DEBUG nova.scheduler.filters.retry_filter
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Re-scheduling is
> disabled host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/
> filters/retry_filter.py:34
> 2017-09-19 19:02:30.530 19741 DEBUG nova.scheduler.filters.retry_filter
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Re-scheduling is
> disabled host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/
> filters/retry_filter.py:34
> 2017-09-19 19:02:30.531 19741 DEBUG nova.filters
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter RetryFilter
> returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-
> packages/nova/filters.py:104
> 2017-09-19 19:02:30.531 19741 DEBUG nova.filters
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter
> AvailabilityZoneFilter returned 2 host(s) get_filtered_objects
> /usr/lib/python2.7/dist-packages/nova/filters.py:104
> 2017-09-19 19:02:30.532 19741 DEBUG nova.filters
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter ComputeFilter
> returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-
> packages/nova/filters.py:104
> 2017-09-19 19:02:30.532 19741 DEBUG nova.filters
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter
> ComputeCapabilitiesFilter returned 2 host(s) get_filtered_objects
> /usr/lib/python2.7/dist-packages/nova/filters.py:104
> 2017-09-19 19:02:30.532 19741 DEBUG nova.filters
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter
> ImagePropertiesFilter returned 2 host(s) get_filtered_objects
> /usr/lib/python2.7/dist-packages/nova/filters.py:104
> 2017-09-19 19:02:30.533 19741 DEBUG nova.filters
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter
> ServerGroupAntiAffinityFilter returned 2 host(s) get_filtered_objects
> /usr/lib/python2.7/dist-packages/nova/filters.py:104
> 2017-09-19 19:02:30.533 19741 DEBUG nova.filters
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter
> ServerGroupAffinityFilter returned 2 host(s) get_filtered_objects
> /usr/lib/python2.7/dist-packages/nova/filters.py:104
> 2017-09-19 19:02:30.533 19741 DEBUG nova.scheduler.filter_scheduler
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Filtered [(kvm01,
> kvm01.c1.us-east-dtw.os.libertycenterone.com) ram: 257403MB disk:
> 199680MB io_ops: 0 instances: 0, (kvm03, kvm03.c1.us-east-dtw.os.
> libertycenterone.com) ram: 257403MB disk: 199680MB io_ops: 0 instances:
> 0] _get_sorted_hosts /usr/lib/python2.7/dist-packages/nova/scheduler/
> filter_scheduler.py:298
> 2017-09-19 19:02:30.534 19741 DEBUG nova.scheduler.filter_scheduler
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Weighed [(kvm01,
> kvm01.c1.us-east-dtw.os.libertycenterone.com) ram: 257403MB disk:
> 199680MB io_ops: 0 instances: 0, (kvm03, kvm03.c1.us-east-dtw.os.
> libertycenterone.com) ram: 257403MB disk: 199680MB io_ops: 0 instances:
> 0] _get_sorted_hosts /usr/lib/python2.7/dist-packages/nova/scheduler/
> filter_scheduler.py:308
> 2017-09-19 19:02:30.534 19741 DEBUG nova.scheduler.filter_scheduler
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Attempting to claim
> resources in the placement API for instance a16f866b-9d82-44b7-bc66-414f919ba067
> _claim_resources /usr/lib/python2.7/dist-packages/nova/scheduler/
> filter_scheduler.py:270
> 2017-09-19 19:02:30.547 19741 DEBUG nova.scheduler.client.report
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Doubling-up
> allocation request for move operation. _move_operation_alloc_request
> /usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py:162
> 2017-09-19 19:02:30.547 19741 DEBUG nova.scheduler.client.report
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] New allocation
> request containing both source and destination hosts in move operation:
> {'allocations': [{'resource_provider': {'uuid': u'fb1b6ab4-df2a-4b0c-9578-6f7dee019e36'},
> 'resources': {u'VCPU': 4, u'MEMORY_MB': 8192, u'DISK_GB': 80}},
> {'resource_provider': {'uuid': u'04fe593a-5af9-4878-a635-e9981a7d4dcd'},
> 'resources': {u'VCPU': 2, u'MEMORY_MB': 4096, u'DISK_GB': 40}}]}
> _move_operation_alloc_request /usr/lib/python2.7/dist-
> packages/nova/scheduler/client/report.py:202
> 2017-09-19 19:02:30.583 19741 DEBUG nova.scheduler.filter_scheduler
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Selected host:
> (kvm01, kvm01.c1.us-east-dtw.os.libertycenterone.com) ram: 257403MB disk:
> 199680MB io_ops: 0 instances: 0 _schedule /usr/lib/python2.7/dist-
> packages/nova/scheduler/filter_scheduler.py:227
> 2017-09-19 19:02:30.584 19741 DEBUG oslo_concurrency.lockutils
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Lock "(u'kvm01', u'
> kvm01.c1.us-east-dtw.os.libertycenterone.com')" acquired by
> "nova.scheduler.host_manager._locked" :: waited 0.000s inner
> /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:273
> 2017-09-19 19:02:30.585 19741 DEBUG nova.virt.hardware
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Require both a host
> and instance NUMA topology to fit instance on host.
> numa_fit_instance_to_host /usr/lib/python2.7/dist-
> packages/nova/virt/hardware.py:1467
> 2017-09-19 19:02:30.588 19741 DEBUG oslo_concurrency.lockutils
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] Lock "(u'kvm01', u'
> kvm01.c1.us-east-dtw.os.libertycenterone.com')" released by
> "nova.scheduler.host_manager._locked" :: held 0.004s inner
> /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285
> *2017-09-19 19:02:30.588 19741 DEBUG nova.scheduler.filter_scheduler
> [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
> 723aa12337a44f818b6d1e1a59f16e49 - default default] There are 1 hosts
> available but 10 instances requested to build. select_destinations
> /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:101*
> root at controller01:/var/log/nova#
> ---------------------------------------------
>
> Migrating from the openstack console.
>
> root at controller01:~# openstack server list --long |grep Linux-Test
> | 1d2866c3-ffc1-43a1-82d7-8bb92f396836 | Linux-Test-10 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.24 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm02 | |
> | f5e17cb0-9679-4322-a480-7a9bd21e4d5b | Linux-Test-9 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.19 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm02 | |
> | 38a84ba2-a410-4903-983b-6d1631ad1f0b | Linux-Test-8 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.17 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm02 | |
> | 2598ed75-9c35-4871-864d-924215eeb5d7 | Linux-Test-7 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.23 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm02 | |
> | 04218d15-6bbd-4687-900d-00ad8260c867 | Linux-Test-6 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.11 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm02 | |
> | 5e28d8ef-a577-4982-8799-adb87ec3359a | Linux-Test-5 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.9 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm02 | |
> | daee8c1b-5c69-4600-86d6-b10adbbe6e21 | Linux-Test-4 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.15 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm02 | |
> | 33c348a7-60a1-4887-8761-e5ae17ea9357 | Linux-Test-3 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.18 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm02 | |
> | f7d50f42-8f55-47a5-8701-fe8c47b62bcc | Linux-Test-2 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.7 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm02 | |
> | f9242792-80c3-4dcf-bdb9-f820156925f7 | Linux-Test-1 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.14 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm02 | |
> root at controller01:~# openstack server migrate --live kvm03 Linux-Test-1
> root at controller01:~# openstack server migrate --live kvm03 Linux-Test-2
> root at controller01:~# openstack server migrate --live kvm03 Linux-Test-3
> root at controller01:~# openstack server migrate --live kvm03 Linux-Test-4
> root at controller01:~# openstack server migrate --live kvm03 Linux-Test-5
> root at controller01:~# openstack server migrate --live kvm03 Linux-Test-6
> root at controller01:~# openstack server migrate --live kvm03 Linux-Test-7
> root at controller01:~# openstack server migrate --live kvm03 Linux-Test-8
> root at controller01:~# openstack server migrate --live kvm03 Linux-Test-9
> root at controller01:~# openstack server migrate --live kvm03 Linux-Test-10
> root at controller01:~# openstack server list --long |grep Linux-Test
> | 1d2866c3-ffc1-43a1-82d7-8bb92f396836 | Linux-Test-10 | MIGRATING |
> migrating | Running | Admin-RFC1918=10.0.0.24 | |
> | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova
> | kvm02 | |
> | f5e17cb0-9679-4322-a480-7a9bd21e4d5b | Linux-Test-9 | MIGRATING |
> migrating | Running | Admin-RFC1918=10.0.0.19 | |
> | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova
> | kvm02 | |
> | 38a84ba2-a410-4903-983b-6d1631ad1f0b | Linux-Test-8 | MIGRATING |
> migrating | Running | Admin-RFC1918=10.0.0.17 | |
> | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova
> | kvm02 | |
> | 2598ed75-9c35-4871-864d-924215eeb5d7 | Linux-Test-7 | MIGRATING |
> migrating | Running | Admin-RFC1918=10.0.0.23 | |
> | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova
> | kvm02 | |
> | 04218d15-6bbd-4687-900d-00ad8260c867 | Linux-Test-6 | MIGRATING |
> migrating | Running | Admin-RFC1918=10.0.0.11 | |
> | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova
> | kvm02 | |
> | 5e28d8ef-a577-4982-8799-adb87ec3359a | Linux-Test-5 | MIGRATING |
> migrating | Running | Admin-RFC1918=10.0.0.9 | |
> | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova
> | kvm02 | |
> | daee8c1b-5c69-4600-86d6-b10adbbe6e21 | Linux-Test-4 | MIGRATING |
> migrating | Running | Admin-RFC1918=10.0.0.15 | |
> | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova
> | kvm02 | |
> | 33c348a7-60a1-4887-8761-e5ae17ea9357 | Linux-Test-3 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.18 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm03 | |
> | f7d50f42-8f55-47a5-8701-fe8c47b62bcc | Linux-Test-2 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.7 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm03 | |
> | f9242792-80c3-4dcf-bdb9-f820156925f7 | Linux-Test-1 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.14 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm03 | |
> root at controller01:~#
> root at controller01:~# openstack server list --long |grep Linux-Test
> | 1d2866c3-ffc1-43a1-82d7-8bb92f396836 | Linux-Test-10 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.24 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm03 | |
> | f5e17cb0-9679-4322-a480-7a9bd21e4d5b | Linux-Test-9 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.19 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm03 | |
> | 38a84ba2-a410-4903-983b-6d1631ad1f0b | Linux-Test-8 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.17 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm03 | |
> | 2598ed75-9c35-4871-864d-924215eeb5d7 | Linux-Test-7 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.23 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm03 | |
> | 04218d15-6bbd-4687-900d-00ad8260c867 | Linux-Test-6 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.11 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm03 | |
> | 5e28d8ef-a577-4982-8799-adb87ec3359a | Linux-Test-5 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.9 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm03 | |
> | daee8c1b-5c69-4600-86d6-b10adbbe6e21 | Linux-Test-4 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.15 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm03 | |
> | 33c348a7-60a1-4887-8761-e5ae17ea9357 | Linux-Test-3 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.18 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm03 | |
> | f7d50f42-8f55-47a5-8701-fe8c47b62bcc | Linux-Test-2 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.7 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm03 | |
> | f9242792-80c3-4dcf-bdb9-f820156925f7 | Linux-Test-1 | ACTIVE | None
> | Running | Admin-RFC1918=10.0.0.14 | | |
> m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova |
> kvm03 | |
> root at controller01:~#
>
>
> Thanks!
>
>
> Steven Searles
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20170919/521b6692/attachment.html>
More information about the Openstack
mailing list