[nova] NUMA live migration - mostly how it's tested
Luyao Zhong
luyao.zhong at intel.com
Fri Mar 1 06:21:07 UTC 2019
Hi all,
There was something wrong with the live migration when using 'dedicated'
cpu_policy in my test. Attached file contains the details.
On 2019/2/28 下午9:28, Sean Mooney wrote:
> On Wed, 2019-02-27 at 21:33 -0500, Artom Lifshitz wrote:
>>
>>
>> On Wed, Feb 27, 2019, 21:27 Matt Riedemann, <mriedemos at gmail.com> wrote:
>>> On 2/27/2019 7:25 PM, Artom Lifshitz wrote:
>>>> What I've been using for testing is this: [3]. It's a series of
>>>> patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch
>>>> of us Nova Red Hatters to automate testing that's outside of Tempest's
>>>> scope.
>>>
>>> And where is that pulling in your nova series of changes and posting
>>> test results (like a 3rd party CI) so anyone can see it? Or do you mean
>>> here are tests, but you need to provide your own environment if you want
>>> to verify the code prior to merging it.
>>
>> Sorry, wasn't clear. It's the latter. The test code exists, and has run against my devstack environment with my
>> patches checked out, but there's no CI or public posting of test results. Getting CI coverage for these NUMA things
>> (like the old Intel one) is a whole other topic.
> on the ci front i resolved the nested vert on the server i bought to set up a personal ci for numa testing.
> that set me back a few weeks in setting up that ci but i hope to run artom whitebox test amoung other in that at some
> point. vexhost also provided nested virt to the gate vms. im going to see if we can actully create a non voting job
> using the ubuntu-bionic-vexxhost nodeset. if ovh or one of the other providers of ci resource renable nested virt
> then we can maybe make that job voting and not need thridparty ci anymor.
>>> Can we really not even have functional tests with the fake libvirt
>>> driver and fake numa resources to ensure the flow doesn't blow up?
>>
>> That's something I have to look into. We have live migration functional tests, and we have NUMA functional tests, but
>> I'm not sure how we can combine the two.
>
> jus as an addtional proof point im am planning to do a bunch of migration and live migration testing in the next 2-4
> weeks.
>
> my current backlog on no particalar order is
> sriov migration
> numa migration
> vtpm migration
> cross-cell migration
> cross-neutron backend migration (ovs<->linuxbridge)
> cross-firwall migraton (iptables<->contrack) (previously tested and worked at end of queens)
>
> narrowong in on the numa migration the current set of testcases i plan to manually verify are as follows:
>
> note assume all flavor will have 256mb of ram and 4 cores unless otherwise stated
>
> basic tests
> pinned guests (hw:cpu_policy=dedicated)
> pinned-isolated guests (hw:cpu_policy=dedicated hw:thread_policy=isolate)
> pinned-prefer guests (hw:cpu_policy=dedicated hw:thread_policy=prefer)
> unpinned-singel-numa guest (hw:numa_nodes=1)
> unpinned-dual-numa guest (hw:numa_nodes=2)
> unpinned-dual-numa-unblanced guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3
> hw:numa_mem.0=64 hw:numa_mem.0=192)
> unpinned-hugepage-implcit numa guest (hw:mem_page_size=large)
> unpinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2)
> pinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2 hw:cpu_policy=dedicated)
> realtime guest (hw:cpu_policy=dedicated hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1)
> emulator-thread-iosolated guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=isolate)
>
> advanced tests (require extra nova.conf changes)
> emulator-thread-shared guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=shared) note cpu_share_set configrued
> unpinned-singel-numa-hetorgious-host guest (hw:numa_nodes=1) note vcpu_pin_set adjusted so that
> host 1 only has cpus on
> numa 1 and host 2 only has cpus on numa node 2.
> supper-optimiesd-guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3
> hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=isolate)
> supper-optimiesd-guest-2 (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192
> hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=share)
>
>
> for each of these test ill provide a test-command file with the command i used to run the tests and reustlts file
> with a summary at the top plus the xmls before and after the migration showing that intially the resouces
> would conflict on migration and then the updated xmls after the migration.
> i will also provide the local.conf for the devstack deployment and some details about the env like distor/qemu/libvirt
> versions.
>
> eventurally i hope all those test cases can be added to the whitebox plugin and verifed in a ci.
> we could also try and valideate them in functional tests.
>
> i have attached the xml for the pinned guest as an example of what to expect but i will be compileing this slowly as i
> go and zip everying up in an email to the list.
> this will take some time to complete and hosestly i had planned to do most of this testing after feature freeze when we
> can focus on testing more.
>
> regards
> sean
>
>
-------------- next part --------------
--------------------------VM cpu pining info------------------------------
# VM server_on_host1 cpu pinning info
<cputune>
<shares>4096</shares>
<vcpupin vcpu='0' cpuset='43'/>
<vcpupin vcpu='1' cpuset='7'/>
<vcpupin vcpu='2' cpuset='16'/>
<vcpupin vcpu='3' cpuset='52'/>
<emulatorpin cpuset='7,16,43,52'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
</numatune>
# VM server_on_host2 cpu pinning info (before migration)
<cputune>
<shares>4096</shares>
<vcpupin vcpu='0' cpuset='43'/>
<vcpupin vcpu='1' cpuset='7'/>
<vcpupin vcpu='2' cpuset='16'/>
<vcpupin vcpu='3' cpuset='52'/>
<emulatorpin cpuset='7,16,43,52'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
</numatune>
# VM server_on_host2 cpu pinning info (after this VM was migrated to host1)
<cputune>
<shares>4096</shares>
<vcpupin vcpu='0' cpuset='43'/>
<vcpupin vcpu='1' cpuset='7'/>
<vcpupin vcpu='2' cpuset='16'/>
<vcpupin vcpu='3' cpuset='52'/>
<emulatorpin cpuset='7,16,43,52'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
</numatune>
--------------------flavor's extra spec setting----------------------
hw:cpu_policy='dedicated'
hw:numa_cpus.0='0-3'
hw:numa_mem.0='512'
hw:numa_nodes='1'
---------------------server_on_host1 domain xml-----------------------
$ sudo virsh dumpxml instance-0000003d
<domain type='kvm' id='36'>
<name>instance-0000003d</name>
<uuid>0f84973c-62de-4eb0-ac3e-ba3f6f8a869b</uuid>
<metadata>
<nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
<nova:package version="18.1.0"/>
<nova:name>server_on_host1</nova:name>
<nova:creationTime>2019-03-01 05:14:59</nova:creationTime>
<nova:flavor name="my.flavor.v2">
<nova:memory>512</nova:memory>
<nova:disk>1</nova:disk>
<nova:swap>0</nova:swap>
<nova:ephemeral>0</nova:ephemeral>
<nova:vcpus>4</nova:vcpus>
</nova:flavor>
<nova:owner>
<nova:user uuid="565fc373b2cb4a058056e777d4e833e6">admin</nova:user>
<nova:project uuid="71b6738db2d149ba8053ca913d3de99f">admin</nova:project>
</nova:owner>
<nova:root type="image" uuid="2822ee87-f612-4f26-9321-44a7301540a8"/>
</nova:instance>
</metadata>
<memory unit='KiB'>524288</memory>
<currentMemory unit='KiB'>524288</currentMemory>
<vcpu placement='static'>4</vcpu>
<cputune>
<shares>4096</shares>
<vcpupin vcpu='0' cpuset='43'/>
<vcpupin vcpu='1' cpuset='7'/>
<vcpupin vcpu='2' cpuset='16'/>
<vcpupin vcpu='3' cpuset='52'/>
<emulatorpin cpuset='7,16,43,52'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
</numatune>
<resource>
<partition>/machine</partition>
</resource>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>OpenStack Foundation</entry>
<entry name='product'>OpenStack Nova</entry>
<entry name='version'>18.1.0</entry>
<entry name='serial'>0f84973c-62de-4eb0-ac3e-ba3f6f8a869b</entry>
<entry name='uuid'>0f84973c-62de-4eb0-ac3e-ba3f6f8a869b</entry>
<entry name='family'>Virtual Machine</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc-i440fx-bionic'>hvm</type>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu>
<topology sockets='2' cores='1' threads='2'/>
<numa>
<cell id='0' cpus='0-3' memory='524288' unit='KiB'/>
</numa>
</cpu>
<clock offset='utc'>
<timer name='pit' tickpolicy='delay'/>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/var/lib/nova/instances/0f84973c-62de-4eb0-ac3e-ba3f6f8a869b/disk'/>
<backingStore type='file' index='1'>
<format type='raw'/>
<source file='/var/lib/nova/instances/_base/d40ac6a29c9af85483c06fe1b8ac167dd325f31f'/>
<backingStore/>
</backingStore>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0' model='piix3-uhci'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<interface type='bridge'>
<mac address='fa:16:3e:39:ab:97'/>
<source bridge='br-int'/>
<virtualport type='openvswitch'>
<parameters interfaceid='5d59772f-660d-4956-82d6-1b6a455ba34d'/>
</virtualport>
<target dev='tap5d59772f-66'/>
<model type='virtio'/>
<mtu size='1450'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/7'/>
<log file='/var/lib/nova/instances/0f84973c-62de-4eb0-ac3e-ba3f6f8a869b/console.log' append='off'/>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/7'>
<source path='/dev/pts/7'/>
<log file='/var/lib/nova/instances/0f84973c-62de-4eb0-ac3e-ba3f6f8a869b/console.log' append='off'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='mouse' bus='ps2'>
<alias name='input0'/>
</input>
<input type='keyboard' bus='ps2'>
<alias name='input1'/>
</input>
<graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1' primary='yes'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<stats period='10'/>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</memballoon>
</devices>
<seclabel type='dynamic' model='apparmor' relabel='yes'>
<label>libvirt-0f84973c-62de-4eb0-ac3e-ba3f6f8a869b</label>
<imagelabel>libvirt-0f84973c-62de-4eb0-ac3e-ba3f6f8a869b</imagelabel>
</seclabel>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+64055:+125</label>
<imagelabel>+64055:+125</imagelabel>
</seclabel>
</domain>
---------------------server_on_host2 domain xml-----------------------
$ sudo virsh dumpxml instance-0000003f
<domain type='kvm' id='38'>
<name>instance-0000003f</name>
<uuid>ef5505e5-746d-4ee5-8941-4715ff3de088</uuid>
<metadata>
<nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
<nova:package version="18.1.0"/>
<nova:name>server_on_host2</nova:name>
<nova:creationTime>2019-03-01 05:43:05</nova:creationTime>
<nova:flavor name="my.flavor.v2">
<nova:memory>512</nova:memory>
<nova:disk>1</nova:disk>
<nova:swap>0</nova:swap>
<nova:ephemeral>0</nova:ephemeral>
<nova:vcpus>4</nova:vcpus>
</nova:flavor>
<nova:owner>
<nova:user uuid="565fc373b2cb4a058056e777d4e833e6">admin</nova:user>
<nova:project uuid="71b6738db2d149ba8053ca913d3de99f">admin</nova:project>
</nova:owner>
</nova:instance>
</metadata>
<memory unit='KiB'>524288</memory>
<currentMemory unit='KiB'>524288</currentMemory>
<vcpu placement='static'>4</vcpu>
<cputune>
<shares>4096</shares>
<vcpupin vcpu='0' cpuset='2'/>
<vcpupin vcpu='1' cpuset='38'/>
<vcpupin vcpu='2' cpuset='8'/>
<vcpupin vcpu='3' cpuset='44'/>
<emulatorpin cpuset='2,8,38,44'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
</numatune>
<resource>
<partition>/machine</partition>
</resource>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>OpenStack Foundation</entry>
<entry name='product'>OpenStack Nova</entry>
<entry name='version'>18.1.0</entry>
<entry name='serial'>ef5505e5-746d-4ee5-8941-4715ff3de088</entry>
<entry name='uuid'>ef5505e5-746d-4ee5-8941-4715ff3de088</entry>
<entry name='family'>Virtual Machine</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc-i440fx-bionic'>hvm</type>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu>
<topology sockets='2' cores='1' threads='2'/>
<numa>
<cell id='0' cpus='0-3' memory='524288' unit='KiB'/>
</numa>
</cpu>
<clock offset='utc'>
<timer name='pit' tickpolicy='delay'/>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/sde'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<serial>c42d2260-46a6-4833-a59a-acfc0da510d5</serial>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0' model='piix3-uhci'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<interface type='bridge'>
<mac address='fa:16:3e:9e:8d:89'/>
<source bridge='br-int'/>
<virtualport type='openvswitch'>
<parameters interfaceid='36afc28c-21ce-4e41-a52f-e5a4bfb90853'/>
</virtualport>
<target dev='tap36afc28c-21'/>
<model type='virtio'/>
<mtu size='1450'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/19'/>
<log file='/var/lib/nova/instances/ef5505e5-746d-4ee5-8941-4715ff3de088/console.log' append='off'/>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/19'>
<source path='/dev/pts/19'/>
<log file='/var/lib/nova/instances/ef5505e5-746d-4ee5-8941-4715ff3de088/console.log' append='off'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='mouse' bus='ps2'>
<alias name='input0'/>
</input>
<input type='keyboard' bus='ps2'>
<alias name='input1'/>
</input>
<graphics type='vnc' port='5901' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1' primary='yes'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<stats period='10'/>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</memballoon>
</devices>
<seclabel type='dynamic' model='apparmor' relabel='yes'>
<label>libvirt-ef5505e5-746d-4ee5-8941-4715ff3de088</label>
<imagelabel>libvirt-ef5505e5-746d-4ee5-8941-4715ff3de088</imagelabel>
</seclabel>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+64055:+125</label>
<imagelabel>+64055:+125</imagelabel>
</seclabel>
</domain>
---------------------------------------debug log----------------------------
$ openstack server migrate server_on_host2 --live jfz1r04h09 --debug # jfz1r04h09 here is 'host1' above
START with options: server migrate server_on_host2 --live jfz1r04h09 --debug
options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', application_credential_id='', application_c
redential_name='', application_credential_secret='***', auth_type='password', auth_url='http://172.16.4.9/identity', cacert=None, cert='', client_id='', client_secret
='***', cloud='', code='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=F
alse, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, interface='public', key='', log
_file=None, openid_scope='', os_baremetal_api_version='1.48', os_beta_command=False, os_compute_api_version='', os_dns_api_version='2', os_identity_api_version='3', o
s_image_api_version='', os_key_manager_api_version='1', os_network_api_version='', os_object_api_version='', os_placement_api_version='1.0', os_project_id=None, os_pr
oject_name=None, os_volume_api_version='3', passcode='', password='***', profile='', project_domain_id='default', project_domain_name='', project_id='', project_name=
'admin', protocol='', redirect_uri='', region_name='RegionOne', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name=
'', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', url='', user_domain_id=
'default', user_domain_name='', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'timing': False, 'auth_url': 'http://172.16.4.9/identity', u'network_api_version': u'2', u'messag
e': u'', u'image_format': u'qcow2', 'networks': [], 'verify': True, 'placement_api_version': '1.0', 'dns_api_version': '2', u'object_store_api_version': u'1', u'statu
s': u'active', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, 'baremetal_api_version': '1.48', 'auth': {'project_name': 'admin', 'user_domain_id
': 'default', 'project_domain_id': 'default'}, 'default_domain': 'default', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, u'interface
': 'public', 'password': '***', 'cacert': None, 'key_manager_api_version': '1', u'baremetal_status_code_retries': '5', 'identity_api_version': '3', 'volume_api_versio
n': '3', 'username': 'admin', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'debug': True, u'disable_vendor_agent': {}}
defaults: {u'auth_type': 'password', u'status': u'active', 'api_timeout': None, 'cacert': None, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key
': None, u'interface': u'public', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'baremetal_status_code_retries': 5, 'verify': True, 'cert
': None, u'secgroup_source': u'neutron', u'object_store_api_version': u'1', u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', 'beta_command': False, u'interface': 'public', 'auth_url': 'http://172.16.4.9/identity', u'network_api_version': u'2', u'message'
: u'', u'image_format': u'qcow2', 'networks': [], 'verify': True, 'placement_api_version': '1.0', 'dns_api_version': '2', u'object_store_api_version': u'1', u'status'
: u'active', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, 'baremetal_api_version': '1.48', 'auth': {'project_name': 'admin', 'user_domain_id':
'default', 'project_domain_id': 'default'}, 'default_domain': 'default', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, 'timing': Fal
se, 'password': '***', 'cacert': None, 'key_manager_api_version': '1', u'baremetal_status_code_retries': '5', 'identity_api_version': '3', 'volume_api_version': '3',
'username': 'admin', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'debug': True, u'disable_vendor_agent': {}}
compute API version 2.1, cmd group openstack.compute.v2
network API version 2, cmd group openstack.network.v2
image API version 2, cmd group openstack.image.v2
volume API version 3, cmd group openstack.volume.v3
identity API version 3, cmd group openstack.identity.v3
object_store API version 1, cmd group openstack.object_store.v1
neutronclient API version 2, cmd group openstack.neutronclient.v2
baremetal API version 1.48, cmd group openstack.baremetal.v1
dns API version 2, cmd group openstack.dns.v2
key_manager API version 1, cmd group openstack.key_manager.v1
placement API version 1.0, cmd group openstack.placement.v1
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'timing': False, 'auth_url': 'http://172.16.4.9/identity', u'network_api_version': u'2', u'messag
e': u'', u'image_format': u'qcow2', 'networks': [], 'verify': True, 'placement_api_version': '1.0', 'dns_api_version': '2', u'object_store_api_version': u'1', u'statu
s': u'active', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, 'baremetal_api_version': '1.48', 'auth': {'project_name': 'admin', 'user_domain_id
': 'default', 'project_domain_id': 'default'}, 'default_domain': 'default', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, u'interface
': 'public', 'password': '***', 'cacert': None, 'key_manager_api_version': '1', u'baremetal_status_code_retries': '5', 'identity_api_version': '3', 'volume_api_versio
n': '3', 'username': 'admin', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'debug': True, u'disable_vendor_agent': {}}
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'timing': False, 'auth_url': 'http://172.16.4.9/identity', u'network_api_version': u'2', u'messag
e': u'', u'image_format': u'qcow2', 'networks': [], 'verify': True, 'placement_api_version': '1.0', 'dns_api_version': '2', u'object_store_api_version': u'1', u'statu
s': u'active', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, 'baremetal_api_version': '1.48', 'auth': {'project_name': 'admin', 'user_domain_id
': 'default', 'project_domain_id': 'default'}, 'default_domain': 'default', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, u'interface
': 'public', 'password': '***', 'cacert': None, 'key_manager_api_version': '1', u'baremetal_status_code_retries': '5', 'identity_api_version': '3', 'volume_api_versio
n': '3', 'username': 'admin', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'debug': True, u'disable_vendor_agent': {}}
command: server migrate -> openstackclient.compute.v2.server.MigrateServer (auth=True)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'cacert': None, 'auth_url': 'http://172.16.4.9/identity', 'additional_user_agent': [('osc-lib', '
1.12.0')], u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], 'verify': True, 'placement_api_version': '1.0', 'dns_api_version'
: '2', u'object_store_api_version': u'1', u'status': u'active', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, 'baremetal_api_version': '1.48',
'auth': {'project_name': 'admin', 'user_domain_id': 'default', 'project_domain_id': 'default'}, 'default_domain': 'default', u'image_api_use_tasks': False, u'floating
_ip_source': u'neutron', 'key': None, u'interface': 'public', 'timing': False, 'password': '***', 'key_manager_api_version': '1', u'baremetal_status_code_retries': '5
', 'identity_api_version': '3', 'volume_api_version': '3', 'username': 'admin', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'debug': True, u
'disable_vendor_agent': {}}
Using auth plugin: password
Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_id': 'default', 'auth_url': 'http://172.16.4.9/identity', 'password': '***', 'project_dom
ain_id': 'default'}
Get auth_ref
REQ: curl -g -i -X GET http://172.16.4.9/identity -H "Accept: application/json" -H "User-Agent: openstacksdk/0.24.0 keystoneauth1/3.11.2 python-requests/2.21.0 CPytho
n/2.7.12"
Starting new HTTP connection (1): 172.16.4.9:80
http://172.16.4.9:80 "GET /identity HTTP/1.1" 300 269
RESP: [300] Connection: close Content-Length: 269 Content-Type: application/json Date: Fri, 01 Mar 2019 05:49:38 GMT Location: http://172.16.4.9/identity/v3/ Server:
Apache/2.4.18 (Ubuntu) Vary: X-Auth-Token x-openstack-request-id: req-6f337232-cffc-4a9d-b1cf-3d68f6912507
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2019-01-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.open
stack.identity-v3+json"}], "id": "v3.12", "links": [{"href": "http://172.16.4.9/identity/v3/", "rel": "self"}]}]}}
GET call to http://172.16.4.9/identity used request id req-6f337232-cffc-4a9d-b1cf-3d68f6912507
Making authentication request to http://172.16.4.9/identity/v3/auth/tokens
Resetting dropped connection: 172.16.4.9
http://172.16.4.9:80 "POST /identity/v3/auth/tokens HTTP/1.1" 201 3219
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "c887ca0273dd4fd6a640189e63479929", "name": "admin"}, {"id": "00291096307f47b49e383e593ec5afb
b", "name": "reader"}, {"id": "7b7da132adf24aae86200a096b6b63d5", "name": "member"}], "expires_at": "2019-03-01T06:49:38.000000Z", "project": {"domain": {"id": "defau
lt", "name": "Default"}, "id": "71b6738db2d149ba8053ca913d3de99f", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://172.16.4.9/volume/v2/71b6738db2d149ba8
053ca913d3de99f", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2d1d7eb5c5b244618f8c1550d7fd27c8"}], "type": "volumev2", "id": "103cc
d29d6db461c85e8b75fbc130795", "name": "cinderv2"}, {"endpoints": [{"url": "http://172.16.4.9/compute/v2.1", "interface": "public", "region": "RegionOne", "region_id":
"RegionOne", "id": "eecb14ca68fb4d798d72cb2f1e38ca72"}], "type": "compute", "id": "454c0774c6d14cc59de539489a57f365", "name": "nova"}, {"endpoints": [{"url": "http:/
/172.16.4.9/image", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2ddb22a2b14c406dbe3972133e40ed03"}], "type": "image", "id": "a5c7e4
8064a24d6f843e55d2144c0707", "name": "glance"}, {"endpoints": [{"url": "http://172.16.4.9/volume/v3/71b6738db2d149ba8053ca913d3de99f", "interface": "public", "region"
: "RegionOne", "region_id": "RegionOne", "id": "6ee1bb828ef0460e9a3c577cfd743bcf"}], "type": "volumev3", "id": "b454ecd89aea4f858da280ca8e33df47", "name": "cinderv3"}
, {"endpoints": [{"url": "http://172.16.4.9/placement", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "4788bf690fd24c0b98b4d9ed617f767
4"}], "type": "placement", "id": "c121205abe7d4fcfac8039026a3e62a2", "name": "placement"}, {"endpoints": [{"url": "http://172.16.4.9/compute/v2/71b6738db2d149ba8053ca
913d3de99f", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "036ebd23810e442393902e7b7d2562b8"}], "type": "compute_legacy", "id": "cc83
66b293c141438d30ccdf8afff314", "name": "nova_legacy"}, {"endpoints": [{"url": "http://172.16.4.9/identity", "interface": "public", "region": "RegionOne", "region_id":
"RegionOne", "id": "10b959060bb54b569ee67629dfc959bd"}, {"url": "http://172.16.4.9/identity", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne",
"id": "71d3213e41494e479835f788b08004de"}], "type": "identity", "id": "d158887b1999468c9d4457e88999d5d9", "name": "keystone"}, {"endpoints": [{"url": "http://172.16.4
.9/volume/v3/71b6738db2d149ba8053ca913d3de99f", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "50309c50b1ec4ef5bfce9d11eeef6025"}], "t
ype": "block-storage", "id": "d1bc7a443bdf4db8a2ed94629ebe777a", "name": "cinder"}, {"endpoints": [{"url": "http://172.16.4.9:9696/", "interface": "public", "region":
"RegionOne", "region_id": "RegionOne", "id": "dc688df80ba24b60a62509aa21b75658"}], "type": "network", "id": "d61b64b8265f4e52a857a69ac64b4a6f", "name": "neutron"}],
"user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "565fc373b2cb4a058056e777d4e833e6"}, "audit_ids": ["KZz3j
REBQvC_joSNTlVEew"], "issued_at": "2019-03-01T05:49:38.000000Z"}}
run(Namespace(block_migration=False, disk_overcommit=False, live=u'jfz1r04h09', server=u'server_on_host2', wait=False))
Instantiating compute client for API Version Major: 2, Minor: 1
Instantiating compute api: <class 'openstackclient.api.compute_v2.APIv2'>
Making authentication request to http://172.16.4.9/identity/v3/auth/tokens
Resetting dropped connection: 172.16.4.9
http://172.16.4.9:80 "POST /identity/v3/auth/tokens HTTP/1.1" 201 3219
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "c887ca0273dd4fd6a640189e63479929", "name": "admin"}, {"id": "00291096307f47b49e383e593ec5afb
b", "name": "reader"}, {"id": "7b7da132adf24aae86200a096b6b63d5", "name": "member"}], "expires_at": "2019-03-01T06:49:38.000000Z", "project": {"domain": {"id": "defau
lt", "name": "Default"}, "id": "71b6738db2d149ba8053ca913d3de99f", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://172.16.4.9/volume/v2/71b6738db2d149ba8
053ca913d3de99f", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2d1d7eb5c5b244618f8c1550d7fd27c8"}], "type": "volumev2", "id": "103cc
d29d6db461c85e8b75fbc130795", "name": "cinderv2"}, {"endpoints": [{"url": "http://172.16.4.9/compute/v2.1", "interface": "public", "region": "RegionOne", "region_id":
"RegionOne", "id": "eecb14ca68fb4d798d72cb2f1e38ca72"}], "type": "compute", "id": "454c0774c6d14cc59de539489a57f365", "name": "nova"}, {"endpoints": [{"url": "http:/
/172.16.4.9/image", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2ddb22a2b14c406dbe3972133e40ed03"}], "type": "image", "id": "a5c7e4
8064a24d6f843e55d2144c0707", "name": "glance"}, {"endpoints": [{"url": "http://172.16.4.9/volume/v3/71b6738db2d149ba8053ca913d3de99f", "interface": "public", "region"
: "RegionOne", "region_id": "RegionOne", "id": "6ee1bb828ef0460e9a3c577cfd743bcf"}], "type": "volumev3", "id": "b454ecd89aea4f858da280ca8e33df47", "name": "cinderv3"}
, {"endpoints": [{"url": "http://172.16.4.9/placement", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "4788bf690fd24c0b98b4d9ed617f767
4"}], "type": "placement", "id": "c121205abe7d4fcfac8039026a3e62a2", "name": "placement"}, {"endpoints": [{"url": "http://172.16.4.9/compute/v2/71b6738db2d149ba8053ca
913d3de99f", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "036ebd23810e442393902e7b7d2562b8"}], "type": "compute_legacy", "id": "cc83
66b293c141438d30ccdf8afff314", "name": "nova_legacy"}, {"endpoints": [{"url": "http://172.16.4.9/identity", "interface": "public", "region": "RegionOne", "region_id":
"RegionOne", "id": "10b959060bb54b569ee67629dfc959bd"}, {"url": "http://172.16.4.9/identity", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne",
"id": "71d3213e41494e479835f788b08004de"}], "type": "identity", "id": "d158887b1999468c9d4457e88999d5d9", "name": "keystone"}, {"endpoints": [{"url": "http://172.16.4
.9/volume/v3/71b6738db2d149ba8053ca913d3de99f", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "50309c50b1ec4ef5bfce9d11eeef6025"}], "t
ype": "block-storage", "id": "d1bc7a443bdf4db8a2ed94629ebe777a", "name": "cinder"}, {"endpoints": [{"url": "http://172.16.4.9:9696/", "interface": "public", "region":
"RegionOne", "region_id": "RegionOne", "id": "dc688df80ba24b60a62509aa21b75658"}], "type": "network", "id": "d61b64b8265f4e52a857a69ac64b4a6f", "name": "neutron"}],
"user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "565fc373b2cb4a058056e777d4e833e6"}, "audit_ids": ["AyH7w
-i0T328oNoh-hVgrQ"], "issued_at": "2019-03-01T05:49:38.000000Z"}}
REQ: curl -g -i -X GET http://172.16.4.9/compute/v2.1/servers/server_on_host2 -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA2
56}edb2bf1028170e0f0e300657de4f038cf91ac856780e6415fcad7d9829fd74f9" -H "X-OpenStack-Nova-API-Version: 2.1"
Resetting dropped connection: 172.16.4.9
http://172.16.4.9:80 "GET /compute/v2.1/servers/server_on_host2 HTTP/1.1" 404 90
RESP: [404] Connection: close Content-Length: 90 Content-Type: application/json; charset=UTF-8 Date: Fri, 01 Mar 2019 05:49:38 GMT OpenStack-API-Version: compute 2.1
Server: Apache/2.4.18 (Ubuntu) Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version X-OpenStack-Nova-API-Version: 2.1 x-compute-request-id: req-53610542-9082-4cb4
-a472-a44e9a53ed89 x-openstack-request-id: req-53610542-9082-4cb4-a472-a44e9a53ed89
RESP BODY: {"itemNotFound": {"message": "Instance server_on_host2 could not be found.", "code": 404}}
GET call to compute for http://172.16.4.9/compute/v2.1/servers/server_on_host2 used request id req-53610542-9082-4cb4-a472-a44e9a53ed89
REQ: curl -g -i -X GET http://172.16.4.9/compute/v2.1/servers?name=server_on_host2 -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token:
{SHA256}edb2bf1028170e0f0e300657de4f038cf91ac856780e6415fcad7d9829fd74f9" -H "X-OpenStack-Nova-API-Version: 2.1"
Resetting dropped connection: 172.16.4.9
http://172.16.4.9:80 "GET /compute/v2.1/servers/ef5505e5-746d-4ee5-8941-4715ff3de088 HTTP/1.1" 200 1450
RESP: [200] Connection: close Content-Length: 1450 Content-Type: application/json Date: Fri, 01 Mar 2019 05:49:38 GMT OpenStack-API-Version: compute 2.1 Server: Apach
e/2.4.18 (Ubuntu) Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version X-OpenStack-Nova-API-Version: 2.1 x-compute-request-id: req-2a581c51-a886-4c66-b68a-f335638
866d3 x-openstack-request-id: req-2a581c51-a886-4c66-b68a-f335638866d3
RESP BODY: {"server": {"OS-EXT-STS:task_state": null, "addresses": {"private": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:9e:8d:89", "version": 4, "addr": "10.0.0.50", "O
S-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://172.16.4.9/compute/v2.1/servers/ef5505e5-746d-4ee5-8941-4715ff3de088", "rel": "self"}, {"href": "http://172.16
.4.9/compute/servers/ef5505e5-746d-4ee5-8941-4715ff3de088", "rel": "bookmark"}], "image": "", "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "insta
nce-0000003f", "OS-SRV-USG:launched_at": "2019-03-01T05:43:10.000000", "flavor": {"id": "f", "links": [{"href": "http://172.16.4.9/compute/flavors/f", "rel": "bookmar
k"}]}, "id": "ef5505e5-746d-4ee5-8941-4715ff3de088", "security_groups": [{"name": "default"}], "user_id": "565fc373b2cb4a058056e777d4e833e6", "OS-DCF:diskConfig": "MA
NUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "status": "ACTIVE",
"updated": "2019-03-01T05:43:11Z", "hostId": "1eb20aeff7eb9cbe94a3f7dcc719b0b2c5f258bab6ca77d9cea9ea7f", "OS-EXT-SRV-ATTR:host": "jfz1r03h15", "OS-SRV-USG:terminated_
at": null, "key_name": null, "OS-EXT-SRV-ATTR:hypervisor_hostname": "jfz1r03h15", "name": "server_on_host2", "created": "2019-03-01T05:43:01Z", "tenant_id": "71b6738d
b2d149ba8053ca913d3de99f", "os-extended-volumes:volumes_attached": [{"id": "c42d2260-46a6-4833-a59a-acfc0da510d5"}], "metadata": {}}}
GET call to compute for http://172.16.4.9/compute/v2.1/servers/ef5505e5-746d-4ee5-8941-4715ff3de088 used request id req-2a581c51-a886-4c66-b68a-f335638866d3
REQ: curl -g -i -X POST http://172.16.4.9/compute/v2.1/servers/ef5505e5-746d-4ee5-8941-4715ff3de088/action -H "Accept: application/json" -H "Content-Type: application
/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA256}edb2bf1028170e0f0e300657de4f038cf91ac856780e6415fcad7d9829fd74f9" -H "X-OpenStack-Nova-API-Version
: 2.1" -d '{"os-migrateLive": {"disk_over_commit": false, "block_migration": false, "host": "jfz1r04h09"}}'
Resetting dropped connection: 172.16.4.9
http://172.16.4.9:80 "POST /compute/v2.1/servers/ef5505e5-746d-4ee5-8941-4715ff3de088/action HTTP/1.1" 202 0
RESP: [202] Connection: close Content-Length: 0 Content-Type: application/json Date: Fri, 01 Mar 2019 05:49:39 GMT OpenStack-API-Version: compute 2.1 Server: Apache/2
.4.18 (Ubuntu) Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version X-OpenStack-Nova-API-Version: 2.1 x-compute-request-id: req-04802da2-f1f7-44a5-a117-65e1694559
30 x-openstack-request-id: req-04802da2-f1f7-44a5-a117-65e169455930
POST call to compute for http://172.16.4.9/compute/v2.1/servers/ef5505e5-746d-4ee5-8941-4715ff3de088/action used request id req-04802da2-f1f7-44a5-a117-65e169455930
clean_up MigrateServer:
END return value: 0
More information about the openstack-discuss
mailing list