Open Stack

Hi all,

There was something wrong with the live migration when using 'dedicated' 
cpu_policy in my test. Attached file contains the details.

The message body will be so big that it will be held if I attach the 
debug info, so I will send another email.

Regards,
Luyao


On 2019/2/28 下午9:28, Sean Mooney wrote:
> On Wed, 2019-02-27 at 21:33 -0500, Artom Lifshitz wrote:
>>
>>
>> On Wed, Feb 27, 2019, 21:27 Matt Riedemann, <mriedemos at gmail.com> wrote:
>>> On 2/27/2019 7:25 PM, Artom Lifshitz wrote:
>>>> What I've been using for testing is this: [3]. It's a series of
>>>> patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch
>>>> of us Nova Red Hatters to automate testing that's outside of Tempest's
>>>> scope.
>>>
>>> And where is that pulling in your nova series of changes and posting
>>> test results (like a 3rd party CI) so anyone can see it? Or do you mean
>>> here are tests, but you need to provide your own environment if you want
>>> to verify the code prior to merging it.
>>
>> Sorry, wasn't clear. It's the latter. The test code exists, and has run against my devstack environment with my
>> patches checked out, but there's no CI or public posting of test results. Getting CI coverage for these NUMA things
>> (like the old Intel one) is a whole other topic.
> on the ci front i resolved the nested vert on the server i bought to set up a personal ci for numa testing.
> that set me back a few weeks in setting up that ci but i hope to run artom whitebox test amoung other in that at some
> point. vexhost also provided nested virt to the gate vms. im going to see if we can actully create a non voting job
> using the ubuntu-bionic-vexxhost nodeset. if ovh or one of the other providers of ci resource renable nested virt
> then we can maybe make that job voting and not need thridparty ci anymor.
>>> Can we really not even have functional tests with the fake libvirt
>>> driver and fake numa resources to ensure the flow doesn't blow up?
>>
>> That's something I have to look into. We have live migration functional tests, and we have NUMA functional tests, but
>> I'm not sure how we can combine the two.
> 
> jus as an addtional proof point im am planning to do a bunch of migration and live migration testing in the next 2-4
> weeks.
> 
> my current backlog on no particalar order is
> sriov migration
> numa migration
> vtpm migration
> cross-cell migration
> cross-neutron backend migration (ovs<->linuxbridge)
> cross-firwall migraton (iptables<->contrack) (previously tested and worked at end of queens)
> 
> narrowong in on the numa migration the current set of testcases i plan to manually verify are as follows:
> 
> note assume all flavor will have 256mb of ram and 4 cores unless otherwise stated
> 
> basic tests
> pinned guests (hw:cpu_policy=dedicated)
> pinned-isolated guests (hw:cpu_policy=dedicated hw:thread_policy=isolate)
> pinned-prefer guests (hw:cpu_policy=dedicated hw:thread_policy=prefer)
> unpinned-singel-numa guest (hw:numa_nodes=1)
> unpinned-dual-numa guest (hw:numa_nodes=2)
> unpinned-dual-numa-unblanced guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3
> hw:numa_mem.0=64 hw:numa_mem.0=192)
> unpinned-hugepage-implcit numa guest (hw:mem_page_size=large)
> unpinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2)
> pinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2 hw:cpu_policy=dedicated)
> realtime guest (hw:cpu_policy=dedicated hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1)
> emulator-thread-iosolated guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=isolate)
> 
> advanced tests (require extra nova.conf changes)
> emulator-thread-shared guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=shared) note cpu_share_set configrued
> unpinned-singel-numa-hetorgious-host guest (hw:numa_nodes=1) note vcpu_pin_set adjusted so that
> host 1 only has cpus on
> numa 1 and host 2 only has cpus on numa node 2.
> supper-optimiesd-guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3
> hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=isolate)
> supper-optimiesd-guest-2 (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192
> hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=share)
> 
> 
> for each of these test ill provide a test-command file with the command i used to run the tests and reustlts file
> with a summary at the top plus the xmls before and after the migration showing that intially the resouces
> would conflict on migration and then the updated xmls after the migration.
> i will also provide the local.conf for the devstack deployment and some details about the env like distor/qemu/libvirt
> versions.
> 
> eventurally i hope all those test cases can be added to the whitebox plugin and verifed in a ci.
> we could also try and valideate them in functional tests.
> 
> i have attached the xml for the pinned guest as an example of what to expect but i will be compileing this slowly as i
> go and zip everying up in an email to the list.
> this will take some time to complete and hosestly i had planned to do most of this testing after feature freeze when we
> can focus on testing more.
> 
> regards
> sean
> 
> 
-------------- next part --------------
--------------------------VM cpu pining info------------------------------
# VM server_on_host1 cpu pinning info
  <cputune>
    <shares>4096</shares>
    <vcpupin vcpu='0' cpuset='43'/>
    <vcpupin vcpu='1' cpuset='7'/>
    <vcpupin vcpu='2' cpuset='16'/>
    <vcpupin vcpu='3' cpuset='52'/>
    <emulatorpin cpuset='7,16,43,52'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>


# VM server_on_host2 cpu pinning info (before migration)
  <cputune>
    <shares>4096</shares>
    <vcpupin vcpu='0' cpuset='43'/>
    <vcpupin vcpu='1' cpuset='7'/>
    <vcpupin vcpu='2' cpuset='16'/>
    <vcpupin vcpu='3' cpuset='52'/>
    <emulatorpin cpuset='7,16,43,52'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>


# VM server_on_host2 cpu pinning info (after this VM was migrated to host1)
  <cputune>
    <shares>4096</shares>
    <vcpupin vcpu='0' cpuset='43'/>
    <vcpupin vcpu='1' cpuset='7'/>
    <vcpupin vcpu='2' cpuset='16'/>
    <vcpupin vcpu='3' cpuset='52'/>
    <emulatorpin cpuset='7,16,43,52'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>

--------------------flavor's extra spec setting----------------------
hw:cpu_policy='dedicated'
hw:numa_cpus.0='0-3'
hw:numa_mem.0='512'
hw:numa_nodes='1'


---------------------server_on_host1 domain xml-----------------------
$ sudo virsh dumpxml instance-0000003d                                                                                        
<domain type='kvm' id='36'>
  <name>instance-0000003d</name>
  <uuid>0f84973c-62de-4eb0-ac3e-ba3f6f8a869b</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="18.1.0"/>
      <nova:name>server_on_host1</nova:name>
      <nova:creationTime>2019-03-01 05:14:59</nova:creationTime>
      <nova:flavor name="my.flavor.v2">
        <nova:memory>512</nova:memory>
        <nova:disk>1</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="565fc373b2cb4a058056e777d4e833e6">admin</nova:user>
        <nova:project uuid="71b6738db2d149ba8053ca913d3de99f">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="2822ee87-f612-4f26-9321-44a7301540a8"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <shares>4096</shares>
    <vcpupin vcpu='0' cpuset='43'/>
    <vcpupin vcpu='1' cpuset='7'/>
    <vcpupin vcpu='2' cpuset='16'/>
    <vcpupin vcpu='3' cpuset='52'/>
    <emulatorpin cpuset='7,16,43,52'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>18.1.0</entry>
      <entry name='serial'>0f84973c-62de-4eb0-ac3e-ba3f6f8a869b</entry>
      <entry name='uuid'>0f84973c-62de-4eb0-ac3e-ba3f6f8a869b</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-bionic'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu>
    <topology sockets='2' cores='1' threads='2'/>
    <numa>
      <cell id='0' cpus='0-3' memory='524288' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/nova/instances/0f84973c-62de-4eb0-ac3e-ba3f6f8a869b/disk'/>
      <backingStore type='file' index='1'>
        <format type='raw'/>
        <source file='/var/lib/nova/instances/_base/d40ac6a29c9af85483c06fe1b8ac167dd325f31f'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:39:ab:97'/>
      <source bridge='br-int'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='5d59772f-660d-4956-82d6-1b6a455ba34d'/>
      </virtualport>
      <target dev='tap5d59772f-66'/>
      <model type='virtio'/>
      <mtu size='1450'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/7'/>
      <log file='/var/lib/nova/instances/0f84973c-62de-4eb0-ac3e-ba3f6f8a869b/console.log' append='off'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/7'>
      <source path='/dev/pts/7'/>
      <log file='/var/lib/nova/instances/0f84973c-62de-4eb0-ac3e-ba3f6f8a869b/console.log' append='off'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-0f84973c-62de-4eb0-ac3e-ba3f6f8a869b</label>
    <imagelabel>libvirt-0f84973c-62de-4eb0-ac3e-ba3f6f8a869b</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+64055:+125</label>
    <imagelabel>+64055:+125</imagelabel>
  </seclabel>
</domain>



---------------------server_on_host2 domain xml-----------------------
$ sudo virsh dumpxml instance-0000003f
<domain type='kvm' id='38'>
  <name>instance-0000003f</name>
  <uuid>ef5505e5-746d-4ee5-8941-4715ff3de088</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="18.1.0"/>
      <nova:name>server_on_host2</nova:name>
      <nova:creationTime>2019-03-01 05:43:05</nova:creationTime>
      <nova:flavor name="my.flavor.v2">
        <nova:memory>512</nova:memory>
        <nova:disk>1</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="565fc373b2cb4a058056e777d4e833e6">admin</nova:user>
        <nova:project uuid="71b6738db2d149ba8053ca913d3de99f">admin</nova:project>
      </nova:owner>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <shares>4096</shares>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='38'/>
    <vcpupin vcpu='2' cpuset='8'/>
    <vcpupin vcpu='3' cpuset='44'/>
    <emulatorpin cpuset='2,8,38,44'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>18.1.0</entry>
      <entry name='serial'>ef5505e5-746d-4ee5-8941-4715ff3de088</entry>
      <entry name='uuid'>ef5505e5-746d-4ee5-8941-4715ff3de088</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-bionic'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu>
    <topology sockets='2' cores='1' threads='2'/>
    <numa>
      <cell id='0' cpus='0-3' memory='524288' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sde'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>c42d2260-46a6-4833-a59a-acfc0da510d5</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:9e:8d:89'/>
      <source bridge='br-int'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='36afc28c-21ce-4e41-a52f-e5a4bfb90853'/>
      </virtualport>
      <target dev='tap36afc28c-21'/>
      <model type='virtio'/>
      <mtu size='1450'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/19'/>
      <log file='/var/lib/nova/instances/ef5505e5-746d-4ee5-8941-4715ff3de088/console.log' append='off'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/19'>
      <source path='/dev/pts/19'/>
      <log file='/var/lib/nova/instances/ef5505e5-746d-4ee5-8941-4715ff3de088/console.log' append='off'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5901' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-ef5505e5-746d-4ee5-8941-4715ff3de088</label>
    <imagelabel>libvirt-ef5505e5-746d-4ee5-8941-4715ff3de088</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+64055:+125</label>
    <imagelabel>+64055:+125</imagelabel>
  </seclabel>
</domain>