[nova] NUMA live migration - mostly how it's tested
Hey all, There won't be much new here for those who've reviewed the patches [1] already, but I wanted to address the testing situation. Until recently, the last patch was WIP because I had functional tests but no unit tests. Even without NUMA anywhere, the claims part of the new code could be tested in functional tests. With the new and improved implementation proposed by Dan Smith [2], this is no longer the case. Any test more involved than unit testing will need "real" NUMA instances on "real" NUMA hosts to trigger the new code. Because of that, I've dropped functional testing altogether, have added unit tests, and have taken the WIP tag off. What I've been using for testing is this: [3]. It's a series of patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch of us Nova Red Hatters to automate testing that's outside of Tempest's scope. Same idea as the intel-nfv-ci plugin [4]. The tests I currently have check that: * CPU pin mapping is updated if the destination has an instance pinned to the same CPUs as the incoming instance * emulator thread pins are updated if the destination has a different cpu_shared_set value and the instance has the hw:emulator_threads_policy set to `share` * NUMA node pins are updated for a hugepages instance if the destination has a hugepages instances consuming the same NUMA node as the incoming instance It's not exhaustive by any means, but I've made sure that all iterations pass those 3 tests. It should be fairly easy to add new tests, as most of the necessary scaffolding is already in place. [1] https://review.openstack.org/#/c/634606/ [2] https://review.openstack.org/#/c/634828/28/nova/virt/driver.py@1147 [3] https://review.rdoproject.org/r/#/c/18832/ [4] https://github.com/openstack/intel-nfv-ci-tests/
On 2/27/2019 7:25 PM, Artom Lifshitz wrote:
What I've been using for testing is this: [3]. It's a series of patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch of us Nova Red Hatters to automate testing that's outside of Tempest's scope.
And where is that pulling in your nova series of changes and posting test results (like a 3rd party CI) so anyone can see it? Or do you mean here are tests, but you need to provide your own environment if you want to verify the code prior to merging it. Can we really not even have functional tests with the fake libvirt driver and fake numa resources to ensure the flow doesn't blow up? -- Thanks, Matt
On Wed, Feb 27, 2019, 21:27 Matt Riedemann, <mriedemos@gmail.com> wrote:
On 2/27/2019 7:25 PM, Artom Lifshitz wrote:
What I've been using for testing is this: [3]. It's a series of patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch of us Nova Red Hatters to automate testing that's outside of Tempest's scope.
And where is that pulling in your nova series of changes and posting test results (like a 3rd party CI) so anyone can see it? Or do you mean here are tests, but you need to provide your own environment if you want to verify the code prior to merging it.
Sorry, wasn't clear. It's the latter. The test code exists, and has run against my devstack environment with my patches checked out, but there's no CI or public posting of test results. Getting CI coverage for these NUMA things (like the old Intel one) is a whole other topic.
Can we really not even have functional tests with the fake libvirt
driver and fake numa resources to ensure the flow doesn't blow up?
That's something I have to look into. We have live migration functional tests, and we have NUMA functional tests, but I'm not sure how we can combine the two.
--
Thanks,
Matt
On Wed, 2019-02-27 at 21:33 -0500, Artom Lifshitz wrote:
On Wed, Feb 27, 2019, 21:27 Matt Riedemann, <mriedemos@gmail.com> wrote:
On 2/27/2019 7:25 PM, Artom Lifshitz wrote:
What I've been using for testing is this: [3]. It's a series of patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch of us Nova Red Hatters to automate testing that's outside of Tempest's scope.
And where is that pulling in your nova series of changes and posting test results (like a 3rd party CI) so anyone can see it? Or do you mean here are tests, but you need to provide your own environment if you want to verify the code prior to merging it.
Sorry, wasn't clear. It's the latter. The test code exists, and has run against my devstack environment with my patches checked out, but there's no CI or public posting of test results. Getting CI coverage for these NUMA things (like the old Intel one) is a whole other topic.
on the ci front i resolved the nested vert on the server i bought to set up a personal ci for numa testing. that set me back a few weeks in setting up that ci but i hope to run artom whitebox test amoung other in that at some point. vexhost also provided nested virt to the gate vms. im going to see if we can actully create a non voting job using the ubuntu-bionic-vexxhost nodeset. if ovh or one of the other providers of ci resource renable nested virt then we can maybe make that job voting and not need thridparty ci anymor.
Can we really not even have functional tests with the fake libvirt driver and fake numa resources to ensure the flow doesn't blow up?
That's something I have to look into. We have live migration functional tests, and we have NUMA functional tests, but I'm not sure how we can combine the two.
jus as an addtional proof point im am planning to do a bunch of migration and live migration testing in the next 2-4 weeks. my current backlog on no particalar order is sriov migration numa migration vtpm migration cross-cell migration cross-neutron backend migration (ovs<->linuxbridge) cross-firwall migraton (iptables<->contrack) (previously tested and worked at end of queens) narrowong in on the numa migration the current set of testcases i plan to manually verify are as follows: note assume all flavor will have 256mb of ram and 4 cores unless otherwise stated basic tests pinned guests (hw:cpu_policy=dedicated) pinned-isolated guests (hw:cpu_policy=dedicated hw:thread_policy=isolate) pinned-prefer guests (hw:cpu_policy=dedicated hw:thread_policy=prefer) unpinned-singel-numa guest (hw:numa_nodes=1) unpinned-dual-numa guest (hw:numa_nodes=2) unpinned-dual-numa-unblanced guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192) unpinned-hugepage-implcit numa guest (hw:mem_page_size=large) unpinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2) pinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2 hw:cpu_policy=dedicated) realtime guest (hw:cpu_policy=dedicated hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1) emulator-thread-iosolated guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=isolate) advanced tests (require extra nova.conf changes) emulator-thread-shared guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=shared) note cpu_share_set configrued unpinned-singel-numa-hetorgious-host guest (hw:numa_nodes=1) note vcpu_pin_set adjusted so that host 1 only has cpus on numa 1 and host 2 only has cpus on numa node 2. supper-optimiesd-guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=isolate) supper-optimiesd-guest-2 (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=share) for each of these test ill provide a test-command file with the command i used to run the tests and reustlts file with a summary at the top plus the xmls before and after the migration showing that intially the resouces would conflict on migration and then the updated xmls after the migration. i will also provide the local.conf for the devstack deployment and some details about the env like distor/qemu/libvirt versions. eventurally i hope all those test cases can be added to the whitebox plugin and verifed in a ci. we could also try and valideate them in functional tests. i have attached the xml for the pinned guest as an example of what to expect but i will be compileing this slowly as i go and zip everying up in an email to the list. this will take some time to complete and hosestly i had planned to do most of this testing after feature freeze when we can focus on testing more. regards sean
Hi all, There was something wrong with the live migration when using 'dedicated' cpu_policy in my test. Attached file contains the details. On 2019/2/28 下午9:28, Sean Mooney wrote:
On Wed, 2019-02-27 at 21:33 -0500, Artom Lifshitz wrote:
On Wed, Feb 27, 2019, 21:27 Matt Riedemann, <mriedemos@gmail.com> wrote:
On 2/27/2019 7:25 PM, Artom Lifshitz wrote:
What I've been using for testing is this: [3]. It's a series of patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch of us Nova Red Hatters to automate testing that's outside of Tempest's scope.
And where is that pulling in your nova series of changes and posting test results (like a 3rd party CI) so anyone can see it? Or do you mean here are tests, but you need to provide your own environment if you want to verify the code prior to merging it.
Sorry, wasn't clear. It's the latter. The test code exists, and has run against my devstack environment with my patches checked out, but there's no CI or public posting of test results. Getting CI coverage for these NUMA things (like the old Intel one) is a whole other topic.
on the ci front i resolved the nested vert on the server i bought to set up a personal ci for numa testing. that set me back a few weeks in setting up that ci but i hope to run artom whitebox test amoung other in that at some point. vexhost also provided nested virt to the gate vms. im going to see if we can actully create a non voting job using the ubuntu-bionic-vexxhost nodeset. if ovh or one of the other providers of ci resource renable nested virt then we can maybe make that job voting and not need thridparty ci anymor.
Can we really not even have functional tests with the fake libvirt driver and fake numa resources to ensure the flow doesn't blow up?
That's something I have to look into. We have live migration functional tests, and we have NUMA functional tests, but I'm not sure how we can combine the two.
jus as an addtional proof point im am planning to do a bunch of migration and live migration testing in the next 2-4 weeks.
my current backlog on no particalar order is sriov migration numa migration vtpm migration cross-cell migration cross-neutron backend migration (ovs<->linuxbridge) cross-firwall migraton (iptables<->contrack) (previously tested and worked at end of queens)
narrowong in on the numa migration the current set of testcases i plan to manually verify are as follows:
note assume all flavor will have 256mb of ram and 4 cores unless otherwise stated
basic tests pinned guests (hw:cpu_policy=dedicated) pinned-isolated guests (hw:cpu_policy=dedicated hw:thread_policy=isolate) pinned-prefer guests (hw:cpu_policy=dedicated hw:thread_policy=prefer) unpinned-singel-numa guest (hw:numa_nodes=1) unpinned-dual-numa guest (hw:numa_nodes=2) unpinned-dual-numa-unblanced guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192) unpinned-hugepage-implcit numa guest (hw:mem_page_size=large) unpinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2) pinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2 hw:cpu_policy=dedicated) realtime guest (hw:cpu_policy=dedicated hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1) emulator-thread-iosolated guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=isolate)
advanced tests (require extra nova.conf changes) emulator-thread-shared guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=shared) note cpu_share_set configrued unpinned-singel-numa-hetorgious-host guest (hw:numa_nodes=1) note vcpu_pin_set adjusted so that host 1 only has cpus on numa 1 and host 2 only has cpus on numa node 2. supper-optimiesd-guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=isolate) supper-optimiesd-guest-2 (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=share)
for each of these test ill provide a test-command file with the command i used to run the tests and reustlts file with a summary at the top plus the xmls before and after the migration showing that intially the resouces would conflict on migration and then the updated xmls after the migration. i will also provide the local.conf for the devstack deployment and some details about the env like distor/qemu/libvirt versions.
eventurally i hope all those test cases can be added to the whitebox plugin and verifed in a ci. we could also try and valideate them in functional tests.
i have attached the xml for the pinned guest as an example of what to expect but i will be compileing this slowly as i go and zip everying up in an email to the list. this will take some time to complete and hosestly i had planned to do most of this testing after feature freeze when we can focus on testing more.
regards sean
Hi all, There was something wrong with the live migration when using 'dedicated' cpu_policy in my test. Attached file contains the details. The message body will be so big that it will be held if I attach the debug info, so I will send another email. Regards, Luyao On 2019/2/28 下午9:28, Sean Mooney wrote:
On Wed, 2019-02-27 at 21:33 -0500, Artom Lifshitz wrote:
On Wed, Feb 27, 2019, 21:27 Matt Riedemann, <mriedemos@gmail.com> wrote:
On 2/27/2019 7:25 PM, Artom Lifshitz wrote:
What I've been using for testing is this: [3]. It's a series of patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch of us Nova Red Hatters to automate testing that's outside of Tempest's scope.
And where is that pulling in your nova series of changes and posting test results (like a 3rd party CI) so anyone can see it? Or do you mean here are tests, but you need to provide your own environment if you want to verify the code prior to merging it.
Sorry, wasn't clear. It's the latter. The test code exists, and has run against my devstack environment with my patches checked out, but there's no CI or public posting of test results. Getting CI coverage for these NUMA things (like the old Intel one) is a whole other topic.
on the ci front i resolved the nested vert on the server i bought to set up a personal ci for numa testing. that set me back a few weeks in setting up that ci but i hope to run artom whitebox test amoung other in that at some point. vexhost also provided nested virt to the gate vms. im going to see if we can actully create a non voting job using the ubuntu-bionic-vexxhost nodeset. if ovh or one of the other providers of ci resource renable nested virt then we can maybe make that job voting and not need thridparty ci anymor.
Can we really not even have functional tests with the fake libvirt driver and fake numa resources to ensure the flow doesn't blow up?
That's something I have to look into. We have live migration functional tests, and we have NUMA functional tests, but I'm not sure how we can combine the two.
jus as an addtional proof point im am planning to do a bunch of migration and live migration testing in the next 2-4 weeks.
my current backlog on no particalar order is sriov migration numa migration vtpm migration cross-cell migration cross-neutron backend migration (ovs<->linuxbridge) cross-firwall migraton (iptables<->contrack) (previously tested and worked at end of queens)
narrowong in on the numa migration the current set of testcases i plan to manually verify are as follows:
note assume all flavor will have 256mb of ram and 4 cores unless otherwise stated
basic tests pinned guests (hw:cpu_policy=dedicated) pinned-isolated guests (hw:cpu_policy=dedicated hw:thread_policy=isolate) pinned-prefer guests (hw:cpu_policy=dedicated hw:thread_policy=prefer) unpinned-singel-numa guest (hw:numa_nodes=1) unpinned-dual-numa guest (hw:numa_nodes=2) unpinned-dual-numa-unblanced guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192) unpinned-hugepage-implcit numa guest (hw:mem_page_size=large) unpinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2) pinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2 hw:cpu_policy=dedicated) realtime guest (hw:cpu_policy=dedicated hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1) emulator-thread-iosolated guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=isolate)
advanced tests (require extra nova.conf changes) emulator-thread-shared guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=shared) note cpu_share_set configrued unpinned-singel-numa-hetorgious-host guest (hw:numa_nodes=1) note vcpu_pin_set adjusted so that host 1 only has cpus on numa 1 and host 2 only has cpus on numa node 2. supper-optimiesd-guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=isolate) supper-optimiesd-guest-2 (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=share)
for each of these test ill provide a test-command file with the command i used to run the tests and reustlts file with a summary at the top plus the xmls before and after the migration showing that intially the resouces would conflict on migration and then the updated xmls after the migration. i will also provide the local.conf for the devstack deployment and some details about the env like distor/qemu/libvirt versions.
eventurally i hope all those test cases can be added to the whitebox plugin and verifed in a ci. we could also try and valideate them in functional tests.
i have attached the xml for the pinned guest as an example of what to expect but i will be compileing this slowly as i go and zip everying up in an email to the list. this will take some time to complete and hosestly i had planned to do most of this testing after feature freeze when we can focus on testing more.
regards sean
Attached is debug log. On 2019/3/1 下午5:30, Luyao Zhong wrote:
Hi all,
There was something wrong with the live migration when using 'dedicated' cpu_policy in my test. Attached file contains the details.
The message body will be so big that it will be held if I attach the debug info, so I will send another email.
Regards, Luyao
On 2019/2/28 下午9:28, Sean Mooney wrote:
On Wed, 2019-02-27 at 21:33 -0500, Artom Lifshitz wrote:
On Wed, Feb 27, 2019, 21:27 Matt Riedemann, <mriedemos@gmail.com> wrote:
On 2/27/2019 7:25 PM, Artom Lifshitz wrote:
What I've been using for testing is this: [3]. It's a series of patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch of us Nova Red Hatters to automate testing that's outside of Tempest's scope.
And where is that pulling in your nova series of changes and posting test results (like a 3rd party CI) so anyone can see it? Or do you mean here are tests, but you need to provide your own environment if you want to verify the code prior to merging it.
Sorry, wasn't clear. It's the latter. The test code exists, and has run against my devstack environment with my patches checked out, but there's no CI or public posting of test results. Getting CI coverage for these NUMA things (like the old Intel one) is a whole other topic.
on the ci front i resolved the nested vert on the server i bought to set up a personal ci for numa testing. that set me back a few weeks in setting up that ci but i hope to run artom whitebox test amoung other in that at some point. vexhost also provided nested virt to the gate vms. im going to see if we can actully create a non voting job using the ubuntu-bionic-vexxhost nodeset. if ovh or one of the other providers of ci resource renable nested virt then we can maybe make that job voting and not need thridparty ci anymor.
Can we really not even have functional tests with the fake libvirt driver and fake numa resources to ensure the flow doesn't blow up?
That's something I have to look into. We have live migration functional tests, and we have NUMA functional tests, but I'm not sure how we can combine the two.
jus as an addtional proof point im am planning to do a bunch of migration and live migration testing in the next 2-4 weeks.
my current backlog on no particalar order is sriov migration numa migration vtpm migration cross-cell migration cross-neutron backend migration (ovs<->linuxbridge) cross-firwall migraton (iptables<->contrack) (previously tested and worked at end of queens)
narrowong in on the numa migration the current set of testcases i plan to manually verify are as follows:
note assume all flavor will have 256mb of ram and 4 cores unless otherwise stated
basic tests pinned guests (hw:cpu_policy=dedicated) pinned-isolated guests (hw:cpu_policy=dedicated hw:thread_policy=isolate) pinned-prefer guests (hw:cpu_policy=dedicated hw:thread_policy=prefer) unpinned-singel-numa guest (hw:numa_nodes=1) unpinned-dual-numa guest (hw:numa_nodes=2) unpinned-dual-numa-unblanced guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192) unpinned-hugepage-implcit numa guest (hw:mem_page_size=large) unpinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2) pinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2 hw:cpu_policy=dedicated) realtime guest (hw:cpu_policy=dedicated hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1) emulator-thread-iosolated guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=isolate)
advanced tests (require extra nova.conf changes) emulator-thread-shared guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=shared) note cpu_share_set configrued unpinned-singel-numa-hetorgious-host guest (hw:numa_nodes=1) note vcpu_pin_set adjusted so that host 1 only has cpus on numa 1 and host 2 only has cpus on numa node 2. supper-optimiesd-guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=isolate) supper-optimiesd-guest-2 (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=share)
for each of these test ill provide a test-command file with the command i used to run the tests and reustlts file with a summary at the top plus the xmls before and after the migration showing that intially the resouces would conflict on migration and then the updated xmls after the migration. i will also provide the local.conf for the devstack deployment and some details about the env like distor/qemu/libvirt versions.
eventurally i hope all those test cases can be added to the whitebox plugin and verifed in a ci. we could also try and valideate them in functional tests.
i have attached the xml for the pinned guest as an example of what to expect but i will be compileing this slowly as i go and zip everying up in an email to the list. this will take some time to complete and hosestly i had planned to do most of this testing after feature freeze when we can focus on testing more.
regards sean
On Fri, 2019-03-01 at 17:30 +0800, Luyao Zhong wrote:
Hi all,
There was something wrong with the live migration when using 'dedicated' cpu_policy in my test. Attached file contains the details.
The message body will be so big that it will be held if I attach the debug info, so I will send another email.
looking at your original email you stated that # VM server_on_host1 cpu pinning info <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='43'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='16'/> <vcpupin vcpu='3' cpuset='52'/> <emulatorpin cpuset='7,16,43,52'/> </cputune> <numatune> <memory mode='strict' nodeset='0'/> <memnode cellid='0' mode='strict' nodeset='0'/> </numatune> # VM server_on_host2 cpu pinning info (before migration) <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='43'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='16'/> <vcpupin vcpu='3' cpuset='52'/> <emulatorpin cpuset='7,16,43,52'/> </cputune> <numatune> <memory mode='strict' nodeset='0'/> <memnode cellid='0' mode='strict' nodeset='0'/> </numatune> however looking at the full domain server_on_host1 was <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='43'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='16'/> <vcpupin vcpu='3' cpuset='52'/> <emulatorpin cpuset='7,16,43,52'/> </cputune> but server_on_host2 was <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='38'/> <vcpupin vcpu='2' cpuset='8'/> <vcpupin vcpu='3' cpuset='44'/> <emulatorpin cpuset='2,8,38,44'/> assuming the full xml attached for server_on_host2 is correct then this shows that the code is working correctly as it nolonger overlaps.
Regards, Luyao
On 2019/2/28 下午9:28, Sean Mooney wrote:
On Wed, 2019-02-27 at 21:33 -0500, Artom Lifshitz wrote:
On Wed, Feb 27, 2019, 21:27 Matt Riedemann, <mriedemos@gmail.com> wrote:
On 2/27/2019 7:25 PM, Artom Lifshitz wrote:
What I've been using for testing is this: [3]. It's a series of patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch of us Nova Red Hatters to automate testing that's outside of Tempest's scope.
And where is that pulling in your nova series of changes and posting test results (like a 3rd party CI) so anyone can see it? Or do you mean here are tests, but you need to provide your own environment if you want to verify the code prior to merging it.
Sorry, wasn't clear. It's the latter. The test code exists, and has run against my devstack environment with my patches checked out, but there's no CI or public posting of test results. Getting CI coverage for these NUMA things (like the old Intel one) is a whole other topic.
on the ci front i resolved the nested vert on the server i bought to set up a personal ci for numa testing. that set me back a few weeks in setting up that ci but i hope to run artom whitebox test amoung other in that at some point. vexhost also provided nested virt to the gate vms. im going to see if we can actully create a non voting job using the ubuntu-bionic-vexxhost nodeset. if ovh or one of the other providers of ci resource renable nested virt then we can maybe make that job voting and not need thridparty ci anymor.
Can we really not even have functional tests with the fake libvirt driver and fake numa resources to ensure the flow doesn't blow up?
That's something I have to look into. We have live migration functional tests, and we have NUMA functional tests, but I'm not sure how we can combine the two.
jus as an addtional proof point im am planning to do a bunch of migration and live migration testing in the next 2-4 weeks.
my current backlog on no particalar order is sriov migration numa migration vtpm migration cross-cell migration cross-neutron backend migration (ovs<->linuxbridge) cross-firwall migraton (iptables<->contrack) (previously tested and worked at end of queens)
narrowong in on the numa migration the current set of testcases i plan to manually verify are as follows:
note assume all flavor will have 256mb of ram and 4 cores unless otherwise stated
basic tests pinned guests (hw:cpu_policy=dedicated) pinned-isolated guests (hw:cpu_policy=dedicated hw:thread_policy=isolate) pinned-prefer guests (hw:cpu_policy=dedicated hw:thread_policy=prefer) unpinned-singel-numa guest (hw:numa_nodes=1) unpinned-dual-numa guest (hw:numa_nodes=2) unpinned-dual-numa-unblanced guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192) unpinned-hugepage-implcit numa guest (hw:mem_page_size=large) unpinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2) pinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2 hw:cpu_policy=dedicated) realtime guest (hw:cpu_policy=dedicated hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1) emulator-thread-iosolated guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=isolate)
advanced tests (require extra nova.conf changes) emulator-thread-shared guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=shared) note cpu_share_set configrued unpinned-singel-numa-hetorgious-host guest (hw:numa_nodes=1) note vcpu_pin_set adjusted so that host 1 only has cpus on numa 1 and host 2 only has cpus on numa node 2. supper-optimiesd-guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=isolate) supper-optimiesd-guest-2 (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=share)
for each of these test ill provide a test-command file with the command i used to run the tests and reustlts file with a summary at the top plus the xmls before and after the migration showing that intially the resouces would conflict on migration and then the updated xmls after the migration. i will also provide the local.conf for the devstack deployment and some details about the env like distor/qemu/libvirt versions.
eventurally i hope all those test cases can be added to the whitebox plugin and verifed in a ci. we could also try and valideate them in functional tests.
i have attached the xml for the pinned guest as an example of what to expect but i will be compileing this slowly as i go and zip everying up in an email to the list. this will take some time to complete and hosestly i had planned to do most of this testing after feature freeze when we can focus on testing more.
regards sean
在 2019年3月1日，下午9:30，Sean Mooney <smooney@redhat.com> 写道：
On Fri, 2019-03-01 at 17:30 +0800, Luyao Zhong wrote: Hi all,
There was something wrong with the live migration when using 'dedicated' cpu_policy in my test. Attached file contains the details.
The message body will be so big that it will be held if I attach the debug info, so I will send another email.
looking at your original email you stated that
# VM server_on_host1 cpu pinning info <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='43'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='16'/> <vcpupin vcpu='3' cpuset='52'/> <emulatorpin cpuset='7,16,43,52'/> </cputune> <numatune> <memory mode='strict' nodeset='0'/> <memnode cellid='0' mode='strict' nodeset='0'/> </numatune>
# VM server_on_host2 cpu pinning info (before migration) <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='43'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='16'/> <vcpupin vcpu='3' cpuset='52'/> <emulatorpin cpuset='7,16,43,52'/> </cputune> <numatune> <memory mode='strict' nodeset='0'/> <memnode cellid='0' mode='strict' nodeset='0'/> </numatune>
however looking at the full domain server_on_host1 was <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='43'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='16'/> <vcpupin vcpu='3' cpuset='52'/> <emulatorpin cpuset='7,16,43,52'/> </cputune>
but server_on_host2 was <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='38'/> <vcpupin vcpu='2' cpuset='8'/> <vcpupin vcpu='3' cpuset='44'/> <emulatorpin cpuset='2,8,38,44'/>
assuming the full xml attached for server_on_host2 is correct then this shows that the code is working correctly as it nolonger overlaps.
I use virsh dumpxml to get the full xml, and I usually use virsh edit to see the xml, so I didn’t notice this difference before. When using virsh edit ，I can see the overlaps. I’m not sure why I will get different results between “virsh edit” and “virsh dumpxml”？
Regards, Luyao
On 2019/2/28 下午9:28, Sean Mooney wrote:
On Wed, 2019-02-27 at 21:33 -0500, Artom Lifshitz wrote:
On Wed, Feb 27, 2019, 21:27 Matt Riedemann, <mriedemos@gmail.com> wrote:
On 2/27/2019 7:25 PM, Artom Lifshitz wrote: What I've been using for testing is this: [3]. It's a series of patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch of us Nova Red Hatters to automate testing that's outside of Tempest's scope.
And where is that pulling in your nova series of changes and posting test results (like a 3rd party CI) so anyone can see it? Or do you mean here are tests, but you need to provide your own environment if you want to verify the code prior to merging it.
Sorry, wasn't clear. It's the latter. The test code exists, and has run against my devstack environment with my patches checked out, but there's no CI or public posting of test results. Getting CI coverage for these NUMA things (like the old Intel one) is a whole other topic.
on the ci front i resolved the nested vert on the server i bought to set up a personal ci for numa testing. that set me back a few weeks in setting up that ci but i hope to run artom whitebox test amoung other in that at some point. vexhost also provided nested virt to the gate vms. im going to see if we can actully create a non voting job using the ubuntu-bionic-vexxhost nodeset. if ovh or one of the other providers of ci resource renable nested virt then we can maybe make that job voting and not need thridparty ci anymor.
Can we really not even have functional tests with the fake libvirt driver and fake numa resources to ensure the flow doesn't blow up?
That's something I have to look into. We have live migration functional tests, and we have NUMA functional tests, but I'm not sure how we can combine the two.
jus as an addtional proof point im am planning to do a bunch of migration and live migration testing in the next 2-4 weeks.
my current backlog on no particalar order is sriov migration numa migration vtpm migration cross-cell migration cross-neutron backend migration (ovs<->linuxbridge) cross-firwall migraton (iptables<->contrack) (previously tested and worked at end of queens)
narrowong in on the numa migration the current set of testcases i plan to manually verify are as follows:
note assume all flavor will have 256mb of ram and 4 cores unless otherwise stated
basic tests pinned guests (hw:cpu_policy=dedicated) pinned-isolated guests (hw:cpu_policy=dedicated hw:thread_policy=isolate) pinned-prefer guests (hw:cpu_policy=dedicated hw:thread_policy=prefer) unpinned-singel-numa guest (hw:numa_nodes=1) unpinned-dual-numa guest (hw:numa_nodes=2) unpinned-dual-numa-unblanced guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192) unpinned-hugepage-implcit numa guest (hw:mem_page_size=large) unpinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2) pinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2 hw:cpu_policy=dedicated) realtime guest (hw:cpu_policy=dedicated hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1) emulator-thread-iosolated guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=isolate)
advanced tests (require extra nova.conf changes) emulator-thread-shared guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=shared) note cpu_share_set configrued unpinned-singel-numa-hetorgious-host guest (hw:numa_nodes=1) note vcpu_pin_set adjusted so that host 1 only has cpus on numa 1 and host 2 only has cpus on numa node 2. supper-optimiesd-guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=isolate) supper-optimiesd-guest-2 (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=share)
for each of these test ill provide a test-command file with the command i used to run the tests and reustlts file with a summary at the top plus the xmls before and after the migration showing that intially the resouces would conflict on migration and then the updated xmls after the migration. i will also provide the local.conf for the devstack deployment and some details about the env like distor/qemu/libvirt versions.
eventurally i hope all those test cases can be added to the whitebox plugin and verifed in a ci. we could also try and valideate them in functional tests.
i have attached the xml for the pinned guest as an example of what to expect but i will be compileing this slowly as i go and zip everying up in an email to the list. this will take some time to complete and hosestly i had planned to do most of this testing after feature freeze when we can focus on testing more.
regards sean
On Sat, 2019-03-02 at 04:38 +0000, Zhong, Luyao wrote:
在 2019年3月1日，下午9:30，Sean Mooney <smooney@redhat.com> 写道：
On Fri, 2019-03-01 at 17:30 +0800, Luyao Zhong wrote: Hi all,
There was something wrong with the live migration when using 'dedicated' cpu_policy in my test. Attached file contains the details.
The message body will be so big that it will be held if I attach the debug info, so I will send another email.
looking at your original email you stated that
# VM server_on_host1 cpu pinning info <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='43'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='16'/> <vcpupin vcpu='3' cpuset='52'/> <emulatorpin cpuset='7,16,43,52'/> </cputune> <numatune> <memory mode='strict' nodeset='0'/> <memnode cellid='0' mode='strict' nodeset='0'/> </numatune>
# VM server_on_host2 cpu pinning info (before migration) <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='43'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='16'/> <vcpupin vcpu='3' cpuset='52'/> <emulatorpin cpuset='7,16,43,52'/> </cputune> <numatune> <memory mode='strict' nodeset='0'/> <memnode cellid='0' mode='strict' nodeset='0'/> </numatune>
however looking at the full domain server_on_host1 was <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='43'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='16'/> <vcpupin vcpu='3' cpuset='52'/> <emulatorpin cpuset='7,16,43,52'/> </cputune>
but server_on_host2 was <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='38'/> <vcpupin vcpu='2' cpuset='8'/> <vcpupin vcpu='3' cpuset='44'/> <emulatorpin cpuset='2,8,38,44'/>
assuming the full xml attached for server_on_host2 is correct then this shows that the code is working correctly as it nolonger overlaps.
I use virsh dumpxml to get the full xml, and I usually use virsh edit to see the xml, so I didn’t notice this difference before. When using virsh edit ，I can see the overlaps. I’m not sure why I will get different results between “virsh edit” and “virsh dumpxml”？
at first i taught that sounded like a libvirt bug however the description of both command differ slightly dumpxml domain [--inactive] [--security-info] [--update-cpu] [--migratable] Output the domain information as an XML dump to stdout, this format can be used by the create command. Additional options affecting the XML dump may be used. --inactive tells virsh to dump domain configuration that will be used on next start of the domain as opposed to the current domain configuration. Using --security-info will also include security sensitive information in the XML dump. --update-cpu updates domain CPU requirements according to host CPU. With --migratable one can request an XML that is suitable for migrations, i.e., compatible with older libvirt releases and possibly amended with internal run-time options. This option may automatically enable other options (--update-cpu, --security-info, ...) as necessary. edit domain Edit the XML configuration file for a domain, which will affect the next boot of the guest. This is equivalent to: virsh dumpxml --inactive --security-info domain > domain.xml vi domain.xml (or make changes with your other text editor) virsh define domain.xml except that it does some error checking. The editor used can be supplied by the $VISUAL or $EDITOR environment variables, and defaults to "vi". i think virsh edit is showing you the state the vm will have on next reboot. which appears to be the original xml not the migration xml that was used to move the instance. since openstack destorys the xml and recates it form scratch every time the outpu of virsh edit can be ignored. virs dumpxml will show the current state of the domain. if you did virsh dumpxml --inactive it would likely match virsh edit. its possible the modified xml we use when migrating a domain is is considered transient but that is my best guess as to why they are different. for evaulating this we should use the values of virsh dumpxml.
Regards, Luyao
On 2019/2/28 下午9:28, Sean Mooney wrote:
On Wed, 2019-02-27 at 21:33 -0500, Artom Lifshitz wrote:
On Wed, Feb 27, 2019, 21:27 Matt Riedemann, <mriedemos@gmail.com> wrote: > On 2/27/2019 7:25 PM, Artom Lifshitz wrote: > What I've been using for testing is this: [3]. It's a series of > patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch > of us Nova Red Hatters to automate testing that's outside of Tempest's > scope.
And where is that pulling in your nova series of changes and posting test results (like a 3rd party CI) so anyone can see it? Or do you mean here are tests, but you need to provide your own environment if you want to verify the code prior to merging it.
Sorry, wasn't clear. It's the latter. The test code exists, and has run against my devstack environment with my patches checked out, but there's no CI or public posting of test results. Getting CI coverage for these NUMA things (like the old Intel one) is a whole other topic.
on the ci front i resolved the nested vert on the server i bought to set up a personal ci for numa testing. that set me back a few weeks in setting up that ci but i hope to run artom whitebox test amoung other in that at some point. vexhost also provided nested virt to the gate vms. im going to see if we can actully create a non voting job using the ubuntu-bionic-vexxhost nodeset. if ovh or one of the other providers of ci resource renable nested virt then we can maybe make that job voting and not need thridparty ci anymor.
Can we really not even have functional tests with the fake libvirt driver and fake numa resources to ensure the flow doesn't blow up?
That's something I have to look into. We have live migration functional tests, and we have NUMA functional tests, but I'm not sure how we can combine the two.
jus as an addtional proof point im am planning to do a bunch of migration and live migration testing in the next 2-4 weeks.
my current backlog on no particalar order is sriov migration numa migration vtpm migration cross-cell migration cross-neutron backend migration (ovs<->linuxbridge) cross-firwall migraton (iptables<->contrack) (previously tested and worked at end of queens)
narrowong in on the numa migration the current set of testcases i plan to manually verify are as follows:
note assume all flavor will have 256mb of ram and 4 cores unless otherwise stated
basic tests pinned guests (hw:cpu_policy=dedicated) pinned-isolated guests (hw:cpu_policy=dedicated hw:thread_policy=isolate) pinned-prefer guests (hw:cpu_policy=dedicated hw:thread_policy=prefer) unpinned-singel-numa guest (hw:numa_nodes=1) unpinned-dual-numa guest (hw:numa_nodes=2) unpinned-dual-numa-unblanced guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192) unpinned-hugepage-implcit numa guest (hw:mem_page_size=large) unpinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2) pinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2 hw:cpu_policy=dedicated) realtime guest (hw:cpu_policy=dedicated hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1) emulator-thread-iosolated guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=isolate)
advanced tests (require extra nova.conf changes) emulator-thread-shared guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=shared) note cpu_share_set configrued unpinned-singel-numa-hetorgious-host guest (hw:numa_nodes=1) note vcpu_pin_set adjusted so that host 1 only has cpus on numa 1 and host 2 only has cpus on numa node 2. supper-optimiesd-guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=isolate) supper-optimiesd-guest-2 (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=share)
for each of these test ill provide a test-command file with the command i used to run the tests and reustlts file with a summary at the top plus the xmls before and after the migration showing that intially the resouces would conflict on migration and then the updated xmls after the migration. i will also provide the local.conf for the devstack deployment and some details about the env like distor/qemu/libvirt versions.
eventurally i hope all those test cases can be added to the whitebox plugin and verifed in a ci. we could also try and valideate them in functional tests.
i have attached the xml for the pinned guest as an example of what to expect but i will be compileing this slowly as i go and zip everying up in an email to the list. this will take some time to complete and hosestly i had planned to do most of this testing after feature freeze when we can focus on testing more.
regards sean
On 2019/3/3 下午9:18, Sean Mooney wrote:
On Sat, 2019-03-02 at 04:38 +0000, Zhong, Luyao wrote:
在 2019年3月1日，下午9:30，Sean Mooney <smooney@redhat.com> 写道：
On Fri, 2019-03-01 at 17:30 +0800, Luyao Zhong wrote: Hi all,
There was something wrong with the live migration when using 'dedicated' cpu_policy in my test. Attached file contains the details.
The message body will be so big that it will be held if I attach the debug info, so I will send another email.
looking at your original email you stated that
# VM server_on_host1 cpu pinning info <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='43'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='16'/> <vcpupin vcpu='3' cpuset='52'/> <emulatorpin cpuset='7,16,43,52'/> </cputune> <numatune> <memory mode='strict' nodeset='0'/> <memnode cellid='0' mode='strict' nodeset='0'/> </numatune>
# VM server_on_host2 cpu pinning info (before migration) <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='43'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='16'/> <vcpupin vcpu='3' cpuset='52'/> <emulatorpin cpuset='7,16,43,52'/> </cputune> <numatune> <memory mode='strict' nodeset='0'/> <memnode cellid='0' mode='strict' nodeset='0'/> </numatune>
however looking at the full domain server_on_host1 was <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='43'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='16'/> <vcpupin vcpu='3' cpuset='52'/> <emulatorpin cpuset='7,16,43,52'/> </cputune>
but server_on_host2 was <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='38'/> <vcpupin vcpu='2' cpuset='8'/> <vcpupin vcpu='3' cpuset='44'/> <emulatorpin cpuset='2,8,38,44'/>
assuming the full xml attached for server_on_host2 is correct then this shows that the code is working correctly as it nolonger overlaps.
I use virsh dumpxml to get the full xml, and I usually use virsh edit to see the xml, so I didn’t notice this difference before. When using virsh edit ，I can see the overlaps. I’m not sure why I will get different results between “virsh edit” and “virsh dumpxml”？
at first i taught that sounded like a libvirt bug however the description of both command differ slightly
dumpxml domain [--inactive] [--security-info] [--update-cpu] [--migratable] Output the domain information as an XML dump to stdout, this format can be used by the create command. Additional options affecting the XML dump may be used. --inactive tells virsh to dump domain configuration that will be used on next start of the domain as opposed to the current domain configuration. Using --security-info will also include security sensitive information in the XML dump. --update-cpu updates domain CPU requirements according to host CPU. With --migratable one can request an XML that is suitable for migrations, i.e., compatible with older libvirt releases and possibly amended with internal run-time options. This option may automatically enable other options (--update-cpu, --security-info, ...) as necessary.
edit domain Edit the XML configuration file for a domain, which will affect the next boot of the guest.
This is equivalent to:
virsh dumpxml --inactive --security-info domain > domain.xml vi domain.xml (or make changes with your other text editor) virsh define domain.xml
except that it does some error checking.
The editor used can be supplied by the $VISUAL or $EDITOR environment variables, and defaults to "vi".
i think virsh edit is showing you the state the vm will have on next reboot. which appears to be the original xml not the migration xml that was used to move the instance.
since openstack destorys the xml and recates it form scratch every time the outpu of virsh edit can be ignored. virs dumpxml will show the current state of the domain. if you did virsh dumpxml --inactive it would likely match virsh edit. its possible the modified xml we use when migrating a domain is is considered transient but that is my best guess as to why they are different.
for evaulating this we should use the values of virsh dumpxml.
I reboot the migrated VM, and then virsh dumpxml would get the same xml as that got from virsh edit before reboot. Artom Lifshitz mentioned that the instance_numa_topology wasn't updated in the database from Windriver folks. So I guess virsh edit show xml produced by OpenStack according to the db, maybe this is why we get different xmls with 'edit' and 'dumpxml'. This is a little complex to me and still needs more test and debug work. Thank you so much for giving these details. Regards, Luyao
Regards, Luyao
On 2019/2/28 下午9:28, Sean Mooney wrote:
On Wed, 2019-02-27 at 21:33 -0500, Artom Lifshitz wrote:
> On Wed, Feb 27, 2019, 21:27 Matt Riedemann, <mriedemos@gmail.com> wrote: >> On 2/27/2019 7:25 PM, Artom Lifshitz wrote: >> What I've been using for testing is this: [3]. It's a series of >> patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch >> of us Nova Red Hatters to automate testing that's outside of Tempest's >> scope. > > And where is that pulling in your nova series of changes and posting > test results (like a 3rd party CI) so anyone can see it? Or do you mean > here are tests, but you need to provide your own environment if you want > to verify the code prior to merging it.
Sorry, wasn't clear. It's the latter. The test code exists, and has run against my devstack environment with my patches checked out, but there's no CI or public posting of test results. Getting CI coverage for these NUMA things (like the old Intel one) is a whole other topic.
on the ci front i resolved the nested vert on the server i bought to set up a personal ci for numa testing. that set me back a few weeks in setting up that ci but i hope to run artom whitebox test amoung other in that at some point. vexhost also provided nested virt to the gate vms. im going to see if we can actully create a non voting job using the ubuntu-bionic-vexxhost nodeset. if ovh or one of the other providers of ci resource renable nested virt then we can maybe make that job voting and not need thridparty ci anymor.
> Can we really not even have functional tests with the fake libvirt > driver and fake numa resources to ensure the flow doesn't blow up?
That's something I have to look into. We have live migration functional tests, and we have NUMA functional tests, but I'm not sure how we can combine the two.
jus as an addtional proof point im am planning to do a bunch of migration and live migration testing in the next 2-4 weeks.
my current backlog on no particalar order is sriov migration numa migration vtpm migration cross-cell migration cross-neutron backend migration (ovs<->linuxbridge) cross-firwall migraton (iptables<->contrack) (previously tested and worked at end of queens)
narrowong in on the numa migration the current set of testcases i plan to manually verify are as follows:
note assume all flavor will have 256mb of ram and 4 cores unless otherwise stated
basic tests pinned guests (hw:cpu_policy=dedicated) pinned-isolated guests (hw:cpu_policy=dedicated hw:thread_policy=isolate) pinned-prefer guests (hw:cpu_policy=dedicated hw:thread_policy=prefer) unpinned-singel-numa guest (hw:numa_nodes=1) unpinned-dual-numa guest (hw:numa_nodes=2) unpinned-dual-numa-unblanced guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192) unpinned-hugepage-implcit numa guest (hw:mem_page_size=large) unpinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2) pinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2 hw:cpu_policy=dedicated) realtime guest (hw:cpu_policy=dedicated hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1) emulator-thread-iosolated guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=isolate)
advanced tests (require extra nova.conf changes) emulator-thread-shared guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=shared) note cpu_share_set configrued unpinned-singel-numa-hetorgious-host guest (hw:numa_nodes=1) note vcpu_pin_set adjusted so that host 1 only has cpus on numa 1 and host 2 only has cpus on numa node 2. supper-optimiesd-guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=isolate) supper-optimiesd-guest-2 (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=share)
for each of these test ill provide a test-command file with the command i used to run the tests and reustlts file with a summary at the top plus the xmls before and after the migration showing that intially the resouces would conflict on migration and then the updated xmls after the migration. i will also provide the local.conf for the devstack deployment and some details about the env like distor/qemu/libvirt versions.
eventurally i hope all those test cases can be added to the whitebox plugin and verifed in a ci. we could also try and valideate them in functional tests.
i have attached the xml for the pinned guest as an example of what to expect but i will be compileing this slowly as i go and zip everying up in an email to the list. this will take some time to complete and hosestly i had planned to do most of this testing after feature freeze when we can focus on testing more.
regards sean
On Wed, Feb 27, 2019 at 8:25 PM Artom Lifshitz <alifshit@redhat.com> wrote:
Hey all,
There won't be much new here for those who've reviewed the patches [1] already, but I wanted to address the testing situation.
Until recently, the last patch was WIP because I had functional tests but no unit tests. Even without NUMA anywhere, the claims part of the new code could be tested in functional tests. With the new and improved implementation proposed by Dan Smith [2], this is no longer the case. Any test more involved than unit testing will need "real" NUMA instances on "real" NUMA hosts to trigger the new code. Because of that, I've dropped functional testing altogether, have added unit tests, and have taken the WIP tag off.
Replying to myself here to address the functional tests situation. I've explored this a bit, and while it's probably doable (it's code, everything is doable), I'm wondering whether functional tests would be worth it. The problem arises from artificially forcing an overlap of the pin mappings. In my integration tests, using CPU pinning as an example, I set vcpu_pin_set to 0,1 on both compute hosts, boot two instances (making sure they're on different hosts by using the DifferentHostFilter and the appropriate scheduler hint); then change vcpu_pin_set to 0-3 on host A, live migrate the instance from host B onto host A, and assert that they don't end up with overlapping pins. Applying the same strategy to functional tests isn't straightforward because the CONF object is very very global, and we can't have different config values for different services in the same test. One basic functional test we could have is just asserting that the live migration is refused if both hosts are "full" with instances - something that currently works just fine, except for resulting in overlapping pin mappings. For more advanced testing, I'm proposing that we shelve functional tests for now and push on setting up some sort of CI job using OpenLab hardware. I've already opened a request [1]. If this doesn't pan out, we can revisit what it would take to have functional tests. Thoughts? [1] https://github.com/theopenlab/openlab/issues/200
What I've been using for testing is this: [3]. It's a series of patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch of us Nova Red Hatters to automate testing that's outside of Tempest's scope. Same idea as the intel-nfv-ci plugin [4]. The tests I currently have check that:
* CPU pin mapping is updated if the destination has an instance pinned to the same CPUs as the incoming instance * emulator thread pins are updated if the destination has a different cpu_shared_set value and the instance has the hw:emulator_threads_policy set to `share` * NUMA node pins are updated for a hugepages instance if the destination has a hugepages instances consuming the same NUMA node as the incoming instance
It's not exhaustive by any means, but I've made sure that all iterations pass those 3 tests. It should be fairly easy to add new tests, as most of the necessary scaffolding is already in place.
[1] https://review.openstack.org/#/c/634606/ [2] https://review.openstack.org/#/c/634828/28/nova/virt/driver.py@1147 [3] https://review.rdoproject.org/r/#/c/18832/ [4] https://github.com/openstack/intel-nfv-ci-tests/
-- -- Artom Lifshitz Software Engineer, OpenStack Compute DFG
On 3/1/2019 9:44 AM, Artom Lifshitz wrote:
Applying the same strategy to functional tests isn't straightforward because the CONF object is very very global, and we can't have different config values for different services in the same test. One basic functional test we could have is just asserting that the live migration is refused if both hosts are "full" with instances - something that currently works just fine, except for resulting in overlapping pin mappings.
Can you run the different compute services with different fakelibvirt connections which report different pins? Seems Stephen and gibi have done some wizardry like that in the libvirt functional tests. -- Thanks, Matt
On Fri, Mar 1, 2019 at 11:00 AM Matt Riedemann <mriedemos@gmail.com> wrote:
On 3/1/2019 9:44 AM, Artom Lifshitz wrote:
Applying the same strategy to functional tests isn't straightforward because the CONF object is very very global, and we can't have different config values for different services in the same test. One basic functional test we could have is just asserting that the live migration is refused if both hosts are "full" with instances - something that currently works just fine, except for resulting in overlapping pin mappings.
Can you run the different compute services with different fakelibvirt connections which report different pins? Seems Stephen and gibi have done some wizardry like that in the libvirt functional tests.
So something like [1], excepting setting side_effect=<list of fake_connection> instead of the return value? I think that's doable. Question is, does it allow us to do what we want - namely, create situations where we can live migrate but expect the pin mappings to be updated? We'd have to bypass fakelibvirt's NUMATopology helper and create the LibvirtConfig classes directly, but we could set it up to have something like this: Host A: NUMA node 0: 4 CPUs NUMA node 1: 2 CPUs Host B: NUMA node 0: 4 CPUs Create an instance with 4 CPUs and 1 NUMA node and live migrate it. Regardless of if it starts on A or B, we expect it's cell pins to be updated. Thanks for being my rubber duck Matt! /me goes off to try that. [1] https://github.com/openstack/nova/blob/a90c8e1a359a236e06f3a78df74f55808bbef...
--
Thanks,
Matt
-- -- Artom Lifshitz Software Engineer, OpenStack Compute DFG
On Fri, 2019-03-01 at 09:58 -0600, Matt Riedemann wrote:
On 3/1/2019 9:44 AM, Artom Lifshitz wrote:
Applying the same strategy to functional tests isn't straightforward because the CONF object is very very global, and we can't have different config values for different services in the same test. One basic functional test we could have is just asserting that the live migration is refused if both hosts are "full" with instances - something that currently works just fine, except for resulting in overlapping pin mappings.
Can you run the different compute services with different fakelibvirt connections which report different pins? Seems Stephen and gibi have done some wizardry like that in the libvirt functional tests. ya i remeber stephen doing somting like that in the past
if both compute nodes had non over lapping cpus in the vcpu_pin_set you could get a similar result by booting just one vm and migrating as it would have to update teh pinnign anyway. i know there have been limitations in the past around using different cofnigs in the fucntional test but i tought we figured out a way to do it.
participants (5)
-
Artom Lifshitz
-
Luyao Zhong
-
Matt Riedemann
-
Sean Mooney
-
Zhong, Luyao