[CI] nested virtualization support in OpenStack CI
Hi, OpenStack CI runs on VM, which is not nested virtualization as: 2019-03-27 06:45:07.282 | flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl cpuid pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes hypervisor lahf_lm pti however, I want to add new CI which depends on nested virtualization. which team can help me? Thanks, -Ruijing
What sort of tests are you looking to run / for what project? Sent from my iPhone
On Mar 27, 2019, at 9:42 PM, Guo, Ruijing <ruijing.guo@intel.com> wrote:
Hi,
OpenStack CI runs on VM, which is not nested virtualization as:
2019-03-27 06:45:07.282 | flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl cpuid pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes hypervisor lahf_lm pti
however, I want to add new CI which depends on nested virtualization. which team can help me?
Thanks, -Ruijing
On 2019-03-28 01:42:24 +0000 (+0000), Guo, Ruijing wrote:
OpenStack CI runs on VM, which is not nested virtualization as:
2019-03-27 06:45:07.282 | flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl cpuid pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes hypervisor lahf_lm pti
however, I want to add new CI which depends on nested virtualization. which team can help me?
Currently the third bullet entry at https://docs.openstack.org/infra/manual/testing.html#known-differences-to-wa... describes the situation with our CI resources. Our CI maintainers lack sufficient insight into the hardware, hypervisors and host kernels of the environments where these virtual machine resources are donated for our use. In particular, until the state of nested virtualization has stabilized long enough for it to become reliable with mixed LTS GNU/Linux distributions and aging servers in many public service providers, it's very hard for us to design for or recommend relying on with upstream testing. We try to coordinate with our donors to troubleshoot the various broken behaviors we observe where possible, but it's far from a trivial exercise. -- Jeremy Stanley
On 2019-03-28 01:42:24 +0000 (+0000), Guo, Ruijing wrote:
OpenStack CI runs on VM, which is not nested virtualization as:
2019-03-27 06:45:07.282 | flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl cpuid pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes hypervisor lahf_lm pti
however, I want to add new CI which depends on nested virtualization. which team can help me?
Currently the third bullet entry at https://docs.openstack.org/infra/manual/testing.html#known-differences-to-wa... describes the situation with our CI resources. Our CI maintainers lack sufficient insight into the hardware, hypervisors and host kernels of the environments where these virtual machine resources are donated for our use. In particular, until the state of nested virtualization has stabilized long enough for it to become reliable with mixed LTS GNU/Linux distributions and aging servers in many public service providers, it's very hard for us to design for or recommend relying on with upstream testing. We try to coordinate with our donors to troubleshoot the various broken behaviors we observe where possible, but it's far from a trivial exercise. so on the stablity of nested virt jobs in general i think we should
On Sat, 2019-03-30 at 16:14 +0000, Jeremy Stanley wrote: try to enable nested virt but perhaps make the jobs non voting in check and excluded nested jobs from gate. that way we get some testing for things we cant other wise test but if we hit kenel bugs its a non voting jobs so if it fails it wont break the zuul jobs.
On 2019-03-31 11:03:34 +0100 (+0100), Sean Mooney wrote: [...]
so on the stablity of nested virt jobs in general i think we should try to enable nested virt but perhaps make the jobs non voting in check and excluded nested jobs from gate. that way we get some testing for things we cant other wise test but if we hit kenel bugs its a non voting jobs so if it fails it wont break the zuul jobs.
Well, having these jobs non-voting means you aren't relying on them to protect you from merging changes which might break nested virtualization or fail to work with it. But that aside, to even have a guarantee it would be available at all we'd need a separate node label for all our images which is restricted to providers where we can expect it. For example, I've heard that doing nested virt acceleration for KVM guests in our Xen-based providers is a long way from being possible. -- Jeremy Stanley
I like Sean's idea. Can we create jobs based on nested KVM & customized OS? If CI is stable enough, we can move non-voting to voting. -----Original Message----- From: Jeremy Stanley [mailto:fungi@yuggoth.org] Sent: Sunday, March 31, 2019 8:43 PM To: openstack-dev@lists.openstack.org Subject: Re: [CI] nested virtualization support in OpenStack CI On 2019-03-31 11:03:34 +0100 (+0100), Sean Mooney wrote: [...]
so on the stablity of nested virt jobs in general i think we should try to enable nested virt but perhaps make the jobs non voting in check and excluded nested jobs from gate. that way we get some testing for things we cant other wise test but if we hit kenel bugs its a non voting jobs so if it fails it wont break the zuul jobs.
Well, having these jobs non-voting means you aren't relying on them to protect you from merging changes which might break nested virtualization or fail to work with it. But that aside, to even have a guarantee it would be available at all we'd need a separate node label for all our images which is restricted to providers where we can expect it. For example, I've heard that doing nested virt acceleration for KVM guests in our Xen-based providers is a long way from being possible. -- Jeremy Stanley
On 2019-04-08 00:42:46 +0000 (+0000), Guo, Ruijing wrote:
I like Sean's idea. Can we create jobs based on nested KVM & customized OS? If CI is stable enough, we can move non-voting to voting. [...]
There's nothing stopping you (or anyone) from creating such a CI system to augment reporting on how specific workloads fair with certain nested virt implementations. The OpenDev CI system on the other hand is made of a heterogenous mix of donations from a variety of service providers with various configurations, and even some of those which advertise via CPU flags that they offer nested virt acceleration don't provide a stable implementation of it. There are already projects opportunistically testing with nested virtualization support turned on in their jobs to varying degrees of success, so it would be good to find out what problems their recent experiences reveal. It's also very much the case that even if it's working for a job today, it can start crashing jobs randomly tomorrow when a kernel version changes on guests or on hosts or we add another service provider into the mix. -- Jeremy Stanley
On Mon, Apr 8, 2019, at 5:21 AM, Jeremy Stanley wrote:
On 2019-04-08 00:42:46 +0000 (+0000), Guo, Ruijing wrote:
I like Sean's idea. Can we create jobs based on nested KVM & customized OS? If CI is stable enough, we can move non-voting to voting. [...]
There's nothing stopping you (or anyone) from creating such a CI system to augment reporting on how specific workloads fair with certain nested virt implementations.
The OpenDev CI system on the other hand is made of a heterogenous mix of donations from a variety of service providers with various configurations, and even some of those which advertise via CPU flags that they offer nested virt acceleration don't provide a stable implementation of it. There are already projects opportunistically testing with nested virtualization support turned on in their jobs to varying degrees of success, so it would be good to find out what problems their recent experiences reveal. It's also very much the case that even if it's working for a job today, it can start crashing jobs randomly tomorrow when a kernel version changes on guests or on hosts or we add another service provider into the mix.
Right one of the biggest concerns we have is the lack of observed stability with nested virt. This isn't a theoretical concern. In the not too distant past Tripleo jobs running on CentOS were crashing the "middle" VM and no one noticed until I started looking into job failures. The fix for that required a base hypervisor kernel update which is not something we control. That said understanding requirements here (which was asked for earlier in the thread, but I don't see a response) is useful when trying to figure out the best way to approach these situations. It is possible there are other reasonable avenues that could be taken. Clark
Hi, Clark I am trying to add ovsdpdk ci job in neutron. Ovsdpdk need nested virtualization. Thanks, -Ruijing -----Original Message----- From: Clark Boylan [mailto:cboylan@sapwetik.org] Sent: Tuesday, April 9, 2019 12:04 AM To: openstack-discuss@lists.openstack.org Subject: Re: [CI] nested virtualization support in OpenStack CI On Mon, Apr 8, 2019, at 5:21 AM, Jeremy Stanley wrote:
On 2019-04-08 00:42:46 +0000 (+0000), Guo, Ruijing wrote:
I like Sean's idea. Can we create jobs based on nested KVM & customized OS? If CI is stable enough, we can move non-voting to voting. [...]
There's nothing stopping you (or anyone) from creating such a CI system to augment reporting on how specific workloads fair with certain nested virt implementations.
The OpenDev CI system on the other hand is made of a heterogenous mix of donations from a variety of service providers with various configurations, and even some of those which advertise via CPU flags that they offer nested virt acceleration don't provide a stable implementation of it. There are already projects opportunistically testing with nested virtualization support turned on in their jobs to varying degrees of success, so it would be good to find out what problems their recent experiences reveal. It's also very much the case that even if it's working for a job today, it can start crashing jobs randomly tomorrow when a kernel version changes on guests or on hosts or we add another service provider into the mix.
Right one of the biggest concerns we have is the lack of observed stability with nested virt. This isn't a theoretical concern. In the not too distant past Tripleo jobs running on CentOS were crashing the "middle" VM and no one noticed until I started looking into job failures. The fix for that required a base hypervisor kernel update which is not something we control. That said understanding requirements here (which was asked for earlier in the thread, but I don't see a response) is useful when trying to figure out the best way to approach these situations. It is possible there are other reasonable avenues that could be taken. Clark
On Mon, Apr 8, 2019, at 3:54 PM, Guo, Ruijing wrote:
Hi, Clark
I am trying to add ovsdpdk ci job in neutron. Ovsdpdk need nested virtualization.
Can you describe this requirement in a bit more detail? Looking at docs it appears you need to have large pages. Maybe a "picture" will help too: Workload VM (cirros or similar small VM running under emulation or virtualized if nested virt is enabled) | V OpenStack Infra Test Node ("middle VM") This is where ovsdpdk runs right? And this requires large pages? | V Hypervisor (managed by our cloud providers) Are we not able to do large pages with emulated workload VMs? Also can we do large pages on the Workload VM if the Infra Test Node isn't configured for them? If the infra test node isn't configured for them do we also need support for large pages from the Hypervisor? If we can do large pages with emulated workload VMs and the hypervisor isn't required to support that we should be able to configure large pages on the Infra Test Node and boot emulated guests, but my guess is that at least one of these things won't work. In any case specific details on what is required and where it is required will be useful. Clark
Hi, Clark Workload VM (cirros or similar small VM running), (don't need large page, memory might be 512M) | V OpenStack Infra Test Node ("middle VM"), (run ovsdpdk here and need 2M huge pages, total memory might be 2G) | V Hypervisor (need to enable nested KVM, don't need large pages) Thanks, -Ruijing -----Original Message----- From: Clark Boylan [mailto:cboylan@sapwetik.org] Sent: Tuesday, April 9, 2019 7:05 AM To: openstack-discuss@lists.openstack.org Subject: Re: [CI] nested virtualization support in OpenStack CI On Mon, Apr 8, 2019, at 3:54 PM, Guo, Ruijing wrote:
Hi, Clark
I am trying to add ovsdpdk ci job in neutron. Ovsdpdk need nested virtualization.
Can you describe this requirement in a bit more detail? Looking at docs it appears you need to have large pages. Maybe a "picture" will help too: Workload VM (cirros or similar small VM running under emulation or virtualized if nested virt is enabled) | V OpenStack Infra Test Node ("middle VM") This is where ovsdpdk runs right? And this requires large pages? | V Hypervisor (managed by our cloud providers) Are we not able to do large pages with emulated workload VMs? Also can we do large pages on the Workload VM if the Infra Test Node isn't configured for them? If the infra test node isn't configured for them do we also need support for large pages from the Hypervisor? If we can do large pages with emulated workload VMs and the hypervisor isn't required to support that we should be able to configure large pages on the Infra Test Node and boot emulated guests, but my guess is that at least one of these things won't work. In any case specific details on what is required and where it is required will be useful. Clark
On Mon, 2019-04-08 at 19:05 -0400, Clark Boylan wrote:
On Mon, Apr 8, 2019, at 3:54 PM, Guo, Ruijing wrote:
Hi, Clark
I am trying to add ovsdpdk ci job in neutron. Ovsdpdk need nested virtualization.
Can you describe this requirement in a bit more detail? Looking at docs it appears you need to have large pages. Maybe a "picture" will help too:
Workload VM (cirros or similar small VM running under emulation or virtualized if nested virt is enabled) | V OpenStack Infra Test Node ("middle VM") This is where ovsdpdk runs right? And this requires large pages?
so ovs-dpdk requires hugepage memeory to be allocated to fucntion but that by its self does not require nested virt. nova on the other hand does require kvm to allow vms to be spawned with hugepages because not enabled partial cpu pinning when hugepages are enabled and qemu does not support per cpu cpu pinning without kvm or the mttcg backend. in the special case of ovs-dpdk i have been meaning to test an alternitive way for deploying with ovs-dpdk but to date i have not had time. about 2 years ago i had a memeber of my team extend libvirt to allow the creation of qemu guests with file backed memory. about a year ago redhat extended nova to support that. it should technically be possibel to run ovs-dpdk with hugepages and the cirros guests with file backed memory to do testing without nested virt. related to this i finished setting up my own personal ci to do limited ovs-dpdk and numa/nfv testing with nested virt. https://review.openstack.org/#/c/650573/ tempest-dpdk https://logs.seanmooney.info/73/650573/1/seans-nfv-ci/tempest-dpdk/fec9044/ tempest-pinning https://logs.seanmooney.info/73/650573/1/seans-nfv-ci/tempest-pinning/193997... i am planning to exprement to see if i can run some of those nested virt jobs upstream or via openlab in the furutre. it shoudl be possible to test with ovn too but i have not tried that pesonally yet the job definitions currently live in https://review.seanmooney.info/plugins/gitiles/test/+/master so perhaps that ill server as a starting point for people wanting to create similar jobs the networking-ovs-dpdk devstack plugin compiles and deploys ovs-dpdk form source which also installs but does not configure or start ovn since its part of the ovs repo. with some tweeks to the networking-ovn plugin it would probably be simple enough to use both plugins to provide the desired versions of ovn,ovs and dpdk. with all that said the intel nvf ci was created to test ovs-dpdk integration with openstack and also some other nfv features like cpu pinning. https://wiki.openstack.org/w/index.php?title=ThirdPartySystems/Intel_NFV_CI this ci was also intended to test odl and ovn with ovs dpdk at somepoint in the future alther that never happend. i woudl suggest reaching out to <timothy.gresham@intel.com>, <openstack-nfv-ci@intel.com> to see if it could be extended to do ovn testing in the absence of upstream support.
| V Hypervisor (managed by our cloud providers)
Are we not able to do large pages with emulated workload VMs? Also can we do large pages on the Workload VM if the Infra Test Node isn't configured for them? If the infra test node isn't configured for them do we also need support for large pages from the Hypervisor?
If we can do large pages with emulated workload VMs and the hypervisor isn't required to support that we should be able to configure large pages on the Infra Test Node and boot emulated guests, but my guess is that at least one of these things won't work. In any case specific details on what is required and where it is required will be useful.
Clark
On Tue, Apr 9, 2019 at 8:05 AM Sean Mooney <smooney@redhat.com> wrote:
with all that said the intel nvf ci was created to test ovs-dpdk integration with openstack and also some other nfv features like cpu pinning. https://wiki.openstack.org/w/index.php?title=ThirdPartySystems/Intel_NFV_CI this ci was also intended to test odl and ovn with ovs dpdk at somepoint in the future alther that never happend. i woudl suggest reaching out to <timothy.gresham@intel.com>, <openstack-nfv-ci@intel.com> to see if it could be extended to do ovn testing in the absence of upstream support.
The Intel 3rd Party CI has been rebuilt from the ground-up using Zuul v3 and the rest of the tooling we all know and love, it is currently in the sort of final internal review where we can not predict the timing, hence my reluctance to make promises of being ready by X date, but it is "soon". These are among the test cases we intend to resurrect, first priority being the testing the old CI setups had been doing. dt -- Dean Troyer dtroyer@gmail.com
Right. I am planning to add ovsdpdk in networking-ovn like networking-ovn-tempest-dsvm-ovs-master. We can leverage networking-ovsdpdk to build ovsdpdk packages. The only issue is that qemu without kvm can be crash in vhostuser interface. So I want to have nested virtualization. -----Original Message----- From: Sean Mooney [mailto:smooney@redhat.com] Sent: Tuesday, April 9, 2019 9:05 PM To: Clark Boylan <cboylan@sapwetik.org>; openstack-discuss@lists.openstack.org; Guo, Ruijing <ruijing.guo@intel.com> Subject: Re: [CI] nested virtualization support in OpenStack CI On Mon, 2019-04-08 at 19:05 -0400, Clark Boylan wrote:
On Mon, Apr 8, 2019, at 3:54 PM, Guo, Ruijing wrote:
Hi, Clark
I am trying to add ovsdpdk ci job in neutron. Ovsdpdk need nested virtualization.
Can you describe this requirement in a bit more detail? Looking at docs it appears you need to have large pages. Maybe a "picture" will help too:
Workload VM (cirros or similar small VM running under emulation or virtualized if nested virt is enabled) | V OpenStack Infra Test Node ("middle VM") This is where ovsdpdk runs right? And this requires large pages?
so ovs-dpdk requires hugepage memeory to be allocated to fucntion but that by its self does not require nested virt. nova on the other hand does require kvm to allow vms to be spawned with hugepages because not enabled partial cpu pinning when hugepages are enabled and qemu does not support per cpu cpu pinning without kvm or the mttcg backend. in the special case of ovs-dpdk i have been meaning to test an alternitive way for deploying with ovs-dpdk but to date i have not had time. about 2 years ago i had a memeber of my team extend libvirt to allow the creation of qemu guests with file backed memory. about a year ago redhat extended nova to support that. it should technically be possibel to run ovs-dpdk with hugepages and the cirros guests with file backed memory to do testing without nested virt. related to this i finished setting up my own personal ci to do limited ovs-dpdk and numa/nfv testing with nested virt. https://review.openstack.org/#/c/650573/ tempest-dpdk https://logs.seanmooney.info/73/650573/1/seans-nfv-ci/tempest-dpdk/fec9044/ tempest-pinning https://logs.seanmooney.info/73/650573/1/seans-nfv-ci/tempest-pinning/193997... i am planning to exprement to see if i can run some of those nested virt jobs upstream or via openlab in the furutre. it shoudl be possible to test with ovn too but i have not tried that pesonally yet the job definitions currently live in https://review.seanmooney.info/plugins/gitiles/test/+/master so perhaps that ill server as a starting point for people wanting to create similar jobs the networking-ovs-dpdk devstack plugin compiles and deploys ovs-dpdk form source which also installs but does not configure or start ovn since its part of the ovs repo. with some tweeks to the networking-ovn plugin it would probably be simple enough to use both plugins to provide the desired versions of ovn,ovs and dpdk. with all that said the intel nvf ci was created to test ovs-dpdk integration with openstack and also some other nfv features like cpu pinning. https://wiki.openstack.org/w/index.php?title=ThirdPartySystems/Intel_NFV_CI this ci was also intended to test odl and ovn with ovs dpdk at somepoint in the future alther that never happend. i woudl suggest reaching out to <timothy.gresham@intel.com>, <openstack-nfv-ci@intel.com> to see if it could be extended to do ovn testing in the absence of upstream support.
| V Hypervisor (managed by our cloud providers)
Are we not able to do large pages with emulated workload VMs? Also can we do large pages on the Workload VM if the Infra Test Node isn't configured for them? If the infra test node isn't configured for them do we also need support for large pages from the Hypervisor?
If we can do large pages with emulated workload VMs and the hypervisor isn't required to support that we should be able to configure large pages on the Infra Test Node and boot emulated guests, but my guess is that at least one of these things won't work. In any case specific details on what is required and where it is required will be useful.
Clark
On Wed, 2019-04-10 at 01:24 +0000, Guo, Ruijing wrote:
Right. I am planning to add ovsdpdk in networking-ovn like networking-ovn-tempest-dsvm-ovs-master. We can leverage networking-ovsdpdk to build ovsdpdk packages.
The only issue is that qemu without kvm can be crash in vhostuser interface. So I want to have nested virtualization. well dpdk only need preallocated mapped shared file descirptor backed memoery. as i siad i belive you can use the nova file_backed memory feature to achive that but i have not tested it. im also tinking of filing a nova spec to enable the qemu mttcg backend which woudl allow us to do testing with hugepages and cpu pinning.
ovs-dpdk does not need kvm and nested virt. the only requirement on nested virt come form how nova does hugepage supprot.
-----Original Message----- From: Sean Mooney [mailto:smooney@redhat.com] Sent: Tuesday, April 9, 2019 9:05 PM To: Clark Boylan <cboylan@sapwetik.org>; openstack-discuss@lists.openstack.org; Guo, Ruijing <ruijing.guo@intel.com> Subject: Re: [CI] nested virtualization support in OpenStack CI
On Mon, 2019-04-08 at 19:05 -0400, Clark Boylan wrote:
On Mon, Apr 8, 2019, at 3:54 PM, Guo, Ruijing wrote:
Hi, Clark
I am trying to add ovsdpdk ci job in neutron. Ovsdpdk need nested virtualization.
Can you describe this requirement in a bit more detail? Looking at docs it appears you need to have large pages. Maybe a "picture" will help too:
Workload VM (cirros or similar small VM running under emulation or virtualized if nested virt is enabled) | V OpenStack Infra Test Node ("middle VM") This is where ovsdpdk runs right? And this requires large pages?
so ovs-dpdk requires hugepage memeory to be allocated to fucntion but that by its self does not require nested virt. nova on the other hand does require kvm to allow vms to be spawned with hugepages because not enabled partial cpu pinning when hugepages are enabled and qemu does not support per cpu cpu pinning without kvm or the mttcg backend.
in the special case of ovs-dpdk i have been meaning to test an alternitive way for deploying with ovs-dpdk but to date i have not had time. about 2 years ago i had a memeber of my team extend libvirt to allow the creation of qemu guests with file backed memory. about a year ago redhat extended nova to support that. it should technically be possibel to run ovs-dpdk with hugepages and the cirros guests with file backed memory to do testing without nested virt.
related to this i finished setting up my own personal ci to do limited ovs-dpdk and numa/nfv testing with nested virt. https://review.openstack.org/#/c/650573/ tempest-dpdk https://logs.seanmooney.info/73/650573/1/seans-nfv-ci/tempest-dpdk/fec9044/ tempest-pinning https://logs.seanmooney.info/73/650573/1/seans-nfv-ci/tempest-pinning/193997...
i am planning to exprement to see if i can run some of those nested virt jobs upstream or via openlab in the furutre.
it shoudl be possible to test with ovn too but i have not tried that pesonally yet the job definitions currently live in https://review.seanmooney.info/plugins/gitiles/test/+/master so perhaps that ill server as a starting point for people wanting to create similar jobs
the networking-ovs-dpdk devstack plugin compiles and deploys ovs-dpdk form source which also installs but does not configure or start ovn since its part of the ovs repo.
with some tweeks to the networking-ovn plugin it would probably be simple enough to use both plugins to provide the desired versions of ovn,ovs and dpdk.
with all that said the intel nvf ci was created to test ovs-dpdk integration with openstack and also some other nfv features like cpu pinning. https://wiki.openstack.org/w/index.php?title=ThirdPartySystems/Intel_NFV_CI this ci was also intended to test odl and ovn with ovs dpdk at somepoint in the future alther that never happend. i woudl suggest reaching out to <timothy.gresham@intel.com>, <openstack-nfv-ci@intel.com> to see if it could be extended to do ovn testing in the absence of upstream support.
| V Hypervisor (managed by our cloud providers)
Are we not able to do large pages with emulated workload VMs? Also can we do large pages on the Workload VM if the Infra Test Node isn't configured for them? If the infra test node isn't configured for them do we also need support for large pages from the Hypervisor?
If we can do large pages with emulated workload VMs and the hypervisor isn't required to support that we should be able to configure large pages on the Infra Test Node and boot emulated guests, but my guess is that at least one of these things won't work. In any case specific details on what is required and where it is required will be useful.
Clark
Hi, Clark Do you have some guide to create CI based on nested virt? We can try it and see how much gap & stable? Thanks, -Ruijing -----Original Message----- From: Clark Boylan [mailto:cboylan@sapwetik.org] Sent: Tuesday, April 9, 2019 7:05 AM To: openstack-discuss@lists.openstack.org Subject: Re: [CI] nested virtualization support in OpenStack CI On Mon, Apr 8, 2019, at 3:54 PM, Guo, Ruijing wrote:
Hi, Clark
I am trying to add ovsdpdk ci job in neutron. Ovsdpdk need nested virtualization.
Can you describe this requirement in a bit more detail? Looking at docs it appears you need to have large pages. Maybe a "picture" will help too: Workload VM (cirros or similar small VM running under emulation or virtualized if nested virt is enabled) | V OpenStack Infra Test Node ("middle VM") This is where ovsdpdk runs right? And this requires large pages? | V Hypervisor (managed by our cloud providers) Are we not able to do large pages with emulated workload VMs? Also can we do large pages on the Workload VM if the Infra Test Node isn't configured for them? If the infra test node isn't configured for them do we also need support for large pages from the Hypervisor? If we can do large pages with emulated workload VMs and the hypervisor isn't required to support that we should be able to configure large pages on the Infra Test Node and boot emulated guests, but my guess is that at least one of these things won't work. In any case specific details on what is required and where it is required will be useful. Clark
Hi, Clark
Do you have some guide to create CI based on nested virt? We can try it and see how much gap & stable?
On Fri, 2019-04-12 at 01:23 +0000, Guo, Ruijing wrote: the intel nfv ci is an example of this. assuming you ment how do i creat a third party ci job this is the basic steps. in my experience it more or less jsut works however the kernel that ships in ubuntu 18.04 is broken the ubuntu 16.04 kernel works fine as does the centos 7 kernel if you build it on ubunut install ukuu and use it to install the unpatched mainline 5.0 kernel. the kernel in the guest is less important then getting the kernel on the host correct. on the host you obviorly need to also enable kvm nested virt in the kernel module parmaters. sean@cloud-5:~$ cat /sys/module/kvm_intel/parameters/nested Y i belive its on by default as of one of the latest kernel its definetly on by default in 5.0 i enable nested virt via the kernel command line sean@cloud-5:~$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-5.0.0-050000-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro quiet intel_iommu=on iommu=pt console=tty0 serial console=ttyS0,115200n8 hugepagesz=2M hugepages=51200 irqaffinity=0,12,24,36 kvm-intel.nested=1 nfs.enable_ino64=1 nohz_full=1-11,13-23,25-35,37-47 numa_balancing=enable rcu_nocbs=1-11,13-23,25-35,37-47 rcu_nocb_poll vsyscall=native the rest is the same as setting up any other ci. if you ment is the a guide to setting up a job that used nested virt instead of how to create a nested virt ci you can ignore the above and its reltvly trivial. you just need to set the libvirt virt dirver to kvm in you job config the issue is on most vms upstream that will cause the build to fail as they dont have nested virt enableded on the host. there are a small set of nodeset that do have nested virt but for now im currently exploing alternitive to nested virt for ovsdpdk testing. i have 2 paths that i am currently looking at. first the mttcg qemu backend with libvirt 5.2 via a fedora based job and second qemu with file based memory. if they prove unviable ill switch my efforts back to nested virt either via openlab or reaching out to the ci providers that currnetly provied nested virt vms to the gate and asking there permission to use there nodesets for the testing. regards sean
Thanks, -Ruijing
-----Original Message----- From: Clark Boylan [mailto:cboylan@sapwetik.org] Sent: Tuesday, April 9, 2019 7:05 AM To: openstack-discuss@lists.openstack.org Subject: Re: [CI] nested virtualization support in OpenStack CI
On Mon, Apr 8, 2019, at 3:54 PM, Guo, Ruijing wrote:
Hi, Clark
I am trying to add ovsdpdk ci job in neutron. Ovsdpdk need nested virtualization.
Can you describe this requirement in a bit more detail? Looking at docs it appears you need to have large pages. Maybe a "picture" will help too:
Workload VM (cirros or similar small VM running under emulation or virtualized if nested virt is enabled) | V OpenStack Infra Test Node ("middle VM") This is where ovsdpdk runs right? And this requires large pages? | V Hypervisor (managed by our cloud providers)
Are we not able to do large pages with emulated workload VMs? Also can we do large pages on the Workload VM if the Infra Test Node isn't configured for them? If the infra test node isn't configured for them do we also need support for large pages from the Hypervisor?
If we can do large pages with emulated workload VMs and the hypervisor isn't required to support that we should be able to configure large pages on the Infra Test Node and boot emulated guests, but my guess is that at least one of these things won't work. In any case specific details on what is required and where it is required will be useful.
Clark
hi just a quick update. i have created an work in progress nfv job definition that works without nested virt in this patch. https://review.openstack.org/#/c/652197/ i will be created a similar tempest-dpdk job before the ptg. but form my testign using libvirt 5.2 we can now utilise the qemu MTTCG backend to do hugepage,emulator thread realtime and cpu pinning in the upstream gate. i have enabled all of the above in that test job. ovs-dpdk just needs hugepages to work so ill port my thridparty ovs-dpdk job to the networking-ovs-dpdk repo in the next week or so and networking-ovn will be able to inhirti form it or create there own using it as a template. the main draw back to this approch is that fedroa 29 with the virt preview repo is the only way to do this in the gate unless we compile libvirt and qemu from scratch. we have devstack plugins that can do that if we wanted to create a ubunut/centos based job but form what i can tell the only feature we cant test that was previously tested by the intel nfv ci using this setup would be multiple guest numa nodes. multi numa node guest can only be created by multi numa node host due to restictions we have in nova. if we want to test that we either need to remvoe that restiction (my perference) or we need to ask the operators the provide our ci resouce if they would be willing to kindly create a private flavor that had 2 numa nodes. .e.g hw:numa_nodes=2 regards sean.
Thanks, Jeremy. Could you pls guide me how to create job based on nested KVM and customized OS? -----Original Message----- From: Jeremy Stanley [mailto:fungi@yuggoth.org] Sent: Monday, April 8, 2019 8:21 PM To: openstack-discuss@lists.openstack.org Subject: Re: [CI] nested virtualization support in OpenStack CI On 2019-04-08 00:42:46 +0000 (+0000), Guo, Ruijing wrote:
I like Sean's idea. Can we create jobs based on nested KVM & customized OS? If CI is stable enough, we can move non-voting to voting. [...]
There's nothing stopping you (or anyone) from creating such a CI system to augment reporting on how specific workloads fair with certain nested virt implementations. The OpenDev CI system on the other hand is made of a heterogenous mix of donations from a variety of service providers with various configurations, and even some of those which advertise via CPU flags that they offer nested virt acceleration don't provide a stable implementation of it. There are already projects opportunistically testing with nested virtualization support turned on in their jobs to varying degrees of success, so it would be good to find out what problems their recent experiences reveal. It's also very much the case that even if it's working for a job today, it can start crashing jobs randomly tomorrow when a kernel version changes on guests or on hosts or we add another service provider into the mix. -- Jeremy Stanley
On 2019-04-08 22:50:14 +0000 (+0000), Guo, Ruijing wrote:
Thanks, Jeremy. Could you pls guide me how to create job based on nested KVM and customized OS? [...]
The message to which you're replying (context elided once again because you keep top-posting) suggested you could do so if you build your own CI system. That aside, as I mentioned some projects (I believe Octavia is one?) turn on nested virtualization support in their jobs if they see the system on which they're running advertises support for it. We don't expressly disable use of nested virtualization in the OpenDev CI system, it's actually DevStack which does so in jobs which use it, and that behavior can presumably be overridden. -- Jeremy Stanley
participants (6)
-
Clark Boylan
-
Dean Troyer
-
Guo, Ruijing
-
Jeremy Stanley
-
Mohammed Naser
-
Sean Mooney