[Neutron][Nova][Ironic][Cinder][Keystone][Glance][Swift] OVN as the default network backend for DevStack
Hi all, As part of the Victoria PTG [0] the Neutron community agreed upon switching the default backend in Devstack to OVN. A lot of work has been done since, from porting the OVN devstack module to the DevStack tree, refactoring the DevStack module to install OVN from distro packages, implementing features to close the parity gap with ML2/OVS, fixing issues with tests and distros, etc... We are now very close to being able to make the switch and we've thought about sending this email to the broader community to raise awareness about this change as well as bring more attention to the patches that are current on review. Note that moving DevStack to ML2/OVN does not mean that ML2/OVS is discontinued and/or not supported anymore. The ML2/OVS driver is still going to be developed and maintained by the upstream Neutron community. Below is a e per project explanation with relevant links and issues of where we stand with this work right now: * Keystone: Everything should be good for Keystone, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/keystone/+/777963 * Glance: Everything should be good for Glace, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/glance/+/748390 * Swift: Everything should be good for Swift, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/swift/+/748403 * Ironic: Since chainloading iPXE by the OVN built-in DHCP server is work in progress, we've changed most of the Ironic jobs to explicitly enable ML2/OVS and everything is merged, so we should be good for Ironic too. Here is the test patch: https://review.opendev.org/c/openstack/ironic/+/748405 * Cinder: Cinder is almost complete. There's one test failure in the "tempest-slow-py3" job run on the "test_port_security_macspoofing_port" test. This failure is due to a bug in core OVN [1]. This bug has already been fixed upstream [2] and the fix has been backported down to the branch-20.03 [3] of the OVN project. However, since we install OVN from packages we are currently waiting for this fix to be included in the packages for Ubuntu Focal (it's based on OVN 20.03). I already contacted the package maintainer which has been very supportive of this work and will work on the package update, but he maintain a handful of backports in that package which is not yet included in OVN 20.03 upstream and he's now working with the core OVN community [4] to include it first in the branch and then create a new package for it. Hopefully this will happen soon. But for now we have a few options moving on with this issue: 1- Wait for the new package version 2- Mark the test as unstable until we get the new package version 3- Compile OVN from source instead of installing it from packages (OVN_BUILD_FROM_SOURCE=True in local.conf) What do you think about it ? Here is the test patch for Cinder: https://review.opendev.org/c/openstack/cinder/+/748227 * Nova: There are a few patches waiting for review for Nova, which are: 1- Adapting the live migration scripts to work with ML2/OVN: Basically the scripts were trying to stop the Neutron agent (q-agt) process which is not part of an ML2/OVN deployment. The patch changes the code to check if that system unit exists before trying to stop it. Patch: https://review.opendev.org/c/openstack/nova/+/776419 2- Explicitly set grenade job to ML2/OVS: This is a temporary change which can be removed one release cycle after we switch DevStack to ML2/OVN. Grenade will test updating from the release version to the master branch but, since the default of the released version is not ML2/OVN, upgrading from ML2/OVS to ML2/OVN as part of the grenade job is not supported. Patch: https://review.opendev.org/c/openstack/nova/+/776934 3- Explicitly set nova-next job to ML2/OVS: This job uses the QoS minimum bandwidth feature which is not yet supported by ML2/OVN [5][6] therefore we are temporarily enabling ML2/OVS for this job until that feature lands in core OVN. Patch: https://review.opendev.org/c/openstack/nova/+/776944 I also spoke briefly with Sean Mooney (irc: sean-k-mooney) about these changes and he suggested keeping all the Nova jobs on ML2/OVS for now because he feels like a change in the default network driver a few weeks prior to the upstream code freeze can be concerning. We do not know yet precisely when we are changing the default due to the current patches we need to get merged but, if this is a shared feeling among the Nova community I can work on enabling ML2/OVS on all jobs in Nova until we get a new release in OpenStack. Here's the test patch for Nova: https://review.opendev.org/c/openstack/nova/+/776945 * DevStack: And this is the final patch that will make this all happen: https://review.opendev.org/c/openstack/devstack/+/735097 It changes the default in DevStack from ML2/OVS to ML2/OVN. It's been a long and bumpy road to get to this point and I would like to say thanks to everyone involved so far and everyone that read the whole email, please let me know your thoughts. [0] https://etherpad.opendev.org/p/neutron-victoria-ptg [1] https://bugs.launchpad.net/tempest/+bug/1728886 [2] https://patchwork.ozlabs.org/project/openvswitch/patch/20200319122641.473776... [3] https://github.com/ovn-org/ovn/commit/0c26bc03064f2c21d208f0f860b48d8ab39380... [4] https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050961.html [5] https://github.com/openstack/nova/blob/ded25f33c734ebff963f06984707a99fe76a9... [6] https://docs.openstack.org/neutron/latest/ovn/gaps.html Cheers, Lucas
On Mon, 2021-03-01 at 16:07 +0000, Lucas Alvares Gomes wrote:
Hi all,
As part of the Victoria PTG [0] the Neutron community agreed upon switching the default backend in Devstack to OVN. A lot of work has been done since, from porting the OVN devstack module to the DevStack tree, refactoring the DevStack module to install OVN from distro packages, implementing features to close the parity gap with ML2/OVS, fixing issues with tests and distros, etc...
We are now very close to being able to make the switch and we've thought about sending this email to the broader community to raise awareness about this change as well as bring more attention to the patches that are current on review.
Note that moving DevStack to ML2/OVN does not mean that ML2/OVS is discontinued and/or not supported anymore. The ML2/OVS driver is still going to be developed and maintained by the upstream Neutron community. can we ensure that this does not happen until the xena release. in generall i think its ok to change the default but not this late in the cycle. i would also like to ensure we keep at least one non ovn based multi node job in nova until https://review.opendev.org/c/openstack/nova/+/602432 is merged and possible after. right now the event/neutorn interaction is not the same during move operations.
Below is a e per project explanation with relevant links and issues of where we stand with this work right now:
* Keystone:
Everything should be good for Keystone, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/keystone/+/777963
* Glance:
Everything should be good for Glace, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/glance/+/748390
* Swift:
Everything should be good for Swift, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/swift/+/748403
* Ironic:
Since chainloading iPXE by the OVN built-in DHCP server is work in progress, we've changed most of the Ironic jobs to explicitly enable ML2/OVS and everything is merged, so we should be good for Ironic too. Here is the test patch: https://review.opendev.org/c/openstack/ironic/+/748405
* Cinder:
Cinder is almost complete. There's one test failure in the "tempest-slow-py3" job run on the "test_port_security_macspoofing_port" test.
This failure is due to a bug in core OVN [1]. This bug has already been fixed upstream [2] and the fix has been backported down to the branch-20.03 [3] of the OVN project. However, since we install OVN from packages we are currently waiting for this fix to be included in the packages for Ubuntu Focal (it's based on OVN 20.03). I already contacted the package maintainer which has been very supportive of this work and will work on the package update, but he maintain a handful of backports in that package which is not yet included in OVN 20.03 upstream and he's now working with the core OVN community [4] to include it first in the branch and then create a new package for it. Hopefully this will happen soon.
But for now we have a few options moving on with this issue:
1- Wait for the new package version 2- Mark the test as unstable until we get the new package version 3- Compile OVN from source instead of installing it from packages (OVN_BUILD_FROM_SOURCE=True in local.conf) i dont think we should default to ovn untill a souce build is not required. compiling form souce while not supper expensice still adds time to the job execution and im not sure we should be paying that cost on every devstack job run.
we could maybe compile it once and bake the package into the image or host it on a mirror but i think we should avoid this option if we have alternitives.
What do you think about it ?
Here is the test patch for Cinder: https://review.opendev.org/c/openstack/cinder/+/748227
* Nova:
There are a few patches waiting for review for Nova, which are:
1- Adapting the live migration scripts to work with ML2/OVN: Basically the scripts were trying to stop the Neutron agent (q-agt) process which is not part of an ML2/OVN deployment. The patch changes the code to check if that system unit exists before trying to stop it.
Patch: https://review.opendev.org/c/openstack/nova/+/776419
2- Explicitly set grenade job to ML2/OVS: This is a temporary change which can be removed one release cycle after we switch DevStack to ML2/OVN. Grenade will test updating from the release version to the master branch but, since the default of the released version is not ML2/OVN, upgrading from ML2/OVS to ML2/OVN as part of the grenade job is not supported.
Patch: https://review.opendev.org/c/openstack/nova/+/776934
3- Explicitly set nova-next job to ML2/OVS: This job uses the QoS minimum bandwidth feature which is not yet supported by ML2/OVN [5][6] therefore we are temporarily enabling ML2/OVS for this job until that feature lands in core OVN.
Patch: https://review.opendev.org/c/openstack/nova/+/776944
I also spoke briefly with Sean Mooney (irc: sean-k-mooney) about these changes and he suggested keeping all the Nova jobs on ML2/OVS for now because he feels like a change in the default network driver a few weeks prior to the upstream code freeze can be concerning. We do not know yet precisely when we are changing the default due to the current patches we need to get merged but, if this is a shared feeling among the Nova community I can work on enabling ML2/OVS on all jobs in Nova until we get a new release in OpenStack.
yep this is still my view. i would suggest we do the work required in the repos but not merge it until the xena release is open. thats technically at RC1 so march 25th i think we can safely do the swich after that but i would not change the defualt in any project before then.
Here's the test patch for Nova: https://review.opendev.org/c/openstack/nova/+/776945
* DevStack:
And this is the final patch that will make this all happen: https://review.opendev.org/c/openstack/devstack/+/735097
It changes the default in DevStack from ML2/OVS to ML2/OVN. It's been a long and bumpy road to get to this point and I would like to say thanks to everyone involved so far and everyone that read the whole email, please let me know your thoughts.
thanks for working on this.
[0] https://etherpad.opendev.org/p/neutron-victoria-ptg [1] https://bugs.launchpad.net/tempest/+bug/1728886 [2] https://patchwork.ozlabs.org/project/openvswitch/patch/20200319122641.473776... [3] https://github.com/ovn-org/ovn/commit/0c26bc03064f2c21d208f0f860b48d8ab39380... [4] https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050961.html [5] https://github.com/openstack/nova/blob/ded25f33c734ebff963f06984707a99fe76a9... [6] https://docs.openstack.org/neutron/latest/ovn/gaps.html
Cheers, Lucas
On Mon, Mar 1, 2021 at 10:29 AM Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2021-03-01 at 16:07 +0000, Lucas Alvares Gomes wrote:
Hi all,
As part of the Victoria PTG [0] the Neutron community agreed upon switching the default backend in Devstack to OVN. A lot of work has been done since, from porting the OVN devstack module to the DevStack tree, refactoring the DevStack module to install OVN from distro packages, implementing features to close the parity gap with ML2/OVS, fixing issues with tests and distros, etc...
We are now very close to being able to make the switch and we've thought about sending this email to the broader community to raise awareness about this change as well as bring more attention to the patches that are current on review.
Note that moving DevStack to ML2/OVN does not mean that ML2/OVS is discontinued and/or not supported anymore. The ML2/OVS driver is still going to be developed and maintained by the upstream Neutron community. can we ensure that this does not happen until the xena release. in generall i think its ok to change the default but not this late in the cycle. i would also like to ensure we keep at least one non ovn based multi node job in nova until https://review.opendev.org/c/openstack/nova/+/602432 is merged and possible after. right now the event/neutorn interaction is not the same during move operations.
Below is a e per project explanation with relevant links and issues of where we stand with this work right now:
* Keystone:
Everything should be good for Keystone, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/keystone/+/777963
* Glance:
Everything should be good for Glace, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/glance/+/748390
* Swift:
Everything should be good for Swift, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/swift/+/748403
* Ironic:
Since chainloading iPXE by the OVN built-in DHCP server is work in progress, we've changed most of the Ironic jobs to explicitly enable ML2/OVS and everything is merged, so we should be good for Ironic too. Here is the test patch: https://review.opendev.org/c/openstack/ironic/+/748405
* Cinder:
Cinder is almost complete. There's one test failure in the "tempest-slow-py3" job run on the "test_port_security_macspoofing_port" test.
This failure is due to a bug in core OVN [1]. This bug has already been fixed upstream [2] and the fix has been backported down to the branch-20.03 [3] of the OVN project. However, since we install OVN from packages we are currently waiting for this fix to be included in the packages for Ubuntu Focal (it's based on OVN 20.03). I already contacted the package maintainer which has been very supportive of this work and will work on the package update, but he maintain a handful of backports in that package which is not yet included in OVN 20.03 upstream and he's now working with the core OVN community [4] to include it first in the branch and then create a new package for it. Hopefully this will happen soon.
But for now we have a few options moving on with this issue:
1- Wait for the new package version 2- Mark the test as unstable until we get the new package version 3- Compile OVN from source instead of installing it from packages (OVN_BUILD_FROM_SOURCE=True in local.conf) i dont think we should default to ovn untill a souce build is not required. compiling form souce while not supper expensice still adds time to the job execution and im not sure we should be paying that cost on every devstack job run.
we could maybe compile it once and bake the package into the image or host it on a mirror but i think we should avoid this option if we have alternitives.
What do you think about it ?
Here is the test patch for Cinder: https://review.opendev.org/c/openstack/cinder/+/748227
* Nova:
There are a few patches waiting for review for Nova, which are:
1- Adapting the live migration scripts to work with ML2/OVN: Basically the scripts were trying to stop the Neutron agent (q-agt) process which is not part of an ML2/OVN deployment. The patch changes the code to check if that system unit exists before trying to stop it.
Patch: https://review.opendev.org/c/openstack/nova/+/776419
2- Explicitly set grenade job to ML2/OVS: This is a temporary change which can be removed one release cycle after we switch DevStack to ML2/OVN. Grenade will test updating from the release version to the master branch but, since the default of the released version is not ML2/OVN, upgrading from ML2/OVS to ML2/OVN as part of the grenade job is not supported.
Patch: https://review.opendev.org/c/openstack/nova/+/776934
3- Explicitly set nova-next job to ML2/OVS: This job uses the QoS minimum bandwidth feature which is not yet supported by ML2/OVN [5][6] therefore we are temporarily enabling ML2/OVS for this job until that feature lands in core OVN.
Patch: https://review.opendev.org/c/openstack/nova/+/776944
I also spoke briefly with Sean Mooney (irc: sean-k-mooney) about these changes and he suggested keeping all the Nova jobs on ML2/OVS for now because he feels like a change in the default network driver a few weeks prior to the upstream code freeze can be concerning. We do not know yet precisely when we are changing the default due to the current patches we need to get merged but, if this is a shared feeling among the Nova community I can work on enabling ML2/OVS on all jobs in Nova until we get a new release in OpenStack.
yep this is still my view. i would suggest we do the work required in the repos but not merge it until the xena release is open. thats technically at RC1 so march 25th i think we can safely do the swich after that but i would not change the defualt in any project before then.
Here's the test patch for Nova: https://review.opendev.org/c/openstack/nova/+/776945
* DevStack:
And this is the final patch that will make this all happen: https://review.opendev.org/c/openstack/devstack/+/735097
It changes the default in DevStack from ML2/OVS to ML2/OVN. It's been a long and bumpy road to get to this point and I would like to say thanks to everyone involved so far and everyone that read the whole email, please let me know your thoughts.
thanks for working on this.
[0] https://etherpad.opendev.org/p/neutron-victoria-ptg [1] https://bugs.launchpad.net/tempest/+bug/1728886 [2] https://patchwork.ozlabs.org/project/openvswitch/patch/20200319122641.473776... [3] https://github.com/ovn-org/ovn/commit/0c26bc03064f2c21d208f0f860b48d8ab39380... [4] https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050961.html [5] https://github.com/openstack/nova/blob/ded25f33c734ebff963f06984707a99fe76a9... [6] https://docs.openstack.org/neutron/latest/ovn/gaps.html
Cheers, Lucas
++ Thank you indeed for working diligently on this important change. Please do note that devstack, and the base job that you're modifying is used by many other projects besides the ones that you have enumerated in the subject line. I suggest using [all] as a better subject line indicator to get the attention of folks like me who have filters based on the subject line. Also, the network substrate is important for the project I help maintain: Manila, which provides shared file systems over a network - so I followed your lead and submitted a dependent patch. I hope to reach out to you in case we see some breakages: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/778346
Hi, Thanks for all your hard work on this. I'm wondering is there any doc proposed for devstack to tell people who are not interested in OVN to keep the current devstack behaviour? I have a feeling that using OVN as default Neutron driver would break the CI jobs for some projects like Octavia, Trove, etc. which rely on ovs port for the set up. --- Lingxian Kong Senior Cloud Engineer (Catalyst Cloud) Trove PTL (OpenStack) OpenStack Cloud Provider Co-Lead (Kubernetes) On Wed, Mar 3, 2021 at 2:03 PM Goutham Pacha Ravi <gouthampravi@gmail.com> wrote:
On Mon, Mar 1, 2021 at 10:29 AM Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2021-03-01 at 16:07 +0000, Lucas Alvares Gomes wrote:
Hi all,
As part of the Victoria PTG [0] the Neutron community agreed upon switching the default backend in Devstack to OVN. A lot of work has been done since, from porting the OVN devstack module to the DevStack tree, refactoring the DevStack module to install OVN from distro packages, implementing features to close the parity gap with ML2/OVS, fixing issues with tests and distros, etc...
We are now very close to being able to make the switch and we've thought about sending this email to the broader community to raise awareness about this change as well as bring more attention to the patches that are current on review.
Note that moving DevStack to ML2/OVN does not mean that ML2/OVS is discontinued and/or not supported anymore. The ML2/OVS driver is still going to be developed and maintained by the upstream Neutron community. can we ensure that this does not happen until the xena release. in generall i think its ok to change the default but not this late in
the cycle.
i would also like to ensure we keep at least one non ovn based multi node job in nova until https://review.opendev.org/c/openstack/nova/+/602432 is merged and possible after. right now the event/neutorn interaction is not the same during move operations.
Below is a e per project explanation with relevant links and issues of where we stand with this work right now:
* Keystone:
Everything should be good for Keystone, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/keystone/+/777963
* Glance:
Everything should be good for Glace, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/glance/+/748390
* Swift:
Everything should be good for Swift, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/swift/+/748403
* Ironic:
Since chainloading iPXE by the OVN built-in DHCP server is work in progress, we've changed most of the Ironic jobs to explicitly enable ML2/OVS and everything is merged, so we should be good for Ironic too. Here is the test patch: https://review.opendev.org/c/openstack/ironic/+/748405
* Cinder:
Cinder is almost complete. There's one test failure in the "tempest-slow-py3" job run on the "test_port_security_macspoofing_port" test.
This failure is due to a bug in core OVN [1]. This bug has already been fixed upstream [2] and the fix has been backported down to the branch-20.03 [3] of the OVN project. However, since we install OVN from packages we are currently waiting for this fix to be included in the packages for Ubuntu Focal (it's based on OVN 20.03). I already contacted the package maintainer which has been very supportive of this work and will work on the package update, but he maintain a handful of backports in that package which is not yet included in OVN 20.03 upstream and he's now working with the core OVN community [4] to include it first in the branch and then create a new package for it. Hopefully this will happen soon.
But for now we have a few options moving on with this issue:
1- Wait for the new package version 2- Mark the test as unstable until we get the new package version 3- Compile OVN from source instead of installing it from packages (OVN_BUILD_FROM_SOURCE=True in local.conf)
i dont think we should default to ovn untill a souce build is not required. compiling form souce while not supper expensice still adds time to the job execution and im not sure we should be paying that cost on every devstack job run.
we could maybe compile it once and bake the package into the image or host it on a mirror but i think we should avoid this option if we have alternitives.
What do you think about it ?
Here is the test patch for Cinder: https://review.opendev.org/c/openstack/cinder/+/748227
* Nova:
There are a few patches waiting for review for Nova, which are:
1- Adapting the live migration scripts to work with ML2/OVN: Basically the scripts were trying to stop the Neutron agent (q-agt) process which is not part of an ML2/OVN deployment. The patch changes the code to check if that system unit exists before trying to stop it.
Patch: https://review.opendev.org/c/openstack/nova/+/776419
2- Explicitly set grenade job to ML2/OVS: This is a temporary change which can be removed one release cycle after we switch DevStack to ML2/OVN. Grenade will test updating from the release version to the master branch but, since the default of the released version is not ML2/OVN, upgrading from ML2/OVS to ML2/OVN as part of the grenade job is not supported.
Patch: https://review.opendev.org/c/openstack/nova/+/776934
3- Explicitly set nova-next job to ML2/OVS: This job uses the QoS minimum bandwidth feature which is not yet supported by ML2/OVN [5][6] therefore we are temporarily enabling ML2/OVS for this job until that feature lands in core OVN.
Patch: https://review.opendev.org/c/openstack/nova/+/776944
I also spoke briefly with Sean Mooney (irc: sean-k-mooney) about these changes and he suggested keeping all the Nova jobs on ML2/OVS for now because he feels like a change in the default network driver a few weeks prior to the upstream code freeze can be concerning. We do not know yet precisely when we are changing the default due to the current patches we need to get merged but, if this is a shared feeling among the Nova community I can work on enabling ML2/OVS on all jobs in Nova until we get a new release in OpenStack.
yep this is still my view. i would suggest we do the work required in the repos but not merge it until the xena release is open. thats technically at RC1 so march 25th i think we can safely do the swich after that but i would not change the defualt in any project before then.
Here's the test patch for Nova: https://review.opendev.org/c/openstack/nova/+/776945
* DevStack:
And this is the final patch that will make this all happen: https://review.opendev.org/c/openstack/devstack/+/735097
It changes the default in DevStack from ML2/OVS to ML2/OVN. It's been a long and bumpy road to get to this point and I would like to say thanks to everyone involved so far and everyone that read the whole email, please let me know your thoughts.
thanks for working on this.
[0] https://etherpad.opendev.org/p/neutron-victoria-ptg [1] https://bugs.launchpad.net/tempest/+bug/1728886 [2]
https://patchwork.ozlabs.org/project/openvswitch/patch/20200319122641.473776...
[3] https://github.com/ovn-org/ovn/commit/0c26bc03064f2c21d208f0f860b48d8ab39380... [4] https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050961.html [5] https://github.com/openstack/nova/blob/ded25f33c734ebff963f06984707a99fe76a9... [6] https://docs.openstack.org/neutron/latest/ovn/gaps.html
Cheers, Lucas
++ Thank you indeed for working diligently on this important change.
Please do note that devstack, and the base job that you're modifying is used by many other projects besides the ones that you have enumerated in the subject line. I suggest using [all] as a better subject line indicator to get the attention of folks like me who have filters based on the subject line. Also, the network substrate is important for the project I help maintain: Manila, which provides shared file systems over a network - so I followed your lead and submitted a dependent patch. I hope to reach out to you in case we see some breakages: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/778346
On Wed, 2021-03-03 at 15:59 +1300, Lingxian Kong wrote:
Hi,
Thanks for all your hard work on this.
I'm wondering is there any doc proposed for devstack to tell people who are not interested in OVN to keep the current devstack behaviour? I have a feeling that using OVN as default Neutron driver would break the CI jobs for some projects like Octavia, Trove, etc. which rely on ovs port for the set up.
well ovn is just an alternivie contoler for ovs. so ovn replace the neutron l2 agent it does not replace ovs. project like octavia or trove that deploy loadblances or dbs in vms should not be able to observe a difference. they may still want to deploy ml2/ovs but unless they are doing something directly on the host like adding port directly to ovs because they are not using vms they should not be aware of this change. my reticense to cahnge at this point in the cyle for nova is motiveated maily by gate stablity. the active contibutes to nova dont really have experince with ovn and how to debug it in the gate. we also are getting close to FF when we tend to get a lot of patches and the gate stablity become even more imporant so adding a new vaiabrly to that mix but swaping out the networkbacked between now and the wallaby release seams problematic. in any cases swapping back should ideallly be as simple as setting Q_AGENT=openvswitch. i have not looked at the patches but to swap betwween ovs and linuxbidge you just define Q_AGENT=linuxbridge for the most part so im expecting that we would just enable Q_AGENT=ovn or somthing simialr for ovn. i know ovn used to have its own devstack plugin but if we are makeing it the default that means it need to be support nativly in devstack not as a plugin so useing Q_AGENT=ovn to enable it and make that the new default would seam to be the simplest way to manage that. but yes documenting how to enabel the old behavior is still important the example nova patch shows how to hardcode the old behavior https://review.opendev.org/c/openstack/nova/+/776944/5/.zuul.yaml#234 it seams to be doing this a little more explictly then i would like but its not that hard. i would suggest adding a second sample loca.conf in devstack for standard ovs deployments.
--- Lingxian Kong Senior Cloud Engineer (Catalyst Cloud) Trove PTL (OpenStack) OpenStack Cloud Provider Co-Lead (Kubernetes)
On Wed, Mar 3, 2021 at 2:03 PM Goutham Pacha Ravi <gouthampravi@gmail.com> wrote:
On Mon, Mar 1, 2021 at 10:29 AM Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2021-03-01 at 16:07 +0000, Lucas Alvares Gomes wrote:
Hi all,
As part of the Victoria PTG [0] the Neutron community agreed upon switching the default backend in Devstack to OVN. A lot of work has been done since, from porting the OVN devstack module to the DevStack tree, refactoring the DevStack module to install OVN from distro packages, implementing features to close the parity gap with ML2/OVS, fixing issues with tests and distros, etc...
We are now very close to being able to make the switch and we've thought about sending this email to the broader community to raise awareness about this change as well as bring more attention to the patches that are current on review.
Note that moving DevStack to ML2/OVN does not mean that ML2/OVS is discontinued and/or not supported anymore. The ML2/OVS driver is still going to be developed and maintained by the upstream Neutron community. can we ensure that this does not happen until the xena release. in generall i think its ok to change the default but not this late in
the cycle.
i would also like to ensure we keep at least one non ovn based multi node job in nova until https://review.opendev.org/c/openstack/nova/+/602432 is merged and possible after. right now the event/neutorn interaction is not the same during move operations.
Below is a e per project explanation with relevant links and issues of where we stand with this work right now:
* Keystone:
Everything should be good for Keystone, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/keystone/+/777963
* Glance:
Everything should be good for Glace, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/glance/+/748390
* Swift:
Everything should be good for Swift, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/swift/+/748403
* Ironic:
Since chainloading iPXE by the OVN built-in DHCP server is work in progress, we've changed most of the Ironic jobs to explicitly enable ML2/OVS and everything is merged, so we should be good for Ironic too. Here is the test patch: https://review.opendev.org/c/openstack/ironic/+/748405
* Cinder:
Cinder is almost complete. There's one test failure in the "tempest-slow-py3" job run on the "test_port_security_macspoofing_port" test.
This failure is due to a bug in core OVN [1]. This bug has already been fixed upstream [2] and the fix has been backported down to the branch-20.03 [3] of the OVN project. However, since we install OVN from packages we are currently waiting for this fix to be included in the packages for Ubuntu Focal (it's based on OVN 20.03). I already contacted the package maintainer which has been very supportive of this work and will work on the package update, but he maintain a handful of backports in that package which is not yet included in OVN 20.03 upstream and he's now working with the core OVN community [4] to include it first in the branch and then create a new package for it. Hopefully this will happen soon.
But for now we have a few options moving on with this issue:
1- Wait for the new package version 2- Mark the test as unstable until we get the new package version 3- Compile OVN from source instead of installing it from packages (OVN_BUILD_FROM_SOURCE=True in local.conf)
i dont think we should default to ovn untill a souce build is not required. compiling form souce while not supper expensice still adds time to the job execution and im not sure we should be paying that cost on every devstack job run.
we could maybe compile it once and bake the package into the image or host it on a mirror but i think we should avoid this option if we have alternitives.
What do you think about it ?
Here is the test patch for Cinder: https://review.opendev.org/c/openstack/cinder/+/748227
* Nova:
There are a few patches waiting for review for Nova, which are:
1- Adapting the live migration scripts to work with ML2/OVN: Basically the scripts were trying to stop the Neutron agent (q-agt) process which is not part of an ML2/OVN deployment. The patch changes the code to check if that system unit exists before trying to stop it.
Patch: https://review.opendev.org/c/openstack/nova/+/776419
2- Explicitly set grenade job to ML2/OVS: This is a temporary change which can be removed one release cycle after we switch DevStack to ML2/OVN. Grenade will test updating from the release version to the master branch but, since the default of the released version is not ML2/OVN, upgrading from ML2/OVS to ML2/OVN as part of the grenade job is not supported.
Patch: https://review.opendev.org/c/openstack/nova/+/776934
3- Explicitly set nova-next job to ML2/OVS: This job uses the QoS minimum bandwidth feature which is not yet supported by ML2/OVN [5][6] therefore we are temporarily enabling ML2/OVS for this job until that feature lands in core OVN.
Patch: https://review.opendev.org/c/openstack/nova/+/776944
I also spoke briefly with Sean Mooney (irc: sean-k-mooney) about these changes and he suggested keeping all the Nova jobs on ML2/OVS for now because he feels like a change in the default network driver a few weeks prior to the upstream code freeze can be concerning. We do not know yet precisely when we are changing the default due to the current patches we need to get merged but, if this is a shared feeling among the Nova community I can work on enabling ML2/OVS on all jobs in Nova until we get a new release in OpenStack.
yep this is still my view. i would suggest we do the work required in the repos but not merge it until the xena release is open. thats technically at RC1 so march 25th i think we can safely do the swich after that but i would not change the defualt in any project before then.
Here's the test patch for Nova: https://review.opendev.org/c/openstack/nova/+/776945
* DevStack:
And this is the final patch that will make this all happen: https://review.opendev.org/c/openstack/devstack/+/735097
It changes the default in DevStack from ML2/OVS to ML2/OVN. It's been a long and bumpy road to get to this point and I would like to say thanks to everyone involved so far and everyone that read the whole email, please let me know your thoughts.
thanks for working on this.
[0] https://etherpad.opendev.org/p/neutron-victoria-ptg [1] https://bugs.launchpad.net/tempest/+bug/1728886 [2]
https://patchwork.ozlabs.org/project/openvswitch/patch/20200319122641.473776...
[3] https://github.com/ovn-org/ovn/commit/0c26bc03064f2c21d208f0f860b48d8ab39380... [4] https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050961.html [5] https://github.com/openstack/nova/blob/ded25f33c734ebff963f06984707a99fe76a9... [6] https://docs.openstack.org/neutron/latest/ovn/gaps.html
Cheers, Lucas
++ Thank you indeed for working diligently on this important change.
Please do note that devstack, and the base job that you're modifying is used by many other projects besides the ones that you have enumerated in the subject line. I suggest using [all] as a better subject line indicator to get the attention of folks like me who have filters based on the subject line. Also, the network substrate is important for the project I help maintain: Manila, which provides shared file systems over a network - so I followed your lead and submitted a dependent patch. I hope to reach out to you in case we see some breakages: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/778346
Hi, Dnia środa, 3 marca 2021 05:01:24 CET Sean Mooney pisze:
On Wed, 2021-03-03 at 15:59 +1300, Lingxian Kong wrote:
Hi,
Thanks for all your hard work on this.
I'm wondering is there any doc proposed for devstack to tell people who are not interested in OVN to keep the current devstack behaviour? I have a feeling that using OVN as default Neutron driver would break the CI jobs for some projects like Octavia, Trove, etc. which rely on ovs port for the set up.
You can look on how we set some of our jobs to use ML2/OVS: https:// review.opendev.org/c/openstack/neutron-tempest-plugin/+/749503/26/zuul.d/ master_jobs.yaml#121
well ovn is just an alternivie contoler for ovs. so ovn replace the neutron l2 agent it does not replace ovs. project like octavia or trove that deploy loadblances or dbs in vms should not be able to observe a difference. they may still want to deploy ml2/ovs but unless they are doing something directly on the host like adding port directly to ovs because they are not using vms they should not be aware of this change.
my reticense to cahnge at this point in the cyle for nova is motiveated maily by gate stablity. the active contibutes to nova dont really have experince with ovn and how to debug it in the gate. we also are getting close to FF when we tend to get a lot of patches and the gate stablity become even more imporant so adding a new vaiabrly to that mix but swaping out the networkbacked between now and the wallaby release seams problematic.
in any cases swapping back should ideallly be as simple as setting Q_AGENT=openvswitch.
i have not looked at the patches but to swap betwween ovs and linuxbidge you just define Q_AGENT=linuxbridge for the most part so im expecting that we would just enable Q_AGENT=ovn or somthing simialr for ovn.
i know ovn used to have its own devstack plugin but if we are makeing it the default that means it need to be support nativly in devstack not as a plugin so useing Q_AGENT=ovn to enable it and make that the new default would seam to be the simplest way to manage that.
but yes documenting how to enabel the old behavior is still important the example nova patch shows how to hardcode the old behavior https://review.opendev.org/c/openstack/nova/+/776944/5/.zuul.yaml#234 it seams to be doing this a little more explictly then i would like but its not that hard. i would suggest adding a second sample loca.conf in devstack for standard ovs deployments.
--- Lingxian Kong Senior Cloud Engineer (Catalyst Cloud) Trove PTL (OpenStack) OpenStack Cloud Provider Co-Lead (Kubernetes)
On Wed, Mar 3, 2021 at 2:03 PM Goutham Pacha Ravi <gouthampravi@gmail.com>
wrote:
On Mon, Mar 1, 2021 at 10:29 AM Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2021-03-01 at 16:07 +0000, Lucas Alvares Gomes wrote:
Hi all,
As part of the Victoria PTG [0] the Neutron community agreed upon switching the default backend in Devstack to OVN. A lot of work has been done since, from porting the OVN devstack module to the DevStack tree, refactoring the DevStack module to install OVN from distro packages, implementing features to close the parity gap with ML2/OVS, fixing issues with tests and distros, etc...
We are now very close to being able to make the switch and we've thought about sending this email to the broader community to raise awareness about this change as well as bring more attention to the patches that are current on review.
Note that moving DevStack to ML2/OVN does not mean that ML2/OVS is discontinued and/or not supported anymore. The ML2/OVS driver is still going to be developed and maintained by the upstream Neutron community.
can we ensure that this does not happen until the xena release. in generall i think its ok to change the default but not this late in
the cycle.
i would also like to ensure we keep at least one non ovn based multi
node job in
nova until https://review.opendev.org/c/openstack/nova/+/602432 is
merged and possible after.
right now the event/neutorn interaction is not the same during move
operations.
Below is a e per project explanation with relevant links and issues of where we stand with this work right now:
* Keystone:
Everything should be good for Keystone, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/keystone/+/777963
* Glance:
Everything should be good for Glace, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/glance/+/748390
* Swift:
Everything should be good for Swift, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/swift/+/748403
* Ironic:
Since chainloading iPXE by the OVN built-in DHCP server is work in progress, we've changed most of the Ironic jobs to explicitly enable ML2/OVS and everything is merged, so we should be good for Ironic too. Here is the test patch: https://review.opendev.org/c/openstack/ironic/+/748405
* Cinder:
Cinder is almost complete. There's one test failure in the "tempest-slow-py3" job run on the "test_port_security_macspoofing_port" test.
This failure is due to a bug in core OVN [1]. This bug has already been fixed upstream [2] and the fix has been backported down to the branch-20.03 [3] of the OVN project. However, since we install OVN from packages we are currently waiting for this fix to be included in the packages for Ubuntu Focal (it's based on OVN 20.03). I already contacted the package maintainer which has been very supportive of this work and will work on the package update, but he maintain a handful of backports in that package which is not yet included in OVN 20.03 upstream and he's now working with the core OVN community [4] to include it first in the branch and then create a new package for it. Hopefully this will happen soon.
But for now we have a few options moving on with this issue:
1- Wait for the new package version 2- Mark the test as unstable until we get the new package version 3- Compile OVN from source instead of installing it from packages (OVN_BUILD_FROM_SOURCE=True in local.conf)
i dont think we should default to ovn untill a souce build is not
required.
compiling form souce while not supper expensice still adds time to the
job
execution and im not sure we should be paying that cost on every
devstack job run.
we could maybe compile it once and bake the package into the image or
host it on a mirror
but i think we should avoid this option if we have alternitives.
What do you think about it ?
Here is the test patch for Cinder: https://review.opendev.org/c/openstack/cinder/+/748227
* Nova:
There are a few patches waiting for review for Nova, which are:
1- Adapting the live migration scripts to work with ML2/OVN: Basically the scripts were trying to stop the Neutron agent (q-agt) process which is not part of an ML2/OVN deployment. The patch changes the code to check if that system unit exists before trying to stop it.
Patch: https://review.opendev.org/c/openstack/nova/+/776419
2- Explicitly set grenade job to ML2/OVS: This is a temporary change which can be removed one release cycle after we switch DevStack to ML2/OVN. Grenade will test updating from the release version to the master branch but, since the default of the released version is not ML2/OVN, upgrading from ML2/OVS to ML2/OVN as part of the grenade job is not supported.
Patch: https://review.opendev.org/c/openstack/nova/+/776934
3- Explicitly set nova-next job to ML2/OVS: This job uses the QoS minimum bandwidth feature which is not yet supported by ML2/OVN [5][6] therefore we are temporarily enabling ML2/OVS for this job until that feature lands in core OVN.
Patch: https://review.opendev.org/c/openstack/nova/+/776944
I also spoke briefly with Sean Mooney (irc: sean-k-mooney) about these changes and he suggested keeping all the Nova jobs on ML2/OVS for now because he feels like a change in the default network driver a few weeks prior to the upstream code freeze can be concerning. We do not know yet precisely when we are changing the default due to the current patches we need to get merged but, if this is a shared feeling among the Nova community I can work on enabling ML2/OVS on all jobs in Nova until we get a new release in OpenStack.
yep this is still my view. i would suggest we do the work required in the repos but not merge it
until the xena release
is open. thats technically at RC1 so march 25th i think we can safely do the swich after that but i would not change the
defualt in any project
before then.
Here's the test patch for Nova: https://review.opendev.org/c/openstack/nova/+/776945
* DevStack:
And this is the final patch that will make this all happen: https://review.opendev.org/c/openstack/devstack/+/735097
It changes the default in DevStack from ML2/OVS to ML2/OVN. It's been a long and bumpy road to get to this point and I would like to say thanks to everyone involved so far and everyone that read the whole email, please let me know your thoughts.
thanks for working on this.
[0] https://etherpad.opendev.org/p/neutron-victoria-ptg [1] https://bugs.launchpad.net/tempest/+bug/1728886 [2]
https://patchwork.ozlabs.org/project/openvswitch/patch/20200319122641.47 3776-1-numans@ovn.org/> >
[3]
https://github.com/ovn-org/ovn/commit/0c26bc03064f2c21d208f0f860b48d8ab3 9380cb> >
[4]
https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050961. html> >
[5]
https://github.com/openstack/nova/blob/ded25f33c734ebff963f06984707a99fe 76a9ee1/gate/post_test_hook.sh#L129-L143> >
[6] https://docs.openstack.org/neutron/latest/ovn/gaps.html
Cheers, Lucas
++ Thank you indeed for working diligently on this important change.
Please do note that devstack, and the base job that you're modifying is used by many other projects besides the ones that you have enumerated in the subject line. I suggest using [all] as a better subject line indicator to get the attention of folks like me who have filters based on the subject line. Also, the network substrate is important for the project I help maintain: Manila, which provides shared file systems over a network - so I followed your lead and submitted a dependent patch. I hope to reach out to you in case we see some breakages: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/778346
-- Slawek Kaplonski Principal Software Engineer Red Hat
On Wed, Mar 3, 2021 at 7:35 AM Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
Dnia środa, 3 marca 2021 05:01:24 CET Sean Mooney pisze:
On Wed, 2021-03-03 at 15:59 +1300, Lingxian Kong wrote:
Hi,
Thanks for all your hard work on this.
I'm wondering is there any doc proposed for devstack to tell people who are not interested in OVN to keep the current devstack behaviour? I have a feeling that using OVN as default Neutron driver would break the CI jobs for some projects like Octavia, Trove, etc. which rely on ovs port for the set up.
You can look on how we set some of our jobs to use ML2/OVS: https:// review.opendev.org/c/openstack/neutron-tempest-plugin/+/749503/26/zuul.d/ master_jobs.yaml#121
Thanks Lingxian and Slaweq for the suggestion/replies. Apart from what Slaweq pointed out, the DevStack documentation have a sample config file (https://docs.openstack.org/devstack/latest/#create-a-local-conf) as part of the steps to deploy it, I was thinking I could propose another sample file that would enable ML2/OVS for the deployment and add a note in that doc; if you think it's worth it. Also what gives us more confidence regarding projects like Octavia is that TripleO already uses ML2/OVN as the default driver for their overcloud, so many of these projects are already being tested/used with OVN.
well ovn is just an alternivie contoler for ovs. so ovn replace the neutron l2 agent it does not replace ovs. project like octavia or trove that deploy loadblances or dbs in vms should not be able to observe a difference. they may still want to deploy ml2/ovs but unless they are doing something directly on the host like adding port directly to ovs because they are not using vms they should not be aware of this change.
my reticense to cahnge at this point in the cyle for nova is motiveated maily by gate stablity. the active contibutes to nova dont really have experince with ovn and how to debug it in the gate. we also are getting close to FF when we tend to get a lot of patches and the gate stablity become even more imporant so adding a new vaiabrly to that mix but swaping out the networkbacked between now and the wallaby release seams problematic.
in any cases swapping back should ideallly be as simple as setting Q_AGENT=openvswitch.
i have not looked at the patches but to swap betwween ovs and linuxbidge you just define Q_AGENT=linuxbridge for the most part so im expecting that we would just enable Q_AGENT=ovn or somthing simialr for ovn.
i know ovn used to have its own devstack plugin but if we are makeing it the default that means it need to be support nativly in devstack not as a plugin so useing Q_AGENT=ovn to enable it and make that the new default would seam to be the simplest way to manage that.
but yes documenting how to enabel the old behavior is still important the example nova patch shows how to hardcode the old behavior https://review.opendev.org/c/openstack/nova/+/776944/5/.zuul.yaml#234 it seams to be doing this a little more explictly then i would like but its not that hard. i would suggest adding a second sample loca.conf in devstack for standard ovs deployments.
--- Lingxian Kong Senior Cloud Engineer (Catalyst Cloud) Trove PTL (OpenStack) OpenStack Cloud Provider Co-Lead (Kubernetes)
On Wed, Mar 3, 2021 at 2:03 PM Goutham Pacha Ravi <gouthampravi@gmail.com>
wrote:
On Mon, Mar 1, 2021 at 10:29 AM Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2021-03-01 at 16:07 +0000, Lucas Alvares Gomes wrote:
Hi all,
As part of the Victoria PTG [0] the Neutron community agreed upon switching the default backend in Devstack to OVN. A lot of work has been done since, from porting the OVN devstack module to the DevStack tree, refactoring the DevStack module to install OVN from distro packages, implementing features to close the parity gap with ML2/OVS, fixing issues with tests and distros, etc...
We are now very close to being able to make the switch and we've thought about sending this email to the broader community to raise awareness about this change as well as bring more attention to the patches that are current on review.
Note that moving DevStack to ML2/OVN does not mean that ML2/OVS is discontinued and/or not supported anymore. The ML2/OVS driver is still going to be developed and maintained by the upstream Neutron community.
can we ensure that this does not happen until the xena release. in generall i think its ok to change the default but not this late in
the cycle.
i would also like to ensure we keep at least one non ovn based multi
node job in
nova until https://review.opendev.org/c/openstack/nova/+/602432 is
merged and possible after.
right now the event/neutorn interaction is not the same during move
operations.
Below is a e per project explanation with relevant links and issues of where we stand with this work right now:
* Keystone:
Everything should be good for Keystone, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/keystone/+/777963
* Glance:
Everything should be good for Glace, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/glance/+/748390
* Swift:
Everything should be good for Swift, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/swift/+/748403
* Ironic:
Since chainloading iPXE by the OVN built-in DHCP server is work in progress, we've changed most of the Ironic jobs to explicitly enable ML2/OVS and everything is merged, so we should be good for Ironic too. Here is the test patch: https://review.opendev.org/c/openstack/ironic/+/748405
* Cinder:
Cinder is almost complete. There's one test failure in the "tempest-slow-py3" job run on the "test_port_security_macspoofing_port" test.
This failure is due to a bug in core OVN [1]. This bug has already been fixed upstream [2] and the fix has been backported down to the branch-20.03 [3] of the OVN project. However, since we install OVN from packages we are currently waiting for this fix to be included in the packages for Ubuntu Focal (it's based on OVN 20.03). I already contacted the package maintainer which has been very supportive of this work and will work on the package update, but he maintain a handful of backports in that package which is not yet included in OVN 20.03 upstream and he's now working with the core OVN community [4] to include it first in the branch and then create a new package for it. Hopefully this will happen soon.
But for now we have a few options moving on with this issue:
1- Wait for the new package version 2- Mark the test as unstable until we get the new package version 3- Compile OVN from source instead of installing it from packages (OVN_BUILD_FROM_SOURCE=True in local.conf)
i dont think we should default to ovn untill a souce build is not
required.
compiling form souce while not supper expensice still adds time to the
job
execution and im not sure we should be paying that cost on every
devstack job run.
we could maybe compile it once and bake the package into the image or
host it on a mirror
but i think we should avoid this option if we have alternitives.
What do you think about it ?
Here is the test patch for Cinder: https://review.opendev.org/c/openstack/cinder/+/748227
* Nova:
There are a few patches waiting for review for Nova, which are:
1- Adapting the live migration scripts to work with ML2/OVN: Basically the scripts were trying to stop the Neutron agent (q-agt) process which is not part of an ML2/OVN deployment. The patch changes the code to check if that system unit exists before trying to stop it.
Patch: https://review.opendev.org/c/openstack/nova/+/776419
2- Explicitly set grenade job to ML2/OVS: This is a temporary change which can be removed one release cycle after we switch DevStack to ML2/OVN. Grenade will test updating from the release version to the master branch but, since the default of the released version is not ML2/OVN, upgrading from ML2/OVS to ML2/OVN as part of the grenade job is not supported.
Patch: https://review.opendev.org/c/openstack/nova/+/776934
3- Explicitly set nova-next job to ML2/OVS: This job uses the QoS minimum bandwidth feature which is not yet supported by ML2/OVN [5][6] therefore we are temporarily enabling ML2/OVS for this job until that feature lands in core OVN.
Patch: https://review.opendev.org/c/openstack/nova/+/776944
I also spoke briefly with Sean Mooney (irc: sean-k-mooney) about these changes and he suggested keeping all the Nova jobs on ML2/OVS for now because he feels like a change in the default network driver a few weeks prior to the upstream code freeze can be concerning. We do not know yet precisely when we are changing the default due to the current patches we need to get merged but, if this is a shared feeling among the Nova community I can work on enabling ML2/OVS on all jobs in Nova until we get a new release in OpenStack.
yep this is still my view. i would suggest we do the work required in the repos but not merge it
until the xena release
is open. thats technically at RC1 so march 25th i think we can safely do the swich after that but i would not change the
defualt in any project
before then.
Here's the test patch for Nova: https://review.opendev.org/c/openstack/nova/+/776945
* DevStack:
And this is the final patch that will make this all happen: https://review.opendev.org/c/openstack/devstack/+/735097
It changes the default in DevStack from ML2/OVS to ML2/OVN. It's been a long and bumpy road to get to this point and I would like to say thanks to everyone involved so far and everyone that read the whole email, please let me know your thoughts.
thanks for working on this.
[0] https://etherpad.opendev.org/p/neutron-victoria-ptg [1] https://bugs.launchpad.net/tempest/+bug/1728886 [2]
https://patchwork.ozlabs.org/project/openvswitch/patch/20200319122641.47 3776-1-numans@ovn.org/> >
[3]
https://github.com/ovn-org/ovn/commit/0c26bc03064f2c21d208f0f860b48d8ab3 9380cb> >
[4]
https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050961. html> >
[5]
https://github.com/openstack/nova/blob/ded25f33c734ebff963f06984707a99fe 76a9ee1/gate/post_test_hook.sh#L129-L143> >
[6] https://docs.openstack.org/neutron/latest/ovn/gaps.html
Cheers, Lucas
++ Thank you indeed for working diligently on this important change.
Please do note that devstack, and the base job that you're modifying is used by many other projects besides the ones that you have enumerated in the subject line. I suggest using [all] as a better subject line indicator to get the attention of folks like me who have filters based on the subject line. Also, the network substrate is important for the project I help maintain: Manila, which provides shared file systems over a network - so I followed your lead and submitted a dependent patch. I hope to reach out to you in case we see some breakages: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/778346
-- Slawek Kaplonski Principal Software Engineer Red Hat
On Wed, Mar 3, 2021 at 5:15 PM Sean Mooney <smooney@redhat.com> wrote:
Hi,
Thanks for all your hard work on this.
I'm wondering is there any doc proposed for devstack to tell people who are not interested in OVN to keep the current devstack behaviour? I have a feeling that using OVN as default Neutron driver would break the CI jobs for some projects like Octavia, Trove, etc. which rely on ovs port for
On Wed, 2021-03-03 at 15:59 +1300, Lingxian Kong wrote: the
set up.
well ovn is just an alternivie contoler for ovs. so ovn replace the neutron l2 agent it does not replace ovs. project like octavia or trove that deploy loadblances or dbs in vms should not be able to observe a difference. they may still want to deploy ml2/ovs but unless they are doing something directly on the host like adding port directly to ovs because they are not using vms they should not be aware of this change.
Yes, they are. Please see https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L466 as an example for Octavia. --- Lingxian Kong Senior Cloud Engineer (Catalyst Cloud) Trove PTL (OpenStack) OpenStack Cloud Provider Co-Lead (Kubernetes)
On Wed, 2021-03-03 at 23:50 +1300, Lingxian Kong wrote:
On Wed, Mar 3, 2021 at 5:15 PM Sean Mooney <smooney@redhat.com> wrote:
Hi,
Thanks for all your hard work on this.
I'm wondering is there any doc proposed for devstack to tell people who are not interested in OVN to keep the current devstack behaviour? I have a feeling that using OVN as default Neutron driver would break the CI jobs for some projects like Octavia, Trove, etc. which rely on ovs port for
On Wed, 2021-03-03 at 15:59 +1300, Lingxian Kong wrote: the
set up.
well ovn is just an alternivie contoler for ovs. so ovn replace the neutron l2 agent it does not replace ovs. project like octavia or trove that deploy loadblances or dbs in vms should not be able to observe a difference. they may still want to deploy ml2/ovs but unless they are doing something directly on the host like adding port directly to ovs because they are not using vms they should not be aware of this change.
Yes, they are.
Please see https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L466 as an example for Octavia. right but octavia really should not be doing that.
i mean it will still work in the sense that ovn shoudl be abel to correalte that port to this hos if you have correcly bound that port too the host but adding internal ports to br-int to provide network connectivy seams like a hack. ml2/ovn still calls the amin ovs bridge br-int so the port add will work as it did before. the br-int is owned by neutron and operators are not allowed to add flows or interface to that brdige normally. so in a real deployment i woudl hope that the woiuld not be adding this port and assigning a mac or ip rules like this. conenctiviy shoudl be provided via the br-ex /phsyical netowrk using provider networking right? in anycase you can swap back to ml2/ovs for ocatvia jobs. if you have correctly bound the manament port to the current host then i think ovn shoudl be able to install the correct openflow rules to allow it to function however so it may be as simipar as extending the elif to work with ovn. the port create on https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L458 does seam to have set the host properly its still kind of troubling to me to see something like this being down our side an openstack agent on the host. for example i would have expected an ml2 l2 openvswigh agent extntion to create the prot via os-vif or similar in respocne to the api action instead of manually doing this.
--- Lingxian Kong Senior Cloud Engineer (Catalyst Cloud) Trove PTL (OpenStack) OpenStack Cloud Provider Co-Lead (Kubernetes)
Hi, Dnia środa, 3 marca 2021 11:50:59 CET Lingxian Kong pisze:
On Wed, Mar 3, 2021 at 5:15 PM Sean Mooney <smooney@redhat.com> wrote:
On Wed, 2021-03-03 at 15:59 +1300, Lingxian Kong wrote:
Hi,
Thanks for all your hard work on this.
I'm wondering is there any doc proposed for devstack to tell people who
are
not interested in OVN to keep the current devstack behaviour? I have a feeling that using OVN as default Neutron driver would break the CI jobs for some projects like Octavia, Trove, etc. which rely on ovs port for
the
set up.
well ovn is just an alternivie contoler for ovs. so ovn replace the neutron l2 agent it does not replace ovs. project like octavia or trove that deploy loadblances or dbs in vms should not be able to observe a difference. they may still want to deploy ml2/ovs but unless they are doing something directly on the host like adding port directly to ovs because they are not using vms they should not be aware of this change.
Yes, they are.
Please see https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L466 as an example for Octavia.
I'm really not an Octavia expert but AFAIK we are testing Octavia with ML2/OVN in TripleO jobs already as we switched default neutron backend in TripleO to be ML2/OVN somewhere around Stein cycle IIRC.
--- Lingxian Kong Senior Cloud Engineer (Catalyst Cloud) Trove PTL (OpenStack) OpenStack Cloud Provider Co-Lead (Kubernetes)
-- Slawek Kaplonski Principal Software Engineer Red Hat
On 3/3/21 5:50 AM, Lingxian Kong wrote:
On Wed, Mar 3, 2021 at 5:15 PM Sean Mooney <smooney@redhat.com <mailto:smooney@redhat.com>> wrote:
On Wed, 2021-03-03 at 15:59 +1300, Lingxian Kong wrote: > Hi, > > Thanks for all your hard work on this. > > I'm wondering is there any doc proposed for devstack to tell people who are > not interested in OVN to keep the current devstack behaviour? I have a > feeling that using OVN as default Neutron driver would break the CI jobs > for some projects like Octavia, Trove, etc. which rely on ovs port for the > set up.
well ovn is just an alternivie contoler for ovs. so ovn replace the neutron l2 agent it does not replace ovs. project like octavia or trove that deploy loadblances or dbs in vms should not be able to observe a difference. they may still want to deploy ml2/ovs but unless they are doing something directly on the host like adding port directly to ovs because they are not using vms they should not be aware of this change.
Yes, they are.
Please see https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L466 as an example for Octavia.
The code to do this configuration was all migrated to the neutron repository, with the last bit of cleanup for the Octavia code you highlighted here: https://review.opendev.org/c/openstack/octavia/+/718192 It just needs a final push to get it merged. -Brian
On 03/03/21 14:29 -0500, Brian Haley wrote:
On 3/3/21 5:50 AM, Lingxian Kong wrote:
On Wed, Mar 3, 2021 at 5:15 PM Sean Mooney <smooney@redhat.com <mailto:smooney@redhat.com>> wrote:
On Wed, 2021-03-03 at 15:59 +1300, Lingxian Kong wrote: > Hi, > > Thanks for all your hard work on this. > > I'm wondering is there any doc proposed for devstack to tell people who are > not interested in OVN to keep the current devstack behaviour? I have a > feeling that using OVN as default Neutron driver would break the CI jobs > for some projects like Octavia, Trove, etc. which rely on ovs port for the > set up.
well ovn is just an alternivie contoler for ovs. so ovn replace the neutron l2 agent it does not replace ovs. project like octavia or trove that deploy loadblances or dbs in vms should not be able to observe a difference. they may still want to deploy ml2/ovs but unless they are doing something directly on the host like adding port directly to ovs because they are not using vms they should not be aware of this change.
Yes, they are.
Please see https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L466 as an example for Octavia.
The code to do this configuration was all migrated to the neutron repository, with the last bit of cleanup for the Octavia code you highlighted here:
https://review.opendev.org/c/openstack/octavia/+/718192
It just needs a final push to get it merged.
-Brian
There's similar plug-into-ovs code used by Manila container and generic backend drivers [1], [2], in the manila codebase, not migrated into neutron as done for octavia. Putting my downstream hat on, I'll remark that we tested Manila with OVN and run it that way now in production but downstream these drivers are not supported so I don't think we know they will work in upstream CI. Incidentally, I have always thought and said that it seems to me to be a layer violation for Manila to manipulate the ovs switches directly. We should be calling a neutron API. This would also free us of the topology restriction of having to run the Manila share service on the node with the OVS switch in question. [1] https://github.com/openstack/manila/blob/master/manila/share/drivers/contain... [2] https://github.com/openstack/manila/blob/master/manila/network/linux/ovs_lib... -- Tom
Octavia doesn't really care what the ML2 is in neutron, there are no dependencies on it as we use stable neutron APIs. Devstack however is creating a port on the management network for the controller processes. Octavia has a function hook to allow the SDN providers to handle creating access to the management network. When OVN moved into neutron this hook was implemented in neutron for linux bridge, OVS, and OVN. I should also note that we have been running Octavia gate jobs, on neutron with OVN, since before the migration of OVN into neutron, so I would not expect any issues from the proposed change to the default ML2 in neutron. Michael On Tue, Mar 2, 2021 at 7:05 PM Lingxian Kong <anlin.kong@gmail.com> wrote:
Hi,
Thanks for all your hard work on this.
I'm wondering is there any doc proposed for devstack to tell people who are not interested in OVN to keep the current devstack behaviour? I have a feeling that using OVN as default Neutron driver would break the CI jobs for some projects like Octavia, Trove, etc. which rely on ovs port for the set up.
--- Lingxian Kong Senior Cloud Engineer (Catalyst Cloud) Trove PTL (OpenStack) OpenStack Cloud Provider Co-Lead (Kubernetes)
On Wed, Mar 3, 2021 at 2:03 PM Goutham Pacha Ravi <gouthampravi@gmail.com> wrote:
On Mon, Mar 1, 2021 at 10:29 AM Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2021-03-01 at 16:07 +0000, Lucas Alvares Gomes wrote:
Hi all,
As part of the Victoria PTG [0] the Neutron community agreed upon switching the default backend in Devstack to OVN. A lot of work has been done since, from porting the OVN devstack module to the DevStack tree, refactoring the DevStack module to install OVN from distro packages, implementing features to close the parity gap with ML2/OVS, fixing issues with tests and distros, etc...
We are now very close to being able to make the switch and we've thought about sending this email to the broader community to raise awareness about this change as well as bring more attention to the patches that are current on review.
Note that moving DevStack to ML2/OVN does not mean that ML2/OVS is discontinued and/or not supported anymore. The ML2/OVS driver is still going to be developed and maintained by the upstream Neutron community. can we ensure that this does not happen until the xena release. in generall i think its ok to change the default but not this late in the cycle. i would also like to ensure we keep at least one non ovn based multi node job in nova until https://review.opendev.org/c/openstack/nova/+/602432 is merged and possible after. right now the event/neutorn interaction is not the same during move operations.
Below is a e per project explanation with relevant links and issues of where we stand with this work right now:
* Keystone:
Everything should be good for Keystone, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/keystone/+/777963
* Glance:
Everything should be good for Glace, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/glance/+/748390
* Swift:
Everything should be good for Swift, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/swift/+/748403
* Ironic:
Since chainloading iPXE by the OVN built-in DHCP server is work in progress, we've changed most of the Ironic jobs to explicitly enable ML2/OVS and everything is merged, so we should be good for Ironic too. Here is the test patch: https://review.opendev.org/c/openstack/ironic/+/748405
* Cinder:
Cinder is almost complete. There's one test failure in the "tempest-slow-py3" job run on the "test_port_security_macspoofing_port" test.
This failure is due to a bug in core OVN [1]. This bug has already been fixed upstream [2] and the fix has been backported down to the branch-20.03 [3] of the OVN project. However, since we install OVN from packages we are currently waiting for this fix to be included in the packages for Ubuntu Focal (it's based on OVN 20.03). I already contacted the package maintainer which has been very supportive of this work and will work on the package update, but he maintain a handful of backports in that package which is not yet included in OVN 20.03 upstream and he's now working with the core OVN community [4] to include it first in the branch and then create a new package for it. Hopefully this will happen soon.
But for now we have a few options moving on with this issue:
1- Wait for the new package version 2- Mark the test as unstable until we get the new package version 3- Compile OVN from source instead of installing it from packages (OVN_BUILD_FROM_SOURCE=True in local.conf) i dont think we should default to ovn untill a souce build is not required. compiling form souce while not supper expensice still adds time to the job execution and im not sure we should be paying that cost on every devstack job run.
we could maybe compile it once and bake the package into the image or host it on a mirror but i think we should avoid this option if we have alternitives.
What do you think about it ?
Here is the test patch for Cinder: https://review.opendev.org/c/openstack/cinder/+/748227
* Nova:
There are a few patches waiting for review for Nova, which are:
1- Adapting the live migration scripts to work with ML2/OVN: Basically the scripts were trying to stop the Neutron agent (q-agt) process which is not part of an ML2/OVN deployment. The patch changes the code to check if that system unit exists before trying to stop it.
Patch: https://review.opendev.org/c/openstack/nova/+/776419
2- Explicitly set grenade job to ML2/OVS: This is a temporary change which can be removed one release cycle after we switch DevStack to ML2/OVN. Grenade will test updating from the release version to the master branch but, since the default of the released version is not ML2/OVN, upgrading from ML2/OVS to ML2/OVN as part of the grenade job is not supported.
Patch: https://review.opendev.org/c/openstack/nova/+/776934
3- Explicitly set nova-next job to ML2/OVS: This job uses the QoS minimum bandwidth feature which is not yet supported by ML2/OVN [5][6] therefore we are temporarily enabling ML2/OVS for this job until that feature lands in core OVN.
Patch: https://review.opendev.org/c/openstack/nova/+/776944
I also spoke briefly with Sean Mooney (irc: sean-k-mooney) about these changes and he suggested keeping all the Nova jobs on ML2/OVS for now because he feels like a change in the default network driver a few weeks prior to the upstream code freeze can be concerning. We do not know yet precisely when we are changing the default due to the current patches we need to get merged but, if this is a shared feeling among the Nova community I can work on enabling ML2/OVS on all jobs in Nova until we get a new release in OpenStack.
yep this is still my view. i would suggest we do the work required in the repos but not merge it until the xena release is open. thats technically at RC1 so march 25th i think we can safely do the swich after that but i would not change the defualt in any project before then.
Here's the test patch for Nova: https://review.opendev.org/c/openstack/nova/+/776945
* DevStack:
And this is the final patch that will make this all happen: https://review.opendev.org/c/openstack/devstack/+/735097
It changes the default in DevStack from ML2/OVS to ML2/OVN. It's been a long and bumpy road to get to this point and I would like to say thanks to everyone involved so far and everyone that read the whole email, please let me know your thoughts.
thanks for working on this.
[0] https://etherpad.opendev.org/p/neutron-victoria-ptg [1] https://bugs.launchpad.net/tempest/+bug/1728886 [2] https://patchwork.ozlabs.org/project/openvswitch/patch/20200319122641.473776... [3] https://github.com/ovn-org/ovn/commit/0c26bc03064f2c21d208f0f860b48d8ab39380... [4] https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050961.html [5] https://github.com/openstack/nova/blob/ded25f33c734ebff963f06984707a99fe76a9... [6] https://docs.openstack.org/neutron/latest/ovn/gaps.html
Cheers, Lucas
++ Thank you indeed for working diligently on this important change.
Please do note that devstack, and the base job that you're modifying is used by many other projects besides the ones that you have enumerated in the subject line. I suggest using [all] as a better subject line indicator to get the attention of folks like me who have filters based on the subject line. Also, the network substrate is important for the project I help maintain: Manila, which provides shared file systems over a network - so I followed your lead and submitted a dependent patch. I hope to reach out to you in case we see some breakages: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/778346
Hi, Dnia piątek, 5 marca 2021 23:13:09 CET Michael Johnson pisze:
Octavia doesn't really care what the ML2 is in neutron, there are no dependencies on it as we use stable neutron APIs.
Devstack however is creating a port on the management network for the controller processes. Octavia has a function hook to allow the SDN providers to handle creating access to the management network. When OVN moved into neutron this hook was implemented in neutron for linux bridge, OVS, and OVN.
I should also note that we have been running Octavia gate jobs, on neutron with OVN, since before the migration of OVN into neutron, so I would not expect any issues from the proposed change to the default ML2 in neutron.
Thx for confirmation that Octavia should be fine with it :)
Michael
On Tue, Mar 2, 2021 at 7:05 PM Lingxian Kong <anlin.kong@gmail.com> wrote:
Hi,
Thanks for all your hard work on this.
I'm wondering is there any doc proposed for devstack to tell people who are not interested in OVN to keep the current devstack behaviour? I have a feeling that using OVN as default Neutron driver would break the CI jobs for some projects like Octavia, Trove, etc. which rely on ovs port for the set up.
--- Lingxian Kong Senior Cloud Engineer (Catalyst Cloud) Trove PTL (OpenStack) OpenStack Cloud Provider Co-Lead (Kubernetes)
On Wed, Mar 3, 2021 at 2:03 PM Goutham Pacha Ravi <gouthampravi@gmail.com>
wrote:
On Mon, Mar 1, 2021 at 10:29 AM Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2021-03-01 at 16:07 +0000, Lucas Alvares Gomes wrote:
Hi all,
As part of the Victoria PTG [0] the Neutron community agreed upon switching the default backend in Devstack to OVN. A lot of work has been done since, from porting the OVN devstack module to the DevStack tree, refactoring the DevStack module to install OVN from distro packages, implementing features to close the parity gap with ML2/OVS, fixing issues with tests and distros, etc...
We are now very close to being able to make the switch and we've thought about sending this email to the broader community to raise awareness about this change as well as bring more attention to the patches that are current on review.
Note that moving DevStack to ML2/OVN does not mean that ML2/OVS is discontinued and/or not supported anymore. The ML2/OVS driver is still going to be developed and maintained by the upstream Neutron community.
can we ensure that this does not happen until the xena release. in generall i think its ok to change the default but not this late in the cycle. i would also like to ensure we keep at least one non ovn based multi node job in nova until https://review.opendev.org/c/openstack/nova/+/602432 is merged and possible after. right now the event/neutorn interaction is not the same during move operations.>> >
Below is a e per project explanation with relevant links and issues of where we stand with this work right now:
* Keystone:
Everything should be good for Keystone, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/keystone/+/777963
* Glance:
Everything should be good for Glace, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/glance/+/748390
* Swift:
Everything should be good for Swift, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/swift/+/748403
* Ironic:
Since chainloading iPXE by the OVN built-in DHCP server is work in progress, we've changed most of the Ironic jobs to explicitly enable ML2/OVS and everything is merged, so we should be good for Ironic too. Here is the test patch: https://review.opendev.org/c/openstack/ironic/+/748405
* Cinder:
Cinder is almost complete. There's one test failure in the "tempest-slow-py3" job run on the "test_port_security_macspoofing_port" test.
This failure is due to a bug in core OVN [1]. This bug has already been fixed upstream [2] and the fix has been backported down to the branch-20.03 [3] of the OVN project. However, since we install OVN from packages we are currently waiting for this fix to be included in the packages for Ubuntu Focal (it's based on OVN 20.03). I already contacted the package maintainer which has been very supportive of this work and will work on the package update, but he maintain a handful of backports in that package which is not yet included in OVN 20.03 upstream and he's now working with the core OVN community [4] to include it first in the branch and then create a new package for it. Hopefully this will happen soon.
But for now we have a few options moving on with this issue:
1- Wait for the new package version 2- Mark the test as unstable until we get the new package version 3- Compile OVN from source instead of installing it from packages (OVN_BUILD_FROM_SOURCE=True in local.conf)
i dont think we should default to ovn untill a souce build is not required. compiling form souce while not supper expensice still adds time to the job execution and im not sure we should be paying that cost on every devstack job run.
we could maybe compile it once and bake the package into the image or host it on a mirror but i think we should avoid this option if we have alternitives.
What do you think about it ?
Here is the test patch for Cinder: https://review.opendev.org/c/openstack/cinder/+/748227
* Nova:
There are a few patches waiting for review for Nova, which are:
1- Adapting the live migration scripts to work with ML2/OVN: Basically the scripts were trying to stop the Neutron agent (q-agt) process which is not part of an ML2/OVN deployment. The patch changes the code to check if that system unit exists before trying to stop it.
Patch: https://review.opendev.org/c/openstack/nova/+/776419
2- Explicitly set grenade job to ML2/OVS: This is a temporary change which can be removed one release cycle after we switch DevStack to ML2/OVN. Grenade will test updating from the release version to the master branch but, since the default of the released version is not ML2/OVN, upgrading from ML2/OVS to ML2/OVN as part of the grenade job is not supported.
Patch: https://review.opendev.org/c/openstack/nova/+/776934
3- Explicitly set nova-next job to ML2/OVS: This job uses the QoS minimum bandwidth feature which is not yet supported by ML2/OVN [5][6] therefore we are temporarily enabling ML2/OVS for this job until that feature lands in core OVN.
Patch: https://review.opendev.org/c/openstack/nova/+/776944
I also spoke briefly with Sean Mooney (irc: sean-k-mooney) about these changes and he suggested keeping all the Nova jobs on ML2/OVS for now because he feels like a change in the default network driver a few weeks prior to the upstream code freeze can be concerning. We do not know yet precisely when we are changing the default due to the current patches we need to get merged but, if this is a shared feeling among the Nova community I can work on enabling ML2/OVS on all jobs in Nova until we get a new release in OpenStack.
yep this is still my view. i would suggest we do the work required in the repos but not merge it until the xena release is open. thats technically at RC1 so march 25th i think we can safely do the swich after that but i would not change the defualt in any project before then.
Here's the test patch for Nova: https://review.opendev.org/c/openstack/nova/+/776945
* DevStack:
And this is the final patch that will make this all happen: https://review.opendev.org/c/openstack/devstack/+/735097
It changes the default in DevStack from ML2/OVS to ML2/OVN. It's been a long and bumpy road to get to this point and I would like to say thanks to everyone involved so far and everyone that read the whole email, please let me know your thoughts.
thanks for working on this.
[0] https://etherpad.opendev.org/p/neutron-victoria-ptg [1] https://bugs.launchpad.net/tempest/+bug/1728886 [2] https://patchwork.ozlabs.org/project/openvswitch/patch/2020031912264 1.473776-1-numans@ovn.org/ [3] https://github.com/ovn-org/ovn/commit/0c26bc03064f2c21d208f0f860b48d 8ab39380cb [4] https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050 961.html [5] https://github.com/openstack/nova/blob/ded25f33c734ebff963f06984707a 99fe76a9ee1/gate/post_test_hook.sh#L129-L143 [6] https://docs.openstack.org/neutron/latest/ovn/gaps.html
Cheers, Lucas
++ Thank you indeed for working diligently on this important change.
Please do note that devstack, and the base job that you're modifying is used by many other projects besides the ones that you have enumerated in the subject line. I suggest using [all] as a better subject line indicator to get the attention of folks like me who have filters based on the subject line. Also, the network substrate is important for the project I help maintain: Manila, which provides shared file systems over a network - so I followed your lead and submitted a dependent patch. I hope to reach out to you in case we see some breakages: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/778346
-- Slawek Kaplonski Principal Software Engineer Red Hat
On Wed, Mar 3, 2021 at 12:57 AM Goutham Pacha Ravi <gouthampravi@gmail.com> wrote:
On Mon, Mar 1, 2021 at 10:29 AM Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2021-03-01 at 16:07 +0000, Lucas Alvares Gomes wrote:
Hi all,
As part of the Victoria PTG [0] the Neutron community agreed upon switching the default backend in Devstack to OVN. A lot of work has been done since, from porting the OVN devstack module to the DevStack tree, refactoring the DevStack module to install OVN from distro packages, implementing features to close the parity gap with ML2/OVS, fixing issues with tests and distros, etc...
We are now very close to being able to make the switch and we've thought about sending this email to the broader community to raise awareness about this change as well as bring more attention to the patches that are current on review.
Note that moving DevStack to ML2/OVN does not mean that ML2/OVS is discontinued and/or not supported anymore. The ML2/OVS driver is still going to be developed and maintained by the upstream Neutron community. can we ensure that this does not happen until the xena release. in generall i think its ok to change the default but not this late in the cycle. i would also like to ensure we keep at least one non ovn based multi node job in nova until https://review.opendev.org/c/openstack/nova/+/602432 is merged and possible after. right now the event/neutorn interaction is not the same during move operations.
Below is a e per project explanation with relevant links and issues of where we stand with this work right now:
* Keystone:
Everything should be good for Keystone, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/keystone/+/777963
* Glance:
Everything should be good for Glace, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/glance/+/748390
* Swift:
Everything should be good for Swift, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/swift/+/748403
* Ironic:
Since chainloading iPXE by the OVN built-in DHCP server is work in progress, we've changed most of the Ironic jobs to explicitly enable ML2/OVS and everything is merged, so we should be good for Ironic too. Here is the test patch: https://review.opendev.org/c/openstack/ironic/+/748405
* Cinder:
Cinder is almost complete. There's one test failure in the "tempest-slow-py3" job run on the "test_port_security_macspoofing_port" test.
This failure is due to a bug in core OVN [1]. This bug has already been fixed upstream [2] and the fix has been backported down to the branch-20.03 [3] of the OVN project. However, since we install OVN from packages we are currently waiting for this fix to be included in the packages for Ubuntu Focal (it's based on OVN 20.03). I already contacted the package maintainer which has been very supportive of this work and will work on the package update, but he maintain a handful of backports in that package which is not yet included in OVN 20.03 upstream and he's now working with the core OVN community [4] to include it first in the branch and then create a new package for it. Hopefully this will happen soon.
But for now we have a few options moving on with this issue:
1- Wait for the new package version 2- Mark the test as unstable until we get the new package version 3- Compile OVN from source instead of installing it from packages (OVN_BUILD_FROM_SOURCE=True in local.conf) i dont think we should default to ovn untill a souce build is not required. compiling form souce while not supper expensice still adds time to the job execution and im not sure we should be paying that cost on every devstack job run.
we could maybe compile it once and bake the package into the image or host it on a mirror but i think we should avoid this option if we have alternitives.
What do you think about it ?
Here is the test patch for Cinder: https://review.opendev.org/c/openstack/cinder/+/748227
* Nova:
There are a few patches waiting for review for Nova, which are:
1- Adapting the live migration scripts to work with ML2/OVN: Basically the scripts were trying to stop the Neutron agent (q-agt) process which is not part of an ML2/OVN deployment. The patch changes the code to check if that system unit exists before trying to stop it.
Patch: https://review.opendev.org/c/openstack/nova/+/776419
2- Explicitly set grenade job to ML2/OVS: This is a temporary change which can be removed one release cycle after we switch DevStack to ML2/OVN. Grenade will test updating from the release version to the master branch but, since the default of the released version is not ML2/OVN, upgrading from ML2/OVS to ML2/OVN as part of the grenade job is not supported.
Patch: https://review.opendev.org/c/openstack/nova/+/776934
3- Explicitly set nova-next job to ML2/OVS: This job uses the QoS minimum bandwidth feature which is not yet supported by ML2/OVN [5][6] therefore we are temporarily enabling ML2/OVS for this job until that feature lands in core OVN.
Patch: https://review.opendev.org/c/openstack/nova/+/776944
I also spoke briefly with Sean Mooney (irc: sean-k-mooney) about these changes and he suggested keeping all the Nova jobs on ML2/OVS for now because he feels like a change in the default network driver a few weeks prior to the upstream code freeze can be concerning. We do not know yet precisely when we are changing the default due to the current patches we need to get merged but, if this is a shared feeling among the Nova community I can work on enabling ML2/OVS on all jobs in Nova until we get a new release in OpenStack.
yep this is still my view. i would suggest we do the work required in the repos but not merge it until the xena release is open. thats technically at RC1 so march 25th i think we can safely do the swich after that but i would not change the defualt in any project before then.
Here's the test patch for Nova: https://review.opendev.org/c/openstack/nova/+/776945
* DevStack:
And this is the final patch that will make this all happen: https://review.opendev.org/c/openstack/devstack/+/735097
It changes the default in DevStack from ML2/OVS to ML2/OVN. It's been a long and bumpy road to get to this point and I would like to say thanks to everyone involved so far and everyone that read the whole email, please let me know your thoughts.
thanks for working on this.
[0] https://etherpad.opendev.org/p/neutron-victoria-ptg [1] https://bugs.launchpad.net/tempest/+bug/1728886 [2] https://patchwork.ozlabs.org/project/openvswitch/patch/20200319122641.473776... [3] https://github.com/ovn-org/ovn/commit/0c26bc03064f2c21d208f0f860b48d8ab39380... [4] https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050961.html [5] https://github.com/openstack/nova/blob/ded25f33c734ebff963f06984707a99fe76a9... [6] https://docs.openstack.org/neutron/latest/ovn/gaps.html
Cheers, Lucas
++ Thank you indeed for working diligently on this important change.
Please do note that devstack, and the base job that you're modifying is used by many other projects besides the ones that you have enumerated in the subject line. I suggest using [all] as a better subject line indicator to get the attention of folks like me who have filters based on the subject line. Also, the network substrate is important for the project I help maintain: Manila, which provides shared file systems over a network - so I followed your lead and submitted a dependent patch. I hope to reach out to you in case we see some breakages: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/778346
Thanks for the suggestion, you are right the [ALL] would make sense. I will try to raise more awareness as possible on this change in the default. Also thanks for proposing the test patch for Manilla, I see the gate is happy there! But yeah, feel free to reach out to me if anything breaks and I will glad looking into it.
On Mon, Mar 1, 2021 at 6:25 PM Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2021-03-01 at 16:07 +0000, Lucas Alvares Gomes wrote:
Hi all,
As part of the Victoria PTG [0] the Neutron community agreed upon switching the default backend in Devstack to OVN. A lot of work has been done since, from porting the OVN devstack module to the DevStack tree, refactoring the DevStack module to install OVN from distro packages, implementing features to close the parity gap with ML2/OVS, fixing issues with tests and distros, etc...
We are now very close to being able to make the switch and we've thought about sending this email to the broader community to raise awareness about this change as well as bring more attention to the patches that are current on review.
Note that moving DevStack to ML2/OVN does not mean that ML2/OVS is discontinued and/or not supported anymore. The ML2/OVS driver is still going to be developed and maintained by the upstream Neutron community. can we ensure that this does not happen until the xena release. in generall i think its ok to change the default but not this late in the cycle. i would also like to ensure we keep at least one non ovn based multi node job in nova until https://review.opendev.org/c/openstack/nova/+/602432 is merged and possible after. right now the event/neutorn interaction is not the same during move operations.
I think it's fair to wait for the new release cycle to start since we are just a few weeks away and then we can flip the default in DevStack. I will state this in the last DevStack patch and set the workflow -1 until then. That said, I also think that other patches could be merged before that, those are just adapting a few scripts to work with ML2/OVN and enabling ML2/OVS explicitly where it makes sense. That way, when time comes, we will just need to merge the DevStack patch.
Below is a e per project explanation with relevant links and issues of where we stand with this work right now:
* Keystone:
Everything should be good for Keystone, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/keystone/+/777963
* Glance:
Everything should be good for Glace, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/glance/+/748390
* Swift:
Everything should be good for Swift, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/swift/+/748403
* Ironic:
Since chainloading iPXE by the OVN built-in DHCP server is work in progress, we've changed most of the Ironic jobs to explicitly enable ML2/OVS and everything is merged, so we should be good for Ironic too. Here is the test patch: https://review.opendev.org/c/openstack/ironic/+/748405
* Cinder:
Cinder is almost complete. There's one test failure in the "tempest-slow-py3" job run on the "test_port_security_macspoofing_port" test.
This failure is due to a bug in core OVN [1]. This bug has already been fixed upstream [2] and the fix has been backported down to the branch-20.03 [3] of the OVN project. However, since we install OVN from packages we are currently waiting for this fix to be included in the packages for Ubuntu Focal (it's based on OVN 20.03). I already contacted the package maintainer which has been very supportive of this work and will work on the package update, but he maintain a handful of backports in that package which is not yet included in OVN 20.03 upstream and he's now working with the core OVN community [4] to include it first in the branch and then create a new package for it. Hopefully this will happen soon.
But for now we have a few options moving on with this issue:
1- Wait for the new package version 2- Mark the test as unstable until we get the new package version 3- Compile OVN from source instead of installing it from packages (OVN_BUILD_FROM_SOURCE=True in local.conf)
i dont think we should default to ovn untill a souce build is not required. compiling form souce while not supper expensice still adds time to the job execution and im not sure we should be paying that cost on every devstack job run.
we could maybe compile it once and bake the package into the image or host it on a mirror but i think we should avoid this option if we have alternitives.
Since this patch https://review.opendev.org/c/openstack/devstack/+/763402 we no longer default to compiling OVN from source anymore, it's installed using the distro packages now. Yeah the alternatives are not straight forward, I was talking to some core OVN folks yesterday regarding the backports proposed by Canonical to the 20.03 branch and they seem to be fine with it, it needs more reviews since there are around ~20 patches being backported there. But I hope they are going to be looking into it and we should get a new OVN package for Ubuntu Focal soon.
What do you think about it ?
Here is the test patch for Cinder: https://review.opendev.org/c/openstack/cinder/+/748227
* Nova:
There are a few patches waiting for review for Nova, which are:
1- Adapting the live migration scripts to work with ML2/OVN: Basically the scripts were trying to stop the Neutron agent (q-agt) process which is not part of an ML2/OVN deployment. The patch changes the code to check if that system unit exists before trying to stop it.
Patch: https://review.opendev.org/c/openstack/nova/+/776419
2- Explicitly set grenade job to ML2/OVS: This is a temporary change which can be removed one release cycle after we switch DevStack to ML2/OVN. Grenade will test updating from the release version to the master branch but, since the default of the released version is not ML2/OVN, upgrading from ML2/OVS to ML2/OVN as part of the grenade job is not supported.
Patch: https://review.opendev.org/c/openstack/nova/+/776934
3- Explicitly set nova-next job to ML2/OVS: This job uses the QoS minimum bandwidth feature which is not yet supported by ML2/OVN [5][6] therefore we are temporarily enabling ML2/OVS for this job until that feature lands in core OVN.
Patch: https://review.opendev.org/c/openstack/nova/+/776944
I also spoke briefly with Sean Mooney (irc: sean-k-mooney) about these changes and he suggested keeping all the Nova jobs on ML2/OVS for now because he feels like a change in the default network driver a few weeks prior to the upstream code freeze can be concerning. We do not know yet precisely when we are changing the default due to the current patches we need to get merged but, if this is a shared feeling among the Nova community I can work on enabling ML2/OVS on all jobs in Nova until we get a new release in OpenStack.
yep this is still my view. i would suggest we do the work required in the repos but not merge it until the xena release is open. thats technically at RC1 so march 25th i think we can safely do the swich after that but i would not change the defualt in any project before then.
Yeah, no problem waiting from my side, but hoping we can keep reviewing the rest of the patches until then.
Here's the test patch for Nova: https://review.opendev.org/c/openstack/nova/+/776945
* DevStack:
And this is the final patch that will make this all happen: https://review.opendev.org/c/openstack/devstack/+/735097
It changes the default in DevStack from ML2/OVS to ML2/OVN. It's been a long and bumpy road to get to this point and I would like to say thanks to everyone involved so far and everyone that read the whole email, please let me know your thoughts.
thanks for working on this.
Thanks for the inputs
[0] https://etherpad.opendev.org/p/neutron-victoria-ptg [1] https://bugs.launchpad.net/tempest/+bug/1728886 [2] https://patchwork.ozlabs.org/project/openvswitch/patch/20200319122641.473776... [3] https://github.com/ovn-org/ovn/commit/0c26bc03064f2c21d208f0f860b48d8ab39380... [4] https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050961.html [5] https://github.com/openstack/nova/blob/ded25f33c734ebff963f06984707a99fe76a9... [6] https://docs.openstack.org/neutron/latest/ovn/gaps.html
Cheers, Lucas
Hi, Dnia środa, 3 marca 2021 11:32:11 CET Lucas Alvares Gomes pisze:
On Mon, Mar 1, 2021 at 6:25 PM Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2021-03-01 at 16:07 +0000, Lucas Alvares Gomes wrote:
Hi all,
As part of the Victoria PTG [0] the Neutron community agreed upon switching the default backend in Devstack to OVN. A lot of work has been done since, from porting the OVN devstack module to the DevStack tree, refactoring the DevStack module to install OVN from distro packages, implementing features to close the parity gap with ML2/OVS, fixing issues with tests and distros, etc...
We are now very close to being able to make the switch and we've thought about sending this email to the broader community to raise awareness about this change as well as bring more attention to the patches that are current on review.
Note that moving DevStack to ML2/OVN does not mean that ML2/OVS is discontinued and/or not supported anymore. The ML2/OVS driver is still going to be developed and maintained by the upstream Neutron community.
can we ensure that this does not happen until the xena release. in generall i think its ok to change the default but not this late in the cycle. i would also like to ensure we keep at least one non ovn based multi node job in nova until https://review.opendev.org/c/openstack/nova/+/602432 is merged and possible after. right now the event/neutorn interaction is not the same during move operations. I think it's fair to wait for the new release cycle to start since we are just a few weeks away and then we can flip the default in DevStack. I will state this in the last DevStack patch and set the workflow -1 until then. That said, I also think that other patches could be merged before that, those are just adapting a few scripts to work with ML2/OVN and enabling ML2/OVS explicitly where it makes sense. That way, when time comes, we will just need to merge the DevStack patch.
+1. Let's do that very early in Xena cycle :)
Below is a e per project explanation with relevant links and issues of where we stand with this work right now:
* Keystone:
Everything should be good for Keystone, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/keystone/+/777963
* Glance:
Everything should be good for Glace, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/glance/+/748390
* Swift:
Everything should be good for Swift, the gate is happy with the changes. Here is the test patch: https://review.opendev.org/c/openstack/swift/+/748403
* Ironic:
Since chainloading iPXE by the OVN built-in DHCP server is work in progress, we've changed most of the Ironic jobs to explicitly enable ML2/OVS and everything is merged, so we should be good for Ironic too. Here is the test patch: https://review.opendev.org/c/openstack/ironic/+/748405
* Cinder:
Cinder is almost complete. There's one test failure in the "tempest-slow-py3" job run on the "test_port_security_macspoofing_port" test.
This failure is due to a bug in core OVN [1]. This bug has already been fixed upstream [2] and the fix has been backported down to the branch-20.03 [3] of the OVN project. However, since we install OVN from packages we are currently waiting for this fix to be included in the packages for Ubuntu Focal (it's based on OVN 20.03). I already contacted the package maintainer which has been very supportive of this work and will work on the package update, but he maintain a handful of backports in that package which is not yet included in OVN 20.03 upstream and he's now working with the core OVN community [4] to include it first in the branch and then create a new package for it. Hopefully this will happen soon.
But for now we have a few options moving on with this issue:
1- Wait for the new package version 2- Mark the test as unstable until we get the new package version 3- Compile OVN from source instead of installing it from packages (OVN_BUILD_FROM_SOURCE=True in local.conf)
i dont think we should default to ovn untill a souce build is not required. compiling form souce while not supper expensice still adds time to the job execution and im not sure we should be paying that cost on every devstack job run.
we could maybe compile it once and bake the package into the image or host it on a mirror but i think we should avoid this option if we have alternitives.
Since this patch https://review.opendev.org/c/openstack/devstack/+/763402 we no longer default to compiling OVN from source anymore, it's installed using the distro packages now.
Yeah the alternatives are not straight forward, I was talking to some core OVN folks yesterday regarding the backports proposed by Canonical to the 20.03 branch and they seem to be fine with it, it needs more reviews since there are around ~20 patches being backported there. But I hope they are going to be looking into it and we should get a new OVN package for Ubuntu Focal soon.
What do you think about it ?
Here is the test patch for Cinder: https://review.opendev.org/c/openstack/cinder/+/748227
* Nova:
There are a few patches waiting for review for Nova, which are:
1- Adapting the live migration scripts to work with ML2/OVN: Basically the scripts were trying to stop the Neutron agent (q-agt) process which is not part of an ML2/OVN deployment. The patch changes the code to check if that system unit exists before trying to stop it.
Patch: https://review.opendev.org/c/openstack/nova/+/776419
2- Explicitly set grenade job to ML2/OVS: This is a temporary change which can be removed one release cycle after we switch DevStack to ML2/OVN. Grenade will test updating from the release version to the master branch but, since the default of the released version is not ML2/OVN, upgrading from ML2/OVS to ML2/OVN as part of the grenade job is not supported.
Patch: https://review.opendev.org/c/openstack/nova/+/776934
3- Explicitly set nova-next job to ML2/OVS: This job uses the QoS minimum bandwidth feature which is not yet supported by ML2/OVN [5][6] therefore we are temporarily enabling ML2/OVS for this job until that feature lands in core OVN.
Patch: https://review.opendev.org/c/openstack/nova/+/776944
I also spoke briefly with Sean Mooney (irc: sean-k-mooney) about these changes and he suggested keeping all the Nova jobs on ML2/OVS for now because he feels like a change in the default network driver a few weeks prior to the upstream code freeze can be concerning. We do not know yet precisely when we are changing the default due to the current patches we need to get merged but, if this is a shared feeling among the Nova community I can work on enabling ML2/OVS on all jobs in Nova until we get a new release in OpenStack.
yep this is still my view. i would suggest we do the work required in the repos but not merge it until the xena release is open. thats technically at RC1 so march 25th i think we can safely do the swich after that but i would not change the defualt in any project before then.
Yeah, no problem waiting from my side, but hoping we can keep reviewing the rest of the patches until then.
Here's the test patch for Nova: https://review.opendev.org/c/openstack/nova/+/776945
* DevStack:
And this is the final patch that will make this all happen: https://review.opendev.org/c/openstack/devstack/+/735097
It changes the default in DevStack from ML2/OVS to ML2/OVN. It's been a long and bumpy road to get to this point and I would like to say thanks to everyone involved so far and everyone that read the whole email, please let me know your thoughts.
thanks for working on this.
Thanks for the inputs
[0] https://etherpad.opendev.org/p/neutron-victoria-ptg [1] https://bugs.launchpad.net/tempest/+bug/1728886 [2] https://patchwork.ozlabs.org/project/openvswitch/patch/20200319122641.4 73776-1-numans@ovn.org/ [3] https://github.com/ovn-org/ovn/commit/0c26bc03064f2c21d208f0f860b48d8ab 39380cb [4] https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050961 .html [5] https://github.com/openstack/nova/blob/ded25f33c734ebff963f06984707a99f e76a9ee1/gate/post_test_hook.sh#L129-L143 [6] https://docs.openstack.org/neutron/latest/ovn/gaps.html
Cheers, Lucas
-- Slawek Kaplonski Principal Software Engineer Red Hat
participants (8)
-
Brian Haley
-
Goutham Pacha Ravi
-
Lingxian Kong
-
Lucas Alvares Gomes
-
Michael Johnson
-
Sean Mooney
-
Slawek Kaplonski
-
Tom Barron