Open Stack

On Mon, Mar 1, 2021 at 6:25 PM Sean Mooney <smooney at redhat.com> wrote:
>
> On Mon, 2021-03-01 at 16:07 +0000, Lucas Alvares Gomes wrote:
> > Hi all,
> >
> > As part of the Victoria PTG [0] the Neutron community agreed upon
> > switching the default backend in Devstack to OVN. A lot of work has
> > been done since, from porting the OVN devstack module to the DevStack
> > tree, refactoring the DevStack module to install OVN from distro
> > packages, implementing features to close the parity gap with ML2/OVS,
> > fixing issues with tests and distros, etc...
> >
> > We are now very close to being able to make the switch and we've
> > thought about sending this email to the broader community to raise
> > awareness about this change as well as bring more attention to the
> > patches that are current on review.
> >
> > Note that moving DevStack to ML2/OVN does not mean that ML2/OVS is
> > discontinued and/or not supported anymore. The ML2/OVS driver is still
> > going to be developed and maintained by the upstream Neutron
> > community.
> can we ensure that this does not happen until the xena release.
> in generall i think its ok to change the default but not this late in the cycle.
> i would also like to ensure we keep at least one non ovn based multi node job in
> nova until https://review.opendev.org/c/openstack/nova/+/602432 is merged and possible after.
> right now the event/neutorn interaction is not the same during move operations.

I think it's fair to wait for the new release cycle to start since we
are just a few weeks away and then we can flip the default in
DevStack. I will state this in the last DevStack patch and set the
workflow -1 until then. That said, I also think that other patches
could be merged before that, those are just adapting a few scripts to
work with ML2/OVN and enabling ML2/OVS explicitly where it makes
sense. That way, when time comes, we will just need to merge the
DevStack patch.

> >
> > Below is a e per project explanation with relevant links and issues of
> > where we stand with this work right now:
> >
> > * Keystone:
> >
> > Everything should be good for Keystone, the gate is happy with the
> > changes. Here is the test patch:
> > https://review.opendev.org/c/openstack/keystone/+/777963
> >
> > * Glance:
> >
> > Everything should be good for Glace, the gate is happy with the
> > changes. Here is the test patch:
> > https://review.opendev.org/c/openstack/glance/+/748390
> >
> > * Swift:
> >
> > Everything should be good for Swift, the gate is happy with the
> > changes. Here is the test patch:
> > https://review.opendev.org/c/openstack/swift/+/748403
> >
> > * Ironic:
> >
> > Since chainloading iPXE by the OVN built-in DHCP server is work in
> > progress, we've changed most of the Ironic jobs to explicitly enable
> > ML2/OVS and everything is merged, so we should be good for Ironic too.
> > Here is the test patch:
> > https://review.opendev.org/c/openstack/ironic/+/748405
> >
> > * Cinder:
> >
> > Cinder is almost complete. There's one test failure in the
> > "tempest-slow-py3" job run on the
> > "test_port_security_macspoofing_port" test.
> >
> > This failure is due to a bug in core OVN [1]. This bug has already
> > been fixed upstream [2] and the fix has been backported down to the
> > branch-20.03 [3] of the OVN project. However, since we install OVN
> > from packages we are currently waiting for this fix to be included in
> > the packages for Ubuntu Focal (it's based on OVN 20.03). I already
> > contacted the package maintainer which has been very supportive of
> > this work and will work on the package update, but he maintain a
> > handful of backports in that package which is not yet included in OVN
> > 20.03 upstream and he's now working with the core OVN community [4] to
> > include it first in the branch and then create a new package for it.
> > Hopefully this will happen soon.
> >
> > But for now we have a few options moving on with this issue:
> >
> > 1- Wait for the new package version
> > 2- Mark the test as unstable until we get the new package version
> > 3- Compile OVN from source instead of installing it from packages
> > (OVN_BUILD_FROM_SOURCE=True in local.conf)
> i dont think we should default to ovn untill a souce build is not required.
> compiling form souce while not supper expensice still adds time to the job
> execution and im not sure we should be paying that cost on every devstack job run.
>
> we could maybe compile it once and bake the package into the image or host it on a mirror
> but i think we should avoid this option if we have alternitives.

Since this patch
https://review.opendev.org/c/openstack/devstack/+/763402 we no longer
default to compiling OVN from source anymore, it's installed using the
distro packages now.

Yeah the alternatives are not straight forward, I was talking to some
core OVN folks yesterday regarding the backports proposed by Canonical
to the 20.03 branch and they seem to be fine with it, it needs more
reviews since there are around ~20 patches being backported there. But
I hope they are going to be looking into it and we should get a new
OVN package for Ubuntu Focal soon.

> >
> > What do you think about it ?
> >
> > Here is the test patch for Cinder:
> > https://review.opendev.org/c/openstack/cinder/+/748227
> >
> > * Nova:
> >
> > There are a few patches waiting for review for Nova, which are:
> >
> > 1- Adapting the live migration scripts to work with ML2/OVN: Basically
> > the scripts were trying to stop the Neutron agent (q-agt) process
> > which is not part of an ML2/OVN deployment. The patch changes the code
> > to check if that system unit exists before trying to stop it.
> >
> > Patch: https://review.opendev.org/c/openstack/nova/+/776419
> >
> > 2- Explicitly set grenade job to ML2/OVS: This is a temporary change
> > which can be removed one release cycle after we switch DevStack to
> > ML2/OVN. Grenade will test updating from the release version to the
> > master branch but, since the default of the released version is not
> > ML2/OVN, upgrading from ML2/OVS to ML2/OVN as part of the grenade job
> > is not supported.
> >
> > Patch: https://review.opendev.org/c/openstack/nova/+/776934
> >
> > 3- Explicitly set nova-next job to ML2/OVS: This job uses the QoS
> > minimum bandwidth feature which is not yet supported by ML2/OVN [5][6]
> > therefore we are temporarily enabling ML2/OVS for this job until that
> > feature lands in core OVN.
> >
> > Patch: https://review.opendev.org/c/openstack/nova/+/776944
> >
> > I also spoke briefly with Sean Mooney (irc: sean-k-mooney) about these
> > changes and he suggested keeping all the Nova jobs on ML2/OVS for now
> > because he feels like a change in the default network driver a few
> > weeks prior to the upstream code freeze can be concerning. We do not
> > know yet precisely when we are changing the default due to the current
> > patches we need to get merged but, if this is a shared feeling among
> > the Nova community I can work on enabling ML2/OVS on all jobs in Nova
> > until we get a new release in OpenStack.
> yep this is still my view.
> i would suggest we do the work required in the repos but not merge it until the xena release
> is open. thats technically at RC1 so march 25th
> i think we can safely do the swich after that but i would not change the defualt in any project
> before then.

Yeah, no problem waiting from my side, but hoping we can keep
reviewing the rest of the patches until then.

> >
> > Here's the test patch for Nova:
> > https://review.opendev.org/c/openstack/nova/+/776945
> >
> > * DevStack:
> >
> > And this is the final patch that will make this all happen:
> > https://review.opendev.org/c/openstack/devstack/+/735097
> >
> > It changes the default in DevStack from ML2/OVS to ML2/OVN. It's been
> > a long and bumpy road to get to this point and I would like to say
> > thanks to everyone involved so far and everyone that read the whole
> > email, please let me know your thoughts.
> thanks for working on this.

Thanks for the inputs

> >
> > [0] https://etherpad.opendev.org/p/neutron-victoria-ptg
> > [1] https://bugs.launchpad.net/tempest/+bug/1728886
> > [2] https://patchwork.ozlabs.org/project/openvswitch/patch/20200319122641.473776-1-numans@ovn.org/
> > [3] https://github.com/ovn-org/ovn/commit/0c26bc03064f2c21d208f0f860b48d8ab39380cb
> > [4] https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050961.html
> > [5] https://github.com/openstack/nova/blob/ded25f33c734ebff963f06984707a99fe76a9ee1/gate/post_test_hook.sh#L129-L143
> > [6] https://docs.openstack.org/neutron/latest/ovn/gaps.html
> >
> > Cheers,
> > Lucas
> >
>
>
>