Open Stack

Hi all,

As part of the Victoria PTG [0] the Neutron community agreed upon
switching the default backend in Devstack to OVN. A lot of work has
been done since, from porting the OVN devstack module to the DevStack
tree, refactoring the DevStack module to install OVN from distro
packages, implementing features to close the parity gap with ML2/OVS,
fixing issues with tests and distros, etc...

We are now very close to being able to make the switch and we've
thought about sending this email to the broader community to raise
awareness about this change as well as bring more attention to the
patches that are current on review.

Note that moving DevStack to ML2/OVN does not mean that ML2/OVS is
discontinued and/or not supported anymore. The ML2/OVS driver is still
going to be developed and maintained by the upstream Neutron
community.

Below is a e per project explanation with relevant links and issues of
where we stand with this work right now:

* Keystone:

Everything should be good for Keystone, the gate is happy with the
changes. Here is the test patch:
https://review.opendev.org/c/openstack/keystone/+/777963

* Glance:

Everything should be good for Glace, the gate is happy with the
changes. Here is the test patch:
https://review.opendev.org/c/openstack/glance/+/748390

* Swift:

Everything should be good for Swift, the gate is happy with the
changes. Here is the test patch:
https://review.opendev.org/c/openstack/swift/+/748403

* Ironic:

Since chainloading iPXE by the OVN built-in DHCP server is work in
progress, we've changed most of the Ironic jobs to explicitly enable
ML2/OVS and everything is merged, so we should be good for Ironic too.
Here is the test patch:
https://review.opendev.org/c/openstack/ironic/+/748405

* Cinder:

Cinder is almost complete. There's one test failure in the
"tempest-slow-py3" job run on the
"test_port_security_macspoofing_port" test.

This failure is due to a bug in core OVN [1]. This bug has already
been fixed upstream [2] and the fix has been backported down to the
branch-20.03 [3] of the OVN project. However, since we install OVN
from packages we are currently waiting for this fix to be included in
the packages for Ubuntu Focal (it's based on OVN 20.03). I already
contacted the package maintainer which has been very supportive of
this work and will work on the package update, but he maintain a
handful of backports in that package which is not yet included in OVN
20.03 upstream and he's now working with the core OVN community [4] to
include it first in the branch and then create a new package for it.
Hopefully this will happen soon.

But for now we have a few options moving on with this issue:

1- Wait for the new package version
2- Mark the test as unstable until we get the new package version
3- Compile OVN from source instead of installing it from packages
(OVN_BUILD_FROM_SOURCE=True in local.conf)

What do you think about it ?

Here is the test patch for Cinder:
https://review.opendev.org/c/openstack/cinder/+/748227

* Nova:

There are a few patches waiting for review for Nova, which are:

1- Adapting the live migration scripts to work with ML2/OVN: Basically
the scripts were trying to stop the Neutron agent (q-agt) process
which is not part of an ML2/OVN deployment. The patch changes the code
to check if that system unit exists before trying to stop it.

Patch: https://review.opendev.org/c/openstack/nova/+/776419

2- Explicitly set grenade job to ML2/OVS: This is a temporary change
which can be removed one release cycle after we switch DevStack to
ML2/OVN. Grenade will test updating from the release version to the
master branch but, since the default of the released version is not
ML2/OVN, upgrading from ML2/OVS to ML2/OVN as part of the grenade job
is not supported.

Patch: https://review.opendev.org/c/openstack/nova/+/776934

3- Explicitly set nova-next job to ML2/OVS: This job uses the QoS
minimum bandwidth feature which is not yet supported by ML2/OVN [5][6]
therefore we are temporarily enabling ML2/OVS for this job until that
feature lands in core OVN.

Patch: https://review.opendev.org/c/openstack/nova/+/776944

I also spoke briefly with Sean Mooney (irc: sean-k-mooney) about these
changes and he suggested keeping all the Nova jobs on ML2/OVS for now
because he feels like a change in the default network driver a few
weeks prior to the upstream code freeze can be concerning. We do not
know yet precisely when we are changing the default due to the current
patches we need to get merged but, if this is a shared feeling among
the Nova community I can work on enabling ML2/OVS on all jobs in Nova
until we get a new release in OpenStack.

Here's the test patch for Nova:
https://review.opendev.org/c/openstack/nova/+/776945

* DevStack:

And this is the final patch that will make this all happen:
https://review.opendev.org/c/openstack/devstack/+/735097

It changes the default in DevStack from ML2/OVS to ML2/OVN. It's been
a long and bumpy road to get to this point and I would like to say
thanks to everyone involved so far and everyone that read the whole
email, please let me know your thoughts.

[0] https://etherpad.opendev.org/p/neutron-victoria-ptg
[1] https://bugs.launchpad.net/tempest/+bug/1728886
[2] https://patchwork.ozlabs.org/project/openvswitch/patch/20200319122641.473776-1-numans@ovn.org/
[3] https://github.com/ovn-org/ovn/commit/0c26bc03064f2c21d208f0f860b48d8ab39380cb
[4] https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050961.html
[5] https://github.com/openstack/nova/blob/ded25f33c734ebff963f06984707a99fe76a9ee1/gate/post_test_hook.sh#L129-L143
[6] https://docs.openstack.org/neutron/latest/ovn/gaps.html

Cheers,
Lucas