Open Stack

On Wed, Mar 3, 2021 at 7:35 AM Slawek Kaplonski <skaplons at redhat.com> wrote:
>
> Hi,
>
> Dnia środa, 3 marca 2021 05:01:24 CET Sean Mooney pisze:
> > On Wed, 2021-03-03 at 15:59 +1300, Lingxian Kong wrote:
> > > Hi,
> > >
> > > Thanks for all your hard work on this.
> > >
> > > I'm wondering is there any doc proposed for devstack to tell people who
> > > are
> > > not interested in OVN to keep the current devstack behaviour? I have a
> > > feeling that using OVN as default Neutron driver would break the CI jobs
> > > for some projects like Octavia, Trove, etc. which rely on ovs port for the
> > > set up.
>
> You can look on how we set some of our jobs to use ML2/OVS: https://
> review.opendev.org/c/openstack/neutron-tempest-plugin/+/749503/26/zuul.d/
> master_jobs.yaml#121

Thanks Lingxian and Slaweq for the suggestion/replies.

Apart from what Slaweq pointed out, the DevStack documentation have a
sample config file
(https://docs.openstack.org/devstack/latest/#create-a-local-conf) as
part of the steps to deploy it, I was thinking I could propose another
sample file that would enable ML2/OVS for the deployment and add a
note in that doc; if you think it's worth it.

Also what gives us more confidence regarding projects like Octavia is
that TripleO already uses ML2/OVN as the default driver for their
overcloud, so many of these projects are already being tested/used
with OVN.

>
> >
> > well ovn is just an alternivie contoler for ovs.
> > so ovn replace the neutron l2 agent it does not replace ovs.
> > project like octavia or trove that deploy loadblances or dbs in vms should
> > not be able to observe a difference. they may still want to deploy ml2/ovs
> > but unless they are doing something directly on the host like adding port
> > directly to ovs because they are not using vms they should not be aware of
> > this change.
> >
> > my reticense to cahnge at this point in the cyle for nova is motiveated
> > maily by gate stablity. the active contibutes to nova dont really have
> > experince with ovn and how to debug it in the gate. we also are getting
> > close to FF when we tend to get a lot of patches and the gate stablity
> > become even more imporant so adding a new vaiabrly to that mix but swaping
> > out the networkbacked between now and the wallaby release seams
> > problematic.
> >
> > in any cases swapping back should ideallly be as simple as setting
> > Q_AGENT=openvswitch.
> >
> > i have not looked at the patches but to swap betwween ovs and linuxbidge you
> > just define Q_AGENT=linuxbridge for the most part so im expecting that we
> > would just enable Q_AGENT=ovn or somthing simialr for ovn.
> >
> > i know ovn used to have its own devstack plugin but if we are makeing it the
> > default that means it need to be support nativly in devstack not  as a
> > plugin so useing Q_AGENT=ovn to enable it and make that the new default
> > would seam to be the simplest way to manage that.
> >
> > but yes documenting how to enabel the old behavior is still important
> > the example nova patch shows how to hardcode the old behavior
> > https://review.opendev.org/c/openstack/nova/+/776944/5/.zuul.yaml#234
> > it seams to be doing this a little more explictly then i would like but its
> > not that hard. i would suggest adding a second sample loca.conf in devstack
> > for standard ovs deployments.
> > > ---
> > > Lingxian Kong
> > > Senior Cloud Engineer (Catalyst Cloud)
> > > Trove PTL (OpenStack)
> > > OpenStack Cloud Provider Co-Lead (Kubernetes)
> > >
> > >
> > > On Wed, Mar 3, 2021 at 2:03 PM Goutham Pacha Ravi <gouthampravi at gmail.com>
> > >
> > > wrote:
> > > > On Mon, Mar 1, 2021 at 10:29 AM Sean Mooney <smooney at redhat.com> wrote:
> > > > > On Mon, 2021-03-01 at 16:07 +0000, Lucas Alvares Gomes wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > As part of the Victoria PTG [0] the Neutron community agreed upon
> > > > > > switching the default backend in Devstack to OVN. A lot of work has
> > > > > > been done since, from porting the OVN devstack module to the
> > > > > > DevStack
> > > > > > tree, refactoring the DevStack module to install OVN from distro
> > > > > > packages, implementing features to close the parity gap with
> > > > > > ML2/OVS,
> > > > > > fixing issues with tests and distros, etc...
> > > > > >
> > > > > > We are now very close to being able to make the switch and we've
> > > > > > thought about sending this email to the broader community to raise
> > > > > > awareness about this change as well as bring more attention to the
> > > > > > patches that are current on review.
> > > > > >
> > > > > > Note that moving DevStack to ML2/OVN does not mean that ML2/OVS is
> > > > > > discontinued and/or not supported anymore. The ML2/OVS driver is
> > > > > > still
> > > > > > going to be developed and maintained by the upstream Neutron
> > > > > > community.
> > > > >
> > > > > can we ensure that this does not happen until the xena release.
> > > > > in generall i think its ok to change the default but not this late in
> > > >
> > > > the cycle.
> > > >
> > > > > i would also like to ensure we keep at least one non ovn based multi
> > > >
> > > > node job in
> > > >
> > > > > nova until https://review.opendev.org/c/openstack/nova/+/602432 is
> > > >
> > > > merged and possible after.
> > > >
> > > > > right now the event/neutorn interaction is not the same during move
> > > >
> > > > operations.
> > > >
> > > > > > Below is a e per project explanation with relevant links and issues
> > > > > > of
> > > > > > where we stand with this work right now:
> > > > > >
> > > > > > * Keystone:
> > > > > >
> > > > > > Everything should be good for Keystone, the gate is happy with the
> > > > > > changes. Here is the test patch:
> > > > > > https://review.opendev.org/c/openstack/keystone/+/777963
> > > > > >
> > > > > > * Glance:
> > > > > >
> > > > > > Everything should be good for Glace, the gate is happy with the
> > > > > > changes. Here is the test patch:
> > > > > > https://review.opendev.org/c/openstack/glance/+/748390
> > > > > >
> > > > > > * Swift:
> > > > > >
> > > > > > Everything should be good for Swift, the gate is happy with the
> > > > > > changes. Here is the test patch:
> > > > > > https://review.opendev.org/c/openstack/swift/+/748403
> > > > > >
> > > > > > * Ironic:
> > > > > >
> > > > > > Since chainloading iPXE by the OVN built-in DHCP server is work in
> > > > > > progress, we've changed most of the Ironic jobs to explicitly enable
> > > > > > ML2/OVS and everything is merged, so we should be good for Ironic
> > > > > > too.
> > > > > > Here is the test patch:
> > > > > > https://review.opendev.org/c/openstack/ironic/+/748405
> > > > > >
> > > > > > * Cinder:
> > > > > >
> > > > > > Cinder is almost complete. There's one test failure in the
> > > > > > "tempest-slow-py3" job run on the
> > > > > > "test_port_security_macspoofing_port" test.
> > > > > >
> > > > > > This failure is due to a bug in core OVN [1]. This bug has already
> > > > > > been fixed upstream [2] and the fix has been backported down to the
> > > > > > branch-20.03 [3] of the OVN project. However, since we install OVN
> > > > > > from packages we are currently waiting for this fix to be included
> > > > > > in
> > > > > > the packages for Ubuntu Focal (it's based on OVN 20.03). I already
> > > > > > contacted the package maintainer which has been very supportive of
> > > > > > this work and will work on the package update, but he maintain a
> > > > > > handful of backports in that package which is not yet included in
> > > > > > OVN
> > > > > > 20.03 upstream and he's now working with the core OVN community [4]
> > > > > > to
> > > > > > include it first in the branch and then create a new package for it.
> > > > > > Hopefully this will happen soon.
> > > > > >
> > > > > > But for now we have a few options moving on with this issue:
> > > > > >
> > > > > > 1- Wait for the new package version
> > > > > > 2- Mark the test as unstable until we get the new package version
> > > > > > 3- Compile OVN from source instead of installing it from packages
> > > > > > (OVN_BUILD_FROM_SOURCE=True in local.conf)
> > > > >
> > > > > i dont think we should default to ovn untill a souce build is not
> > > >
> > > > required.
> > > >
> > > > > compiling form souce while not supper expensice still adds time to the
> > > >
> > > > job
> > > >
> > > > > execution and im not sure we should be paying that cost on every
> > > >
> > > > devstack job run.
> > > >
> > > > > we could maybe compile it once and bake the package into the image or
> > > >
> > > > host it on a mirror
> > > >
> > > > > but i think we should avoid this option if we have alternitives.
> > > > >
> > > > > > What do you think about it ?
> > > > > >
> > > > > > Here is the test patch for Cinder:
> > > > > > https://review.opendev.org/c/openstack/cinder/+/748227
> > > > > >
> > > > > > * Nova:
> > > > > >
> > > > > > There are a few patches waiting for review for Nova, which are:
> > > > > >
> > > > > > 1- Adapting the live migration scripts to work with ML2/OVN:
> > > > > > Basically
> > > > > > the scripts were trying to stop the Neutron agent (q-agt) process
> > > > > > which is not part of an ML2/OVN deployment. The patch changes the
> > > > > > code
> > > > > > to check if that system unit exists before trying to stop it.
> > > > > >
> > > > > > Patch: https://review.opendev.org/c/openstack/nova/+/776419
> > > > > >
> > > > > > 2- Explicitly set grenade job to ML2/OVS: This is a temporary change
> > > > > > which can be removed one release cycle after we switch DevStack to
> > > > > > ML2/OVN. Grenade will test updating from the release version to the
> > > > > > master branch but, since the default of the released version is not
> > > > > > ML2/OVN, upgrading from ML2/OVS to ML2/OVN as part of the grenade
> > > > > > job
> > > > > > is not supported.
> > > > > >
> > > > > > Patch: https://review.opendev.org/c/openstack/nova/+/776934
> > > > > >
> > > > > > 3- Explicitly set nova-next job to ML2/OVS: This job uses the QoS
> > > > > > minimum bandwidth feature which is not yet supported by ML2/OVN
> > > > > > [5][6]
> > > > > > therefore we are temporarily enabling ML2/OVS for this job until
> > > > > > that
> > > > > > feature lands in core OVN.
> > > > > >
> > > > > > Patch: https://review.opendev.org/c/openstack/nova/+/776944
> > > > > >
> > > > > > I also spoke briefly with Sean Mooney (irc: sean-k-mooney) about
> > > > > > these
> > > > > > changes and he suggested keeping all the Nova jobs on ML2/OVS for
> > > > > > now
> > > > > > because he feels like a change in the default network driver a few
> > > > > > weeks prior to the upstream code freeze can be concerning. We do not
> > > > > > know yet precisely when we are changing the default due to the
> > > > > > current
> > > > > > patches we need to get merged but, if this is a shared feeling among
> > > > > > the Nova community I can work on enabling ML2/OVS on all jobs in
> > > > > > Nova
> > > > > > until we get a new release in OpenStack.
> > > > >
> > > > > yep this is still my view.
> > > > > i would suggest we do the work required in the repos but not merge it
> > > >
> > > > until the xena release
> > > >
> > > > > is open. thats technically at RC1 so march 25th
> > > > > i think we can safely do the swich after that but i would not change
> > > > > the
> > > >
> > > > defualt in any project
> > > >
> > > > > before then.
> > > > >
> > > > > > Here's the test patch for Nova:
> > > > > > https://review.opendev.org/c/openstack/nova/+/776945
> > > > > >
> > > > > > * DevStack:
> > > > > >
> > > > > > And this is the final patch that will make this all happen:
> > > > > > https://review.opendev.org/c/openstack/devstack/+/735097
> > > > > >
> > > > > > It changes the default in DevStack from ML2/OVS to ML2/OVN. It's
> > > > > > been
> > > > > > a long and bumpy road to get to this point and I would like to say
> > > > > > thanks to everyone involved so far and everyone that read the whole
> > > > > > email, please let me know your thoughts.
> > > > >
> > > > > thanks for working on this.
> > > > >
> > > > > > [0] https://etherpad.opendev.org/p/neutron-victoria-ptg
> > > > > > [1] https://bugs.launchpad.net/tempest/+bug/1728886
> > > > > > [2]
> > > >
> > > > https://patchwork.ozlabs.org/project/openvswitch/patch/20200319122641.47
> > > > 3776-1-numans at ovn.org/> >
> > > > > > [3]
> > > >
> > > > https://github.com/ovn-org/ovn/commit/0c26bc03064f2c21d208f0f860b48d8ab3
> > > > 9380cb> >
> > > > > > [4]
> > > >
> > > > https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050961.
> > > > html> >
> > > > > > [5]
> > > >
> > > > https://github.com/openstack/nova/blob/ded25f33c734ebff963f06984707a99fe
> > > > 76a9ee1/gate/post_test_hook.sh#L129-L143> >
> > > > > > [6] https://docs.openstack.org/neutron/latest/ovn/gaps.html
> > > > > >
> > > > > > Cheers,
> > > > > > Lucas
> > > >
> > > > ++ Thank you indeed for working diligently on this important change.
> > > >
> > > > Please do note that devstack, and the base job that you're modifying
> > > > is used by many other projects besides the ones that you have
> > > > enumerated in the subject line.
> > > > I suggest using [all] as a better subject line indicator to get the
> > > > attention of folks like me who have filters based on the subject line.
> > > > Also, the network substrate is important for the project I help
> > > > maintain: Manila, which provides shared file systems over a network -
> > > > so I followed your lead and submitted a dependent patch. I hope to
> > > > reach out to you in case we see some breakages:
> > > > https://review.opendev.org/c/openstack/manila-tempest-plugin/+/778346
>
>
> --
> Slawek Kaplonski
> Principal Software Engineer
> Red Hat