Thanks Jakub for starting this thread, and sorry to get late on to my reply, getting today as Eduardo reminded about that patch to me.
Hi all,
I wanted to bring this topic up for a discussion again.
I’ve been working on a Neutron feature [1] for some time, the feature
requires the BGP support in OVN. The functional job currently uses
a moving OVN branch based on the OVN LTS version - which is 24.03 -
almost 2 years old. I tried to change this and requested a patch to
switch to using a pinned OVN mostly for two benefits:
- we won’t get gate broken if a bad patch is backported to the branch-24.03
- I get the BGP features needed for the work on [1]
The patch was denied to keep the policy we set on ourselves - ie. to go with the branch based
on the OVN LTS version. The workaround was to skip multiple tests that rely on the OVN
changes specific to BGP. This is not ideal as now majority of the functionality is
not tested and we’re just waiting for the LTS release, which may bring some surprises.
As the original reviewer, I wanted to clarify that I never denied the patch; I only sought more clarification and others' opinions to reach a consensus, as others stated regarding the coverage we have in the periodics/experimental pipeline. That patch was later abandoned. Perhaps I didn't communicate clearly, so I apologize for that.
This was a concrete example but if we put it into a more general situation, in Neutron
we heavily rely on OVN and its features and any development of a potential feature that
requires some work in OVN gets cumbersome because we can’t validate our code against
OVN until the point a next OVN LTS version is released, which in the worst case scenario
can take up to 2 years.
Moving to new versions based on a required feature is quite valid.
Hence I propose a change and use a pinned hash from the OVN tree that
is updated on two occasions:
- A next OVN version is released (non-LTS)
- We’re developing a feature that relies on bleeding edge OVN - then we pick a good
hash and pin to it in the functional job definition (not in the scripts)
- I see you pushed an update to switch to branch-26.03 (an upcoming LTS) so i think it's fine we can keep it as it is and
Don't pin to a tag release, as the last few years show it's gone well for us, i.e We get new
fixes in those branches and hasn't seen any breakages yet. However, we can revisit this later if we see any difference in the trend. And if another requirement comes in future it's totally find to rebump even to non LTS for functional tests.
- We currently pin functional scripts so that the local setup uses the same OVS/OVN versions as CI.
This would apply solely to the functional tests because tempest jobs
can define their own OVN version and unit tests should be isolated from the used
OVN version, and only to the master branch - because that is the development branch
after all.
+1
Such a change brings multiple benefits, we won’t need to skip tests if
the required functionality is not present in current OVN version and have
the development go smoother.
We may uncover issues in OVN early.
For this reason and to catch any regressions, we also run periodic/experimental jobs with the OVS/OVN main branches. It's not one to one mapping with other versions but it at least has some coverage.
We will not get a broken gate if a change containing a bug is backported to OVN stable branch.
Considering past trends, I think we can stick to the current branch that you already proposed and revisit only if it doesn't go as expected, avoiding tag releases bump.
At this point I can’t see any benefit of using the moving LTS branch, but if there
is one please bring it up here.
Not raising anything specific to functional jobs here, but generically, one reason discussed long back was to also consider distros, as they normally pin to a major version and bump on next point release of that that version when it comes out. Distros could also consider bumping the major version but that's not that frequent and depends on feature requirements, etc. Also development branch normally not followed. So if we can keep the same even with functional jobs unless and until there is no specific reason to not to.
Any thoughts on this?
Thanks,
Kuba
[1] https://opendev.org/openstack/neutron-specs/src/branch/master/specs/2025.2/ovn-bgp-integration.rst
[2] https://review.opendev.org/c/openstack/neutron/+/961243