On Feb 27, 2026, at 10:05 AM, Lajos Katona <katonalala@gmail.com> wrote:

Hi, Just a notification to be written here as well: please update the related documentation also if you need to update the OVS/OVN versions: https://opendev.org/openstack/neutron/src/branch/master/doc/source/install/o...

Thanks Lajos for pointing that out. Is that something we’d do when we change the job? It seems to me it looks more like something we’d do after the OpenStack release. Kuba

Best wishes Lajos (lajoskatona)

Slawek Kaplonski <skaplons@redhat.com> ezt írta (időpont: 2026. febr. 27., P, 15:58): Hi,

Dnia piątek, 27 lutego 2026 15:17:21 czas środkowoeuropejski standardowy Jakub Libosvar pisze:

...

Hi all,

I wanted to bring this topic up for a discussion again.

I’ve been working on a Neutron feature [1] for some time, the feature requires the BGP support in OVN. The functional job currently uses a moving OVN branch based on the OVN LTS version - which is 24.03 - almost 2 years old. I tried to change this and requested a patch to switch to using a pinned OVN mostly for two benefits: - we won’t get gate broken if a bad patch is backported to the branch-24.03 - I get the BGP features needed for the work on [1]

The patch was denied to keep the policy we set on ourselves - ie. to go with the branch based on the OVN LTS version. The workaround was to skip multiple tests that rely on the OVN changes specific to BGP. This is not ideal as now majority of the functionality is not tested and we’re just waiting for the LTS release, which may bring some surprises.

I didn't know that we have such policy TBH. And I know that we have periodic tempest job which is testing with OVS and OVN from master branch daily: https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovn-tempest-full-multinode-ovs-master&skip=0 In other scenario jobs AFAIR OVS and OVN is installed from the packages provided by distro.

...

This was a concrete example but if we put it into a more general situation, in Neutron we heavily rely on OVN and its features and any development of a potential feature that requires some work in OVN gets cumbersome because we can’t validate our code against OVN until the point a next OVN LTS version is released, which in the worst case scenario can take up to 2 years.

Hence I propose a change and use a pinned hash from the OVN tree that is updated on two occasions: - A next OVN version is released (non-LTS) - We’re developing a feature that relies on bleeding edge OVN - then we pick a good hash and pin to it in the functional job definition (not in the scripts)

This would apply solely to the functional tests because tempest jobs can define their own OVN version and unit tests should be isolated from the used OVN version, and only to the master branch - because that is the development branch after all.

Such a change brings multiple benefits, we won’t need to skip tests if the required functionality is not present in current OVN version and have the development go smoother. We may uncover issues in OVN early. We will not get a broken gate if a change containing a bug is backported to OVN stable branch.

At this point I can’t see any benefit of using the moving LTS branch, but if there is one please bring it up here.

Any thoughts on this?

IMO it makes perfect sense to use newer OVN in functional tests job.

...

Thanks, Kuba

[1] https://opendev.org/openstack/neutron-specs/src/branch/master/specs/2025.2/o... [2] https://review.opendev.org/c/openstack/neutron/+/961243

-- Slawek Kaplonski Principal Software Engineer Red Hat

Hi Kuba, On 2/27/26 9:57 AM, Slawek Kaplonski wrote:

Hi,

Dnia piątek, 27 lutego 2026 15:17:21 czas środkowoeuropejski standardowy Jakub Libosvar pisze:

...

Hi all,

I wanted to bring this topic up for a discussion again.

I’ve been working on a Neutron feature [1] for some time, the feature requires the BGP support in OVN. The functional job currently uses a moving OVN branch based on the OVN LTS version - which is 24.03 - almost 2 years old. I tried to change this and requested a patch to switch to using a pinned OVN mostly for two benefits: - we won’t get gate broken if a bad patch is backported to the branch-24.03 - I get the BGP features needed for the work on [1]

The patch was denied to keep the policy we set on ourselves - ie. to go with the branch based on the OVN LTS version. The workaround was to skip multiple tests that rely on the OVN changes specific to BGP. This is not ideal as now majority of the functionality is not tested and we’re just waiting for the LTS release, which may bring some surprises.

I didn't know that we have such policy TBH. And I know that we have periodic tempest job which is testing with OVS and OVN from master branch daily: https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovn-tempest-full-multinode-ovs-master&skip=0 In other scenario jobs AFAIR OVS and OVN is installed from the packages provided by distro.

I don't think we ever decided on a policy, although I remember discussing it at a PTG, would have to go through the notes. Like Slawek mentioned, we have other jobs in the periodic/experimental queues using main and compiling, can we do the same for functional so we get coverage here? -Brian

...

This was a concrete example but if we put it into a more general situation, in Neutron we heavily rely on OVN and its features and any development of a potential feature that requires some work in OVN gets cumbersome because we can’t validate our code against OVN until the point a next OVN LTS version is released, which in the worst case scenario can take up to 2 years.

Hence I propose a change and use a pinned hash from the OVN tree that is updated on two occasions: - A next OVN version is released (non-LTS) - We’re developing a feature that relies on bleeding edge OVN - then we pick a good hash and pin to it in the functional job definition (not in the scripts)

This would apply solely to the functional tests because tempest jobs can define their own OVN version and unit tests should be isolated from the used OVN version, and only to the master branch - because that is the development branch after all.

Such a change brings multiple benefits, we won’t need to skip tests if the required functionality is not present in current OVN version and have the development go smoother. We may uncover issues in OVN early. We will not get a broken gate if a change containing a bug is backported to OVN stable branch.

At this point I can’t see any benefit of using the moving LTS branch, but if there is one please bring it up here.

Any thoughts on this?

IMO it makes perfect sense to use newer OVN in functional tests job.

...

Thanks, Kuba

[1] https://opendev.org/openstack/neutron-specs/src/branch/master/specs/2025.2/o... [2] https://review.opendev.org/c/openstack/neutron/+/961243

Hi, Dnia wtorek, 3 marca 2026 14:36:41 czas środkowoeuropejski standardowy Jakub Libosvar pisze:

Thank you everyone for the replies. Please see inline.

...

On Feb 27, 2026, at 4:58 PM, Brian Haley <haleyb.dev@gmail.com> wrote:

Hi Kuba,

On 2/27/26 9:57 AM, Slawek Kaplonski wrote:

...

Hi, Dnia piątek, 27 lutego 2026 15:17:21 czas środkowoeuropejski standardowy Jakub Libosvar pisze:

...

Hi all,

I wanted to bring this topic up for a discussion again.

I’ve been working on a Neutron feature [1] for some time, the feature requires the BGP support in OVN. The functional job currently uses a moving OVN branch based on the OVN LTS version - which is 24.03 - almost 2 years old. I tried to change this and requested a patch to switch to using a pinned OVN mostly for two benefits: - we won’t get gate broken if a bad patch is backported to the branch-24.03 - I get the BGP features needed for the work on [1]

The patch was denied to keep the policy we set on ourselves - ie. to go with the branch based on the OVN LTS version. The workaround was to skip multiple tests that rely on the OVN changes specific to BGP. This is not ideal as now majority of the functionality is not tested and we’re just waiting for the LTS release, which may bring some surprises. I didn't know that we have such policy TBH. And I know that we have periodic tempest job which is testing with OVS and OVN from master branch daily: https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovn-tempest-full-multinode-ovs-master&skip=0 In other scenario jobs AFAIR OVS and OVN is installed from the packages provided by distro.

I don't think we ever decided on a policy, although I remember discussing it at a PTG, would have to go through the notes.

I remember discussing that too and, maybe I’m wrong, I remember it as we decided going always with the LTS version - I don’t remember why though, I can’t see any benefit in doing that.

...

Like Slawek mentioned, we have other jobs in the periodic/experimental queues using main and compiling, can we do the same for functional so we get coverage here?

Do you suggest to create a new experimental job with newer OVN? What we do right now is that we decorate tests that require schema changes related to the BGP feature [1]. The more I think about it the more I’m convinced that was a bad decision as the correct behavior of the functional test testing BGP in such case should be a failure, because the production code can’t work with that given OVN version and hence should fail to indicate something is wrong.

I can see two path forwards. Preferable use new OVN in the current functional job, or if I understand the it correctly create another experimental job. In both cases I’d delete the “skipping” decorator and blacklist the tests from the check/gate functional job.

What do you guys think is a better way forward?

The reason I mentioned that we have scenario jobs with OVS/OVN from master was simply to show that we already do testing with master branch, not only LTS so I don't see a reason not to do changes in functional job. I don't think we need yet another periodic/experimental job for that coverage. I would be personally fine with changing existing functional job and run those tests there.

Thanks! Kuba

[1] https://opendev.org/openstack/neutron/src/branch/master/neutron/tests/functi...

...

-Brian

...

...

This was a concrete example but if we put it into a more general situation, in Neutron we heavily rely on OVN and its features and any development of a potential feature that requires some work in OVN gets cumbersome because we can’t validate our code against OVN until the point a next OVN LTS version is released, which in the worst case scenario can take up to 2 years.

Hence I propose a change and use a pinned hash from the OVN tree that is updated on two occasions: - A next OVN version is released (non-LTS) - We’re developing a feature that relies on bleeding edge OVN - then we pick a good hash and pin to it in the functional job definition (not in the scripts)

This would apply solely to the functional tests because tempest jobs can define their own OVN version and unit tests should be isolated from the used OVN version, and only to the master branch - because that is the development branch after all.

Such a change brings multiple benefits, we won’t need to skip tests if the required functionality is not present in current OVN version and have the development go smoother. We may uncover issues in OVN early. We will not get a broken gate if a change containing a bug is backported to OVN stable branch.

At this point I can’t see any benefit of using the moving LTS branch, but if there is one please bring it up here. Any thoughts on this? IMO it makes perfect sense to use newer OVN in functional tests job.

Thanks, Kuba

[1] https://opendev.org/openstack/neutron-specs/src/branch/master/specs/2025.2/o... [2] https://review.opendev.org/c/openstack/neutron/+/961243

-- Slawek Kaplonski Principal Software Engineer Red Hat

On Mar 12, 2026, at 7:04 AM, Yatin Karel <ykarel@redhat.com> wrote:

Thanks Jakub for starting this thread, and sorry to get late on to my reply, getting today as Eduardo reminded about that patch to me.

On Fri, Feb 27, 2026 at 7:48 PM Jakub Libosvar <jlibosva@redhat.com> wrote: Hi all,

I wanted to bring this topic up for a discussion again.

I’ve been working on a Neutron feature [1] for some time, the feature requires the BGP support in OVN. The functional job currently uses a moving OVN branch based on the OVN LTS version - which is 24.03 - almost 2 years old. I tried to change this and requested a patch to switch to using a pinned OVN mostly for two benefits: - we won’t get gate broken if a bad patch is backported to the branch-24.03 - I get the BGP features needed for the work on [1]

The patch was denied to keep the policy we set on ourselves - ie. to go with the branch based on the OVN LTS version. The workaround was to skip multiple tests that rely on the OVN changes specific to BGP. This is not ideal as now majority of the functionality is not tested and we’re just waiting for the LTS release, which may bring some surprises. As the original reviewer, I wanted to clarify that I never denied the patch; I only sought more clarification and others' opinions to reach a consensus, as others stated regarding the coverage we have in the periodics/experimental pipeline. That patch was later abandoned. Perhaps I didn't communicate clearly, so I apologize for that.

The patch was abandoned because I didn’t think it was important enough at that time with the existing patches. At some later stage there were more and more failures due to the “old” OVN version and it started to make not much sense to me to skip half of the BGP tests. And to raise awareness I started this thread so more people can engage. I didn’t seek any blame, I assume good intentions on the patch comments :)

This was a concrete example but if we put it into a more general situation, in Neutron we heavily rely on OVN and its features and any development of a potential feature that requires some work in OVN gets cumbersome because we can’t validate our code against OVN until the point a next OVN LTS version is released, which in the worst case scenario can take up to 2 years.

Moving to new versions based on a required feature is quite valid.

Hence I propose a change and use a pinned hash from the OVN tree that is updated on two occasions: - A next OVN version is released (non-LTS) - We’re developing a feature that relies on bleeding edge OVN - then we pick a good hash and pin to it in the functional job definition (not in the scripts) - I see you pushed an update to switch to branch-26.03 (an upcoming LTS) so i think it's fine we can keep it as it is and Don't pin to a tag release, as the last few years show it's gone well for us, i.e We get new fixes in those branches and hasn't seen any breakages yet. However, we can revisit this later if we see any difference in the trend. And if another requirement comes in future it's totally find to rebump even to non LTS for functional tests.

It’s fine with me but I pushed the branch- mostly because the 26.03 OVN version is to be released at the end of the month, so we can get a headstart. There are tradeoffs to each approach, I’d be more comfortable using release tags and treat OVN as any other requirement - just being a one we follow more closely because of the high feature dependency. I remember in the past we got the gate broken a few times by following a branch for OVS.

- We currently pin functional scripts so that the local setup uses the same OVS/OVN versions as CI.

This would apply solely to the functional tests because tempest jobs can define their own OVN version and unit tests should be isolated from the used OVN version, and only to the master branch - because that is the development branch after all. +1 Such a change brings multiple benefits, we won’t need to skip tests if the required functionality is not present in current OVN version and have the development go smoother. We may uncover issues in OVN early.

For this reason and to catch any regressions, we also run periodic/experimental jobs with the OVS/OVN main branches. It's not one to one mapping with other versions but it at least has some coverage.

We may want to bring this topic and discuss it - there were a few things [3][4] that made tests failing in 25.03 - thus it’s been more than a year like that - and we did not notice.

We will not get a broken gate if a change containing a bug is backported to OVN stable branch. Considering past trends, I think we can stick to the current branch that you already proposed and revisit only if it doesn't go as expected, avoiding tag releases bump.

At this point I can’t see any benefit of using the moving LTS branch, but if there is one please bring it up here. Not raising anything specific to functional jobs here, but generically, one reason discussed long back was to also consider distros, as they normally pin to a major version and bump on next point release of that that version when it comes out. Distros could also consider bumping the major version but that's not that frequent and depends on feature requirements, etc. Also development branch normally not followed. So if we can keep the same even with functional jobs unless and until there is no specific reason to not to.

That was the case in the past and that’s how [5] was born, to overcome the distro dependency since we do not update distros often and some feature may also have kernel module dependencies. Thanks Yatin for your inputs! Kuba [3] https://review.opendev.org/c/openstack/neutron/+/979274 [4] https://review.opendev.org/c/openstack/ovsdbapp/+/979081 [5] https://review.opendev.org/c/openstack/neutron/+/266423

Any thoughts on this?

Thanks, Kuba

[1] https://opendev.org/openstack/neutron-specs/src/branch/master/specs/2025.2/o... [2] https://review.opendev.org/c/openstack/neutron/+/961243

-- Thanks and Regards Yatin Karel