[requirements]Flamingo pyroute2 regression
Hi requirements folks, We identified a bug[1] in pyroute2 0.9.1 that causes intermittent failures during interface attach / detach operations. Originally we got the indication that the pyroute2 0.9.4 fixed the issue and our tests showed green results so we bumped to that version. But since then we observed that the problem still happens in 0.9.4 as well. Can we roll back our pyroute2 dependency to < 0.9 in Flamingo so close to the release? If not, can we later roll back the requirement on stable/2025.2? Or the only way is to roll forward on stable/2025.2 (and on master) when the fix appears in pyroute2? Thanks, gibi [1]https://bugs.launchpad.net/os-vif/+bug/2109396
(sending again as I probably sent it originally from an unregistered address) Hi requirements folks, We identified a bug[1] in pyroute2 0.9.1 that causes intermittent failures during interface attach / detach operations. Originally we got the indication that the pyroute2 0.9.4 fixed the issue and our tests showed green results so we bumped to that version. But since then we observed that the problem still happens in 0.9.4 as well. Can we roll back our pyroute2 dependency to < 0.9 in Flamingo so close to the release? If not, can we later roll back the requirement on stable/2025.2? Or the only way is to roll forward on stable/2025.2 (and on master) when the fix appears in pyroute2? Thanks, gibi [1]https://bugs.launchpad.net/os-vif/+bug/2109396 On Wed, Sep 10, 2025 at 1:40 PM Balazs Gibizer <bgibizer@redhat.com> wrote:
Hi requirements folks,
We identified a bug[1] in pyroute2 0.9.1 that causes intermittent failures during interface attach / detach operations. Originally we got the indication that the pyroute2 0.9.4 fixed the issue and our tests showed green results so we bumped to that version. But since then we observed that the problem still happens in 0.9.4 as well.
Can we roll back our pyroute2 dependency to < 0.9 in Flamingo so close to the release?
If not, can we later roll back the requirement on stable/2025.2? Or the only way is to roll forward on stable/2025.2 (and on master) when the fix appears in pyroute2?
Thanks, gibi
On Wed, Sep 10, 2025 at 1:53 PM Balazs Gibizer <gibi@redhat.com> wrote:
(sending again as I probably sent it originally from an unregistered address) Hi requirements folks,
We identified a bug[1] in pyroute2 0.9.1 that causes intermittent failures during interface attach / detach operations. Originally we got the indication that the pyroute2 0.9.4 fixed the issue and our tests showed green results so we bumped to that version. But since then we observed that the problem still happens in 0.9.4 as well.
Can we roll back our pyroute2 dependency to < 0.9 in Flamingo so close to the release?
If not, can we later roll back the requirement on stable/2025.2? Or the only way is to roll forward on stable/2025.2 (and on master) when the fix appears in pyroute2?
And I forgot to add that I pushed the rollback patch https://review.opendev.org/c/openstack/requirements/+/960366 gibi
Thanks, gibi
On 25-09-10 13:57:06, Balazs Gibizer wrote:
On Wed, Sep 10, 2025 at 1:53 PM Balazs Gibizer <gibi@redhat.com> wrote:
(sending again as I probably sent it originally from an unregistered address) Hi requirements folks,
We identified a bug[1] in pyroute2 0.9.1 that causes intermittent failures during interface attach / detach operations. Originally we got the indication that the pyroute2 0.9.4 fixed the issue and our tests showed green results so we bumped to that version. But since then we observed that the problem still happens in 0.9.4 as well.
Can we roll back our pyroute2 dependency to < 0.9 in Flamingo so close to the release?
If not, can we later roll back the requirement on stable/2025.2? Or the only way is to roll forward on stable/2025.2 (and on master) when the fix appears in pyroute2?
And I forgot to add that I pushed the rollback patch https://review.opendev.org/c/openstack/requirements/+/960366
gibi
Thanks, gibi
Rolling back at this time is somthing I don't think would be a good idea, pyroute2 is used in a lot of places and I'm not sure what (if any) changes those projects would have done that could break when rolling back to 0.8.1. Is there any workaround the the attach/detach that could be made instead (retry)? -- Matthew Thode
On Wed, Sep 10, 2025 at 6:00 PM Matthew Thode <mthode@mthode.org> wrote:
On 25-09-10 13:57:06, Balazs Gibizer wrote:
On Wed, Sep 10, 2025 at 1:53 PM Balazs Gibizer <gibi@redhat.com> wrote:
(sending again as I probably sent it originally from an unregistered address) Hi requirements folks,
We identified a bug[1] in pyroute2 0.9.1 that causes intermittent failures during interface attach / detach operations. Originally we got the indication that the pyroute2 0.9.4 fixed the issue and our tests showed green results so we bumped to that version. But since then we observed that the problem still happens in 0.9.4 as well.
Can we roll back our pyroute2 dependency to < 0.9 in Flamingo so close to the release?
If not, can we later roll back the requirement on stable/2025.2? Or the only way is to roll forward on stable/2025.2 (and on master) when the fix appears in pyroute2?
And I forgot to add that I pushed the rollback patch https://review.opendev.org/c/openstack/requirements/+/960366
gibi
Thanks, gibi
Rolling back at this time is somthing I don't think would be a good idea, pyroute2 is used in a lot of places and I'm not sure what (if any) changes those projects would have done that could break when rolling back to 0.8.1. Is there any workaround the the attach/detach that could be made instead (retry)?
As this is an intermittent fault (probably a race condition in pyroute2) if client retries the operation there is a good chance it will succeed.
-- Matthew Thode
As far as I know, testing were done mostly with 0.8.1 during the cycle. So probably it is less harm, especially considering that Rodolfo wrote that however neutron projects work fine with 0.8.x and 0.9.x versions as well, they are mostly using 0.8.1 anyway and he also suggests capping to 0.8.1 if i understand correctly. @Matthew : either we cap pyroute2 to 0.8.1 or leave it 0.9.4, is that possible to bump to 0.9.5 (or whatever version fixes the bug) later on after the release on stable/2025.2? Thanks, Előd ________________________________________ From: Matthew Thode Sent: Wednesday, September 10, 2025 17:51 To: Balazs Gibizer Cc: openstack-discuss Subject: Re: [requirements]Flamingo pyroute2 regression On 25-09-10 13:57:06, Balazs Gibizer wrote:
On Wed, Sep 10, 2025 at 1:53 PM Balazs Gibizer <gibi@redhat.com> wrote:
(sending again as I probably sent it originally from an unregistered address)
Hi requirements folks,
We identified a bug[1] in pyroute2 0.9.1 that causes intermittent
failures during interface attach / detach operations. Originally we
got the indication that the pyroute2 0.9.4 fixed the issue and our
tests showed green results so we bumped to that version. But since
then we observed that the problem still happens in 0.9.4 as well.
Can we roll back our pyroute2 dependency to < 0.9 in Flamingo so close
to the release?
If not, can we later roll back the requirement on stable/2025.2? Or
the only way is to roll forward on stable/2025.2 (and on master) when
the fix appears in pyroute2?
And I forgot to add that I pushed the rollback patch
https://review.opendev.org/c/openstack/requirements/+/960366
gibi
Thanks,
gibi
Rolling back at this time is somthing I don't think would be a good idea, pyroute2 is used in a lot of places and I'm not sure what (if any) changes those projects would have done that could break when rolling back to 0.8.1. Is there any workaround the the attach/detach that could be made instead (retry)? -- Matthew Thode
Right, if 0.8.1 is working fine (including os-vif), I would suggest limiting Flamingos pyroute to 0.8.1. I think it is too late to start testing a new version (0.9.5 or a newer one). On Fri, Sep 12, 2025 at 11:56 AM Elõd Illés <elod.illes@est.tech> wrote:
As far as I know, testing were done mostly with 0.8.1 during the cycle. So probably it is less harm, especially considering that Rodolfo wrote that however neutron projects work fine with 0.8.x and 0.9.x versions as well, they are mostly using 0.8.1 anyway and he also suggests capping to 0.8.1 if i understand correctly.
@Matthew : either we cap pyroute2 to 0.8.1 or leave it 0.9.4, is that possible to bump to 0.9.5 (or whatever version fixes the bug) later on after the release on stable/2025.2?
Thanks, Előd
________________________________________ From: Matthew Thode Sent: Wednesday, September 10, 2025 17:51 To: Balazs Gibizer Cc: openstack-discuss Subject: Re: [requirements]Flamingo pyroute2 regression
On 25-09-10 13:57:06, Balazs Gibizer wrote:
On Wed, Sep 10, 2025 at 1:53 PM Balazs Gibizer <gibi@redhat.com> wrote:
(sending again as I probably sent it originally from an unregistered address)
Hi requirements folks,
We identified a bug[1] in pyroute2 0.9.1 that causes intermittent
failures during interface attach / detach operations. Originally we
got the indication that the pyroute2 0.9.4 fixed the issue and our
tests showed green results so we bumped to that version. But since
then we observed that the problem still happens in 0.9.4 as well.
Can we roll back our pyroute2 dependency to < 0.9 in Flamingo so close
to the release?
If not, can we later roll back the requirement on stable/2025.2? Or
the only way is to roll forward on stable/2025.2 (and on master) when
the fix appears in pyroute2?
And I forgot to add that I pushed the rollback patch
https://review.opendev.org/c/openstack/requirements/+/960366
gibi
Thanks,
gibi
Rolling back at this time is somthing I don't think would be a good
idea, pyroute2 is used in a lot of places and I'm not sure what (if any)
changes those projects would have done that could break when rolling
back to 0.8.1. Is there any workaround the the attach/detach that could
be made instead (retry)?
--
Matthew Thode
Just to be clear I don't think we have a fix in a later version yet. So our current options is * a) release with 0.9.4 as is with a known issue * b) put https://review.opendev.org/c/openstack/requirements/+/960366 somehow into Flamingo I think Elod is only asking about rolling forward for some future versions that will fix the bug in 2025.2. gibi On Fri, Sep 12, 2025 at 1:00 PM Rodolfo Alonso Hernandez <ralonsoh@redhat.com> wrote:
Right, if 0.8.1 is working fine (including os-vif), I would suggest limiting Flamingos pyroute to 0.8.1. I think it is too late to start testing a new version (0.9.5 or a newer one).
On Fri, Sep 12, 2025 at 11:56 AM Elõd Illés <elod.illes@est.tech> wrote:
As far as I know, testing were done mostly with 0.8.1 during the cycle. So probably it is less harm, especially considering that Rodolfo wrote that however neutron projects work fine with 0.8.x and 0.9.x versions as well, they are mostly using 0.8.1 anyway and he also suggests capping to 0.8.1 if i understand correctly.
@Matthew : either we cap pyroute2 to 0.8.1 or leave it 0.9.4, is that possible to bump to 0.9.5 (or whatever version fixes the bug) later on after the release on stable/2025.2?
Thanks, Előd
________________________________________ From: Matthew Thode Sent: Wednesday, September 10, 2025 17:51 To: Balazs Gibizer Cc: openstack-discuss Subject: Re: [requirements]Flamingo pyroute2 regression
On 25-09-10 13:57:06, Balazs Gibizer wrote:
On Wed, Sep 10, 2025 at 1:53 PM Balazs Gibizer <gibi@redhat.com> wrote:
(sending again as I probably sent it originally from an unregistered address)
Hi requirements folks,
We identified a bug[1] in pyroute2 0.9.1 that causes intermittent
failures during interface attach / detach operations. Originally we
got the indication that the pyroute2 0.9.4 fixed the issue and our
tests showed green results so we bumped to that version. But since
then we observed that the problem still happens in 0.9.4 as well.
Can we roll back our pyroute2 dependency to < 0.9 in Flamingo so close
to the release?
If not, can we later roll back the requirement on stable/2025.2? Or
the only way is to roll forward on stable/2025.2 (and on master) when
the fix appears in pyroute2?
And I forgot to add that I pushed the rollback patch
https://review.opendev.org/c/openstack/requirements/+/960366
gibi
Thanks,
gibi
Rolling back at this time is somthing I don't think would be a good
idea, pyroute2 is used in a lot of places and I'm not sure what (if any)
changes those projects would have done that could break when rolling
back to 0.8.1. Is there any workaround the the attach/detach that could
be made instead (retry)?
--
Matthew Thode
On 25-09-12 13:41:14, Balazs Gibizer wrote:
Just to be clear I don't think we have a fix in a later version yet. So our current options is * a) release with 0.9.4 as is with a known issue * b) put https://review.opendev.org/c/openstack/requirements/+/960366 somehow into Flamingo
I think Elod is only asking about rolling forward for some future versions that will fix the bug in 2025.2.
gibi
On Fri, Sep 12, 2025 at 1:00 PM Rodolfo Alonso Hernandez <ralonsoh@redhat.com> wrote:
Right, if 0.8.1 is working fine (including os-vif), I would suggest limiting Flamingos pyroute to 0.8.1. I think it is too late to start testing a new version (0.9.5 or a newer one).
On Fri, Sep 12, 2025 at 11:56 AM Elõd Illés <elod.illes@est.tech> wrote:
As far as I know, testing were done mostly with 0.8.1 during the cycle. So probably it is less harm, especially considering that Rodolfo wrote that however neutron projects work fine with 0.8.x and 0.9.x versions as well, they are mostly using 0.8.1 anyway and he also suggests capping to 0.8.1 if i understand correctly.
@Matthew : either we cap pyroute2 to 0.8.1 or leave it 0.9.4, is that possible to bump to 0.9.5 (or whatever version fixes the bug) later on after the release on stable/2025.2?
Thanks, Előd
________________________________________ From: Matthew Thode Sent: Wednesday, September 10, 2025 17:51 To: Balazs Gibizer Cc: openstack-discuss Subject: Re: [requirements]Flamingo pyroute2 regression
On 25-09-10 13:57:06, Balazs Gibizer wrote:
On Wed, Sep 10, 2025 at 1:53 PM Balazs Gibizer <gibi@redhat.com> wrote:
(sending again as I probably sent it originally from an unregistered address)
Hi requirements folks,
We identified a bug[1] in pyroute2 0.9.1 that causes intermittent
failures during interface attach / detach operations. Originally we
got the indication that the pyroute2 0.9.4 fixed the issue and our
tests showed green results so we bumped to that version. But since
then we observed that the problem still happens in 0.9.4 as well.
Can we roll back our pyroute2 dependency to < 0.9 in Flamingo so close
to the release?
If not, can we later roll back the requirement on stable/2025.2? Or
the only way is to roll forward on stable/2025.2 (and on master) when
the fix appears in pyroute2?
And I forgot to add that I pushed the rollback patch
https://review.opendev.org/c/openstack/requirements/+/960366
gibi
Thanks,
gibi
Rolling back at this time is somthing I don't think would be a good
idea, pyroute2 is used in a lot of places and I'm not sure what (if any)
changes those projects would have done that could break when rolling
back to 0.8.1. Is there any workaround the the attach/detach that could
be made instead (retry)?
--
Matthew Thode
I approved this, given neutron is OK with it. -- Matthew Thode
Hello all: So far, the Neutron related projects (Neutron, ovs-bgp-agent, networking-bagpipe, neutron-fwaas), that import pyroute2 are working fine with any 0.8.x or 0.9.x version. So it will be no problem to pin it to <0.9. Using codesearch, I don't see any other project explicitly requesting a pyroute2 version above 0.8.x, so I think that could be safe to enforce this limitation. Checking the issues with os-vif and pyroute 0.9.x, it is obvious that we can't deliver both for Flamingos. At this point (RC1 week), after several pyroute2 versions released trying to fix this problem, we need to enforce this version cap. Regards. On Wed, Sep 10, 2025 at 9:43 PM <thomas@goirand.fr> wrote:
On Sep 10, 2025 13:54, Balazs Gibizer <gibi@redhat.com> wrote:
Can we roll back our pyroute2 dependency to < 0.9 in Flamingo so close to the release?
FYI, not a problem from the point of view of Debian (I didn't package 0.9 yet...).
Cheers,
Thomas
participants (6)
-
Balazs Gibizer
-
Balazs Gibizer
-
Elõd Illés
-
Matthew Thode
-
Rodolfo Alonso Hernandez
-
thomas＠goirand.fr