[neutron][operators][all] Watch out for updates of stable/train and stable/stein releases in Neutron
Hi, I want to warn all of You about terrible mistake which we made in Neutron some time ago. We backported to stable releases patch [1] which broke update workflow. So if You are now updating Your Stein or Train Neutron to latest version and You will do it as it should be done, so first neutron-server and then agents, Your neutron-ovs-agents will not work properly with newer neutron-server. Details are in reported bug [2] Broken versions are: * for Train 15.2.0 and 15.3.0 * for Stein 14.3.1, 14.4.0 and 14.4.1 We proposed reverts of [1] and those reverts are now in gate. As soon as they will be merged we will release new, fixed versions for both Stein and Train. So if You didn't update to those broken versions yet, please don't do it now and wait for next version with fix. If You already updated and fixed that issue on Your clusters - You will have exactly same problem during next update again. I know it's very bad but unfortunately we don't have any other way to fix that issue. [1] https://review.opendev.org/#/c/744133/ [2] https://bugs.launchpad.net/neutron/+bug/1903531 -- Slawek Kaplonski Principal Software Engineer Red Hat
Does this affect Queens / Rocky as well? I saw that they got a patch related to this reverted a few days ago. Best Regards, Erik Olof Gunnar Andersson Technical Lead, Senior Cloud Engineer -----Original Message----- From: Slawek Kaplonski <skaplons@redhat.com> Sent: Tuesday, November 24, 2020 1:59 PM To: openstack-discuss@lists.openstack.org Subject: [neutron][operators][all] Watch out for updates of stable/train and stable/stein releases in Neutron Hi, I want to warn all of You about terrible mistake which we made in Neutron some time ago. We backported to stable releases patch [1] which broke update workflow. So if You are now updating Your Stein or Train Neutron to latest version and You will do it as it should be done, so first neutron-server and then agents, Your neutron-ovs-agents will not work properly with newer neutron-server. Details are in reported bug [2] Broken versions are: * for Train 15.2.0 and 15.3.0 * for Stein 14.3.1, 14.4.0 and 14.4.1 We proposed reverts of [1] and those reverts are now in gate. As soon as they will be merged we will release new, fixed versions for both Stein and Train. So if You didn't update to those broken versions yet, please don't do it now and wait for next version with fix. If You already updated and fixed that issue on Your clusters - You will have exactly same problem during next update again. I know it's very bad but unfortunately we don't have any other way to fix that issue. [1] https://urldefense.com/v3/__https://review.opendev.org/*/c/744133/__;Iw!!Ci6... [2] https://urldefense.com/v3/__https://bugs.launchpad.net/neutron/*bug/1903531_... -- Slawek Kaplonski Principal Software Engineer Red Hat
Hi, On Tue, Nov 24, 2020 at 10:37:45PM +0000, Erik Olof Gunnar Andersson wrote:
Does this affect Queens / Rocky as well? I saw that they got a patch related to this reverted a few days ago.
Yes, this affects Queens/Rocky too but in case of those branches, this bad patch wasn't included in any release as both are in EM phase for long time already. So that's why I forgot to mention about them in the previous email. Thx for mentioning them too :)
Best Regards, Erik Olof Gunnar Andersson Technical Lead, Senior Cloud Engineer
-----Original Message----- From: Slawek Kaplonski <skaplons@redhat.com> Sent: Tuesday, November 24, 2020 1:59 PM To: openstack-discuss@lists.openstack.org Subject: [neutron][operators][all] Watch out for updates of stable/train and stable/stein releases in Neutron
Hi,
I want to warn all of You about terrible mistake which we made in Neutron some time ago. We backported to stable releases patch [1] which broke update workflow. So if You are now updating Your Stein or Train Neutron to latest version and You will do it as it should be done, so first neutron-server and then agents, Your neutron-ovs-agents will not work properly with newer neutron-server. Details are in reported bug [2]
Broken versions are: * for Train 15.2.0 and 15.3.0 * for Stein 14.3.1, 14.4.0 and 14.4.1
We proposed reverts of [1] and those reverts are now in gate. As soon as they will be merged we will release new, fixed versions for both Stein and Train. So if You didn't update to those broken versions yet, please don't do it now and wait for next version with fix.
If You already updated and fixed that issue on Your clusters - You will have exactly same problem during next update again. I know it's very bad but unfortunately we don't have any other way to fix that issue.
[1] https://urldefense.com/v3/__https://review.opendev.org/*/c/744133/__;Iw!!Ci6... [2] https://urldefense.com/v3/__https://bugs.launchpad.net/neutron/*bug/1903531_...
-- Slawek Kaplonski Principal Software Engineer Red Hat
-- Slawek Kaplonski Principal Software Engineer Red Hat
Hello, So to be clear in our case here, we are running 15.1.0 for neutron-server and 15.3.0 for neutron agents. That means that the agents does work but there is a security issue,as described regarding allowed address-pair, have I understood it correctly? Best regards Tobias ________________________________ From: Slawek Kaplonski <skaplons@redhat.com> Sent: Tuesday, November 24, 2020 11:53:55 PM To: Erik Olof Gunnar Andersson Cc: openstack-discuss@lists.openstack.org Subject: Re: [neutron][operators][all] Watch out for updates of stable/train and stable/stein releases in Neutron Hi, On Tue, Nov 24, 2020 at 10:37:45PM +0000, Erik Olof Gunnar Andersson wrote:
Does this affect Queens / Rocky as well? I saw that they got a patch related to this reverted a few days ago.
Yes, this affects Queens/Rocky too but in case of those branches, this bad patch wasn't included in any release as both are in EM phase for long time already. So that's why I forgot to mention about them in the previous email. Thx for mentioning them too :)
Best Regards, Erik Olof Gunnar Andersson Technical Lead, Senior Cloud Engineer
-----Original Message----- From: Slawek Kaplonski <skaplons@redhat.com> Sent: Tuesday, November 24, 2020 1:59 PM To: openstack-discuss@lists.openstack.org Subject: [neutron][operators][all] Watch out for updates of stable/train and stable/stein releases in Neutron
Hi,
I want to warn all of You about terrible mistake which we made in Neutron some time ago. We backported to stable releases patch [1] which broke update workflow. So if You are now updating Your Stein or Train Neutron to latest version and You will do it as it should be done, so first neutron-server and then agents, Your neutron-ovs-agents will not work properly with newer neutron-server. Details are in reported bug [2]
Broken versions are: * for Train 15.2.0 and 15.3.0 * for Stein 14.3.1, 14.4.0 and 14.4.1
We proposed reverts of [1] and those reverts are now in gate. As soon as they will be merged we will release new, fixed versions for both Stein and Train. So if You didn't update to those broken versions yet, please don't do it now and wait for next version with fix.
If You already updated and fixed that issue on Your clusters - You will have exactly same problem during next update again. I know it's very bad but unfortunately we don't have any other way to fix that issue.
[1] https://urldefense.com/v3/__https://review.opendev.org/*/c/744133/__;Iw!!Ci6... [2] https://urldefense.com/v3/__https://bugs.launchpad.net/neutron/*bug/1903531_...
-- Slawek Kaplonski Principal Software Engineer Red Hat
-- Slawek Kaplonski Principal Software Engineer Red Hat
Hi, On Wed, Nov 25, 2020 at 08:47:03AM +0000, Tobias Urdin wrote:
Hello,
So to be clear in our case here, we are running 15.1.0 for neutron-server and 15.3.0 for neutron agents.
That means that the agents does work but there is a security issue,as described regarding allowed address-pair, have I understood it correctly?
Yes, as it may have errors while applying SG rules.
Best regards
Tobias
________________________________ From: Slawek Kaplonski <skaplons@redhat.com> Sent: Tuesday, November 24, 2020 11:53:55 PM To: Erik Olof Gunnar Andersson Cc: openstack-discuss@lists.openstack.org Subject: Re: [neutron][operators][all] Watch out for updates of stable/train and stable/stein releases in Neutron
Hi,
On Tue, Nov 24, 2020 at 10:37:45PM +0000, Erik Olof Gunnar Andersson wrote:
Does this affect Queens / Rocky as well? I saw that they got a patch related to this reverted a few days ago.
Yes, this affects Queens/Rocky too but in case of those branches, this bad patch wasn't included in any release as both are in EM phase for long time already. So that's why I forgot to mention about them in the previous email. Thx for mentioning them too :)
Best Regards, Erik Olof Gunnar Andersson Technical Lead, Senior Cloud Engineer
-----Original Message----- From: Slawek Kaplonski <skaplons@redhat.com> Sent: Tuesday, November 24, 2020 1:59 PM To: openstack-discuss@lists.openstack.org Subject: [neutron][operators][all] Watch out for updates of stable/train and stable/stein releases in Neutron
Hi,
I want to warn all of You about terrible mistake which we made in Neutron some time ago. We backported to stable releases patch [1] which broke update workflow. So if You are now updating Your Stein or Train Neutron to latest version and You will do it as it should be done, so first neutron-server and then agents, Your neutron-ovs-agents will not work properly with newer neutron-server. Details are in reported bug [2]
Broken versions are: * for Train 15.2.0 and 15.3.0 * for Stein 14.3.1, 14.4.0 and 14.4.1
We proposed reverts of [1] and those reverts are now in gate. As soon as they will be merged we will release new, fixed versions for both Stein and Train. So if You didn't update to those broken versions yet, please don't do it now and wait for next version with fix.
If You already updated and fixed that issue on Your clusters - You will have exactly same problem during next update again. I know it's very bad but unfortunately we don't have any other way to fix that issue.
[1] https://urldefense.com/v3/__https://review.opendev.org/*/c/744133/__;Iw!!Ci6... [2] https://urldefense.com/v3/__https://bugs.launchpad.net/neutron/*bug/1903531_...
-- Slawek Kaplonski Principal Software Engineer Red Hat
-- Slawek Kaplonski Principal Software Engineer Red Hat
-- Slawek Kaplonski Principal Software Engineer Red Hat
Hi, On Wed, Nov 25, 2020 at 09:58:23AM +0100, Slawek Kaplonski wrote:
Hi,
On Wed, Nov 25, 2020 at 08:47:03AM +0000, Tobias Urdin wrote:
Hello,
So to be clear in our case here, we are running 15.1.0 for neutron-server and 15.3.0 for neutron agents.
That means that the agents does work but there is a security issue,as described regarding allowed address-pair, have I understood it correctly?
Yes, as it may have errors while applying SG rules.
But one more thing. I'm not really sure if that is security issue TBH. By default neutron is dropping traffic to/from instances and You need to allow some kind of traffic by setting security group rules. So if rules will not be applied, some traffic will be dropped but nothing unwanted shouldn't be allowed.
Best regards
Tobias
________________________________ From: Slawek Kaplonski <skaplons@redhat.com> Sent: Tuesday, November 24, 2020 11:53:55 PM To: Erik Olof Gunnar Andersson Cc: openstack-discuss@lists.openstack.org Subject: Re: [neutron][operators][all] Watch out for updates of stable/train and stable/stein releases in Neutron
Hi,
On Tue, Nov 24, 2020 at 10:37:45PM +0000, Erik Olof Gunnar Andersson wrote:
Does this affect Queens / Rocky as well? I saw that they got a patch related to this reverted a few days ago.
Yes, this affects Queens/Rocky too but in case of those branches, this bad patch wasn't included in any release as both are in EM phase for long time already. So that's why I forgot to mention about them in the previous email. Thx for mentioning them too :)
Best Regards, Erik Olof Gunnar Andersson Technical Lead, Senior Cloud Engineer
-----Original Message----- From: Slawek Kaplonski <skaplons@redhat.com> Sent: Tuesday, November 24, 2020 1:59 PM To: openstack-discuss@lists.openstack.org Subject: [neutron][operators][all] Watch out for updates of stable/train and stable/stein releases in Neutron
Hi,
I want to warn all of You about terrible mistake which we made in Neutron some time ago. We backported to stable releases patch [1] which broke update workflow. So if You are now updating Your Stein or Train Neutron to latest version and You will do it as it should be done, so first neutron-server and then agents, Your neutron-ovs-agents will not work properly with newer neutron-server. Details are in reported bug [2]
Broken versions are: * for Train 15.2.0 and 15.3.0 * for Stein 14.3.1, 14.4.0 and 14.4.1
We proposed reverts of [1] and those reverts are now in gate. As soon as they will be merged we will release new, fixed versions for both Stein and Train. So if You didn't update to those broken versions yet, please don't do it now and wait for next version with fix.
If You already updated and fixed that issue on Your clusters - You will have exactly same problem during next update again. I know it's very bad but unfortunately we don't have any other way to fix that issue.
[1] https://urldefense.com/v3/__https://review.opendev.org/*/c/744133/__;Iw!!Ci6... [2] https://urldefense.com/v3/__https://bugs.launchpad.net/neutron/*bug/1903531_...
-- Slawek Kaplonski Principal Software Engineer Red Hat
-- Slawek Kaplonski Principal Software Engineer Red Hat
-- Slawek Kaplonski Principal Software Engineer Red Hat
-- Slawek Kaplonski Principal Software Engineer Red Hat
On 2020-11-25 10:00:22 +0100 (+0100), Slawek Kaplonski wrote:
On Wed, Nov 25, 2020 at 09:58:23AM +0100, Slawek Kaplonski wrote:
On Wed, Nov 25, 2020 at 08:47:03AM +0000, Tobias Urdin wrote:
So to be clear in our case here, we are running 15.1.0 for neutron-server and 15.3.0 for neutron agents.
That means that the agents does work but there is a security issue,as described regarding allowed address-pair, have I understood it correctly?
Yes, as it may have errors while applying SG rules.
But one more thing. I'm not really sure if that is security issue TBH. By default neutron is dropping traffic to/from instances and You need to allow some kind of traffic by setting security group rules. So if rules will not be applied, some traffic will be dropped but nothing unwanted shouldn't be allowed. [...]
I think maybe he was referring specifically to https://launchpad.net/bugs/1867119 (which really should have been marked as a duplicate of https://launchpad.net/bugs/1793029 and the older one reopened instead). In short, it describes an intended/expected behavior, and any potential changes to make it less of a potential foot-cannon were deemed in 1793029 to constitute an API break, so would not have been safe to backport to stable branches. Instead the behavior was highlighted with a warning here: https://docs.openstack.org/api-ref/network/v2/index.html#allowed-address-pai... Probably if 1867119 had been redirected to 1793029 as a duplicate and the discussion continued there, attempts to backport the "fix" for it would have gotten shut down quickly, but that's all hindsight now I suppose. -- Jeremy Stanley
On 2020-11-24 22:58, Slawek Kaplonski wrote:
So if You are now updating Your Stein or Train Neutron to latest version and You will do it as it should be done, so first neutron-server and then agents, Your neutron-ovs-agents will not work properly with newer neutron-server. Details are in reported bug [2]
Even though the original change might be OVS related, the incompatibility / issue I reported (also) arises with neutron-linuxbridge-agent, which my bug report is about actually. There the creation / update of ipsets / iptables fails. What exactly happens in the case of an installation running OVS I cannot say. Regards Christian
participants (5)
-
Christian Rohmann
-
Erik Olof Gunnar Andersson
-
Jeremy Stanley
-
Slawek Kaplonski
-
Tobias Urdin