[TripleO] openvswitch is broken - avoid rechecks in the next couple hours
Hi folks, A new DPDK version landed in CentOS which is not compatible with the current Open vSwitch version that we have in RDO (error below). RDOfolks++ are working on it to make a new OVS version available without DPDK support so that we can unblock our jobs until we get a proper fix. Please, avoid rechecks in the next ~3 hours or so as no tests are expected to pass. Once [0] is merged, we'll need to wait around 30 more minutes for it to be available in CI jobs. Thanks! [0] https://review.rdoproject.org/r/#/c/18853 2019-02-14 07:35:06.464494 | primary | 2019-02-14 07:35:05 | Error: Package: 1:openvswitch-2.10.1-1.el7.x86_64 (delorean-master-deps) 2019-02-14 07:35:06.464603 | primary | 2019-02-14 07:35:05 | Requires: librte_table.so.3()(64bit) 2019-02-14 07:35:06.464711 | primary | 2019-02-14 07:35:05 | Available: dpdk-17.11-13.el7.x86_64 (quickstart-centos-extras)
We should be fine now :) On Thu, Feb 14, 2019 at 11:25 AM Daniel Alvarez Sanchez <dalvarez@redhat.com> wrote:
Hi folks,
A new DPDK version landed in CentOS which is not compatible with the current Open vSwitch version that we have in RDO (error below).
RDOfolks++ are working on it to make a new OVS version available without DPDK support so that we can unblock our jobs until we get a proper fix. Please, avoid rechecks in the next ~3 hours or so as no tests are expected to pass.
Once [0] is merged, we'll need to wait around 30 more minutes for it to be available in CI jobs.
Thanks!
[0] https://review.rdoproject.org/r/#/c/18853
2019-02-14 07:35:06.464494 | primary | 2019-02-14 07:35:05 | Error: Package: 1:openvswitch-2.10.1-1.el7.x86_64 (delorean-master-deps) 2019-02-14 07:35:06.464603 | primary | 2019-02-14 07:35:05 | Requires: librte_table.so.3()(64bit) 2019-02-14 07:35:06.464711 | primary | 2019-02-14 07:35:05 | Available: dpdk-17.11-13.el7.x86_64 (quickstart-centos-extras)
Thanks Daniel ! On Thu, Feb 14, 2019 at 6:41 AM Daniel Alvarez Sanchez <dalvarez@redhat.com> wrote:
We should be fine now :)
On Thu, Feb 14, 2019 at 11:25 AM Daniel Alvarez Sanchez <dalvarez@redhat.com> wrote:
Hi folks,
A new DPDK version landed in CentOS which is not compatible with the current Open vSwitch version that we have in RDO (error below).
RDOfolks++ are working on it to make a new OVS version available without DPDK support so that we can unblock our jobs until we get a proper fix. Please, avoid rechecks in the next ~3 hours or so as no tests are expected to pass.
Once [0] is merged, we'll need to wait around 30 more minutes for it to be available in CI jobs.
Thanks!
[0] https://review.rdoproject.org/r/#/c/18853
2019-02-14 07:35:06.464494 | primary | 2019-02-14 07:35:05 | Error: Package: 1:openvswitch-2.10.1-1.el7.x86_64 (delorean-master-deps) 2019-02-14 07:35:06.464603 | primary | 2019-02-14 07:35:05 | Requires: librte_table.so.3()(64bit) 2019-02-14 07:35:06.464711 | primary | 2019-02-14 07:35:05 | Available: dpdk-17.11-13.el7.x86_64 (quickstart-centos-extras)
Rocky still broken, be prepare. On Fri, Feb 15, 2019 at 3:26 AM Wesley Hayutin <whayutin@redhat.com> wrote:
Thanks Daniel !
On Thu, Feb 14, 2019 at 6:41 AM Daniel Alvarez Sanchez < dalvarez@redhat.com> wrote:
We should be fine now :)
On Thu, Feb 14, 2019 at 11:25 AM Daniel Alvarez Sanchez <dalvarez@redhat.com> wrote:
Hi folks,
A new DPDK version landed in CentOS which is not compatible with the current Open vSwitch version that we have in RDO (error below).
RDOfolks++ are working on it to make a new OVS version available without DPDK support so that we can unblock our jobs until we get a proper fix. Please, avoid rechecks in the next ~3 hours or so as no tests are expected to pass.
Once [0] is merged, we'll need to wait around 30 more minutes for it to be available in CI jobs.
Thanks!
[0] https://review.rdoproject.org/r/#/c/18853
2019-02-14 07:35:06.464494 | primary | 2019-02-14 07:35:05 | Error: Package: 1:openvswitch-2.10.1-1.el7.x86_64 (delorean-master-deps) 2019-02-14 07:35:06.464603 | primary | 2019-02-14 07:35:05 | Requires: librte_table.so.3()(64bit) 2019-02-14 07:35:06.464711 | primary | 2019-02-14 07:35:05 | Available: dpdk-17.11-13.el7.x86_64 (quickstart-centos-extras)
-- Quique Llorente Openstack TripleO CI
Is there something we can do to prevent this in the future? Unrelated to openvswitch itself, it happened with other packages too and will happen again. -- sorin
On 15 Feb 2019, at 07:37, Felix Enrique Llorente Pastora <ellorent@redhat.com> wrote:
Rocky still broken, be prepare.
On Fri, Feb 15, 2019 at 3:26 AM Wesley Hayutin <whayutin@redhat.com> wrote: Thanks Daniel !
On Thu, Feb 14, 2019 at 6:41 AM Daniel Alvarez Sanchez <dalvarez@redhat.com> wrote: We should be fine now :)
On Thu, Feb 14, 2019 at 11:25 AM Daniel Alvarez Sanchez <dalvarez@redhat.com> wrote:
Hi folks,
A new DPDK version landed in CentOS which is not compatible with the current Open vSwitch version that we have in RDO (error below).
RDOfolks++ are working on it to make a new OVS version available without DPDK support so that we can unblock our jobs until we get a proper fix. Please, avoid rechecks in the next ~3 hours or so as no tests are expected to pass.
Once [0] is merged, we'll need to wait around 30 more minutes for it to be available in CI jobs.
Thanks!
[0] https://review.rdoproject.org/r/#/c/18853
2019-02-14 07:35:06.464494 | primary | 2019-02-14 07:35:05 | Error: Package: 1:openvswitch-2.10.1-1.el7.x86_64 (delorean-master-deps) 2019-02-14 07:35:06.464603 | primary | 2019-02-14 07:35:05 | Requires: librte_table.so.3()(64bit) 2019-02-14 07:35:06.464711 | primary | 2019-02-14 07:35:05 | Available: dpdk-17.11-13.el7.x86_64 (quickstart-centos-extras)
-- Quique Llorente
Openstack TripleO CI
On 2019-02-15 08:23:31 +0000 (+0000), Sorin Sbarnea wrote:
Is there something we can do to prevent this in the future?
Unrelated to openvswitch itself, it happened with other packages too and will happen again. [...]
As in how to prevent distros from updating their packages in ways which require some adjustments in our software? That's a big part of why our CI system works the way it does: so we know as soon as possible when we need to make modifications to keep our software compatible with distributions we care about. Hiding or deferring such failures just means that we get to spend more time ignoring our users who are getting regular updates from their operating system and are suddenly unable to use our software on it. -- Jeremy Stanley
On Fri, Feb 15, 2019 at 10:49 AM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2019-02-15 08:23:31 +0000 (+0000), Sorin Sbarnea wrote:
Is there something we can do to prevent this in the future?
Unrelated to openvswitch itself, it happened with other packages too and will happen again. [...]
As in how to prevent distros from updating their packages in ways which require some adjustments in our software? That's a big part of why our CI system works the way it does: so we know as soon as possible when we need to make modifications to keep our software compatible with distributions we care about. Hiding or deferring such failures just means that we get to spend more time ignoring our users who are getting regular updates from their operating system and are suddenly unable to use our software on it.
So it's not necessarily hiding it if you can be notified a head of time and it doesn't disrupt the world. Yes we need to fix it, no we shouldn't completely break the world on updates if possible. Being able to track these changes earlier in testing is one way that we can get ahead of the upcoming changes and get fixes in sooner. I know in tripleo we use the centos continous release repo + periodic to try and find these things before it breaks the world. I'm not sure of the specifics of this change and as to why the continuous release repository didn't help in this instance. As a practice I believe we generally don't like pinning things on master specifically for this reason however we do need to be aware of the risks to the system as a whole and how can we mitigate the potential breakages to allow development to continue while still allowing updates to function as intended. Thanks, -Alex
-- Jeremy Stanley
On Fri, Feb 15, 2019 at 1:21 PM Alex Schultz <aschultz@redhat.com> wrote:
On Fri, Feb 15, 2019 at 10:49 AM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2019-02-15 08:23:31 +0000 (+0000), Sorin Sbarnea wrote:
Is there something we can do to prevent this in the future?
Unrelated to openvswitch itself, it happened with other packages too and will happen again. [...]
As in how to prevent distros from updating their packages in ways which require some adjustments in our software? That's a big part of why our CI system works the way it does: so we know as soon as possible when we need to make modifications to keep our software compatible with distributions we care about. Hiding or deferring such failures just means that we get to spend more time ignoring our users who are getting regular updates from their operating system and are suddenly unable to use our software on it.
So it's not necessarily hiding it if you can be notified a head of time and it doesn't disrupt the world. Yes we need to fix it, no we shouldn't completely break the world on updates if possible. Being able to track these changes earlier in testing is one way that we can get ahead of the upcoming changes and get fixes in sooner. I know in tripleo we use the centos continous release repo + periodic to try and find these things before it breaks the world. I'm not sure of the specifics of this change and as to why the continuous release repository didn't help in this instance. As a practice I believe we generally don't like pinning things on master specifically for this reason however we do need to be aware of the risks to the system as a whole and how can we mitigate the potential breakages to allow development to continue while still allowing updates to function as intended.
Thanks, -Alex
It's my observation, not a fact that the rt repo is not a staging area for every update. The rt repo is well populated in advance of a minor update, but for updates in the same release I think it's very much hit or miss. http://mirror.centos.org/centos/7/rt/x86_64/Packages/?C=M;O=D
-- Jeremy Stanley
participants (6)
-
Alex Schultz
-
Daniel Alvarez Sanchez
-
Felix Enrique Llorente Pastora
-
Jeremy Stanley
-
Sorin Sbarnea
-
Wesley Hayutin