[tripleo] gate blocker tripleo-ci-centos-9-undercloud-upgrade
o/ tripleo
please hold rechecks on master tripleo branch - we have a gate blocker for tripleo-ci-centos-9-undercloud-upgrade
https://bugs.launchpad.net/tripleo/+bug/1984175
package/conflict/mirror issue not clear yet - grateful for pointers if you have them
Hi,
I'd say AFS mirror is out of sync.
Standard CentOS repo:
[root@77b891bf9fd8 ~]# dnf repoquery -q NetworkManager-ovs NetworkManager-ovs-1:1.39.10-1.el9.x86_64 NetworkManager-ovs-1:1.39.12-1.el9.x86_64 NetworkManager-ovs-1:1.39.5-1.el9.x86_64 NetworkManager-ovs-1:1.39.6-1.el9.x86_64 NetworkManager-ovs-1:1.39.7-2.el9.x8
Checking AFS:
# dnf repoquery --repofrompath=afs, http://mirror.regionone.vexxhost-nodepool-sf.rdoproject.org/centos-stream/9-... --disablerepo="*" --enablerepo=afs -q NetworkManager-ovs
NetworkManager-ovs-1:1.39.10-1.el9.x86_64 NetworkManager-ovs-1:1.39.3-1.el9.x86_64 NetworkManager-ovs-1:1.39.5-1.el9.x86_64 NetworkManager-ovs-1:1.39.6-1.el9.x86_64 NetworkManager-ovs-1:1.39.7-2.el9.x86_64
I hope that helps,
Alfredo
On Wed, Aug 10, 2022 at 4:42 PM Marios Andreou marios@redhat.com wrote:
o/ tripleo
please hold rechecks on master tripleo branch - we have a gate blocker for tripleo-ci-centos-9-undercloud-upgrade
https://bugs.launchpad.net/tripleo/+bug/1984175
package/conflict/mirror issue not clear yet - grateful for pointers if you have them
On 2022-08-10 18:05:45 +0200 (+0200), Alfredo Moralejo Alonso wrote:
I'd say AFS mirror is out of sync.
[...]
http://mirror.regionone.vexxhost-nodepool-sf.rdoproject.org/centos-stream/9-...
[...]
To be clear, this is RDO's SoftwareFactory CentOS mirror. Is it backed by OpenDev's AFS tree, or are you creating that mirror independently?
On 2022-08-10 16:21:04 +0000 (+0000), Jeremy Stanley wrote:
On 2022-08-10 18:05:45 +0200 (+0200), Alfredo Moralejo Alonso wrote:
I'd say AFS mirror is out of sync.
[...]
http://mirror.regionone.vexxhost-nodepool-sf.rdoproject.org/centos-stream/9-...
[...]
To be clear, this is RDO's SoftwareFactory CentOS mirror. Is it backed by OpenDev's AFS tree, or are you creating that mirror independently?
Poking around in it a bit, it does seem to be. In that case, we're mirroring from rsync://mirror.facebook.net/centos-stream/9-stream/ and the timestamp at the base of that tree indicates it was last updated around 12:00 UTC today. The cronjob for that runs every 6 hours, so should refresh ~1.5 hours from now.
On 2022-08-10 18:05:45 +0200 (+0200), Alfredo Moralejo Alonso wrote:
Standard CentOS repo:
[root@77b891bf9fd8 ~]# dnf repoquery -q NetworkManager-ovs NetworkManager-ovs-1:1.39.10-1.el9.x86_64 NetworkManager-ovs-1:1.39.12-1.el9.x86_64 NetworkManager-ovs-1:1.39.5-1.el9.x86_64 NetworkManager-ovs-1:1.39.6-1.el9.x86_64 NetworkManager-ovs-1:1.39.7-2.el9.x8
Checking AFS:
# dnf repoquery --repofrompath=afs, http://mirror.regionone.vexxhost-nodepool-sf.rdoproject.org/centos-stream/9-... --disablerepo="*" --enablerepo=afs -q NetworkManager-ovs
NetworkManager-ovs-1:1.39.10-1.el9.x86_64 NetworkManager-ovs-1:1.39.3-1.el9.x86_64 NetworkManager-ovs-1:1.39.5-1.el9.x86_64 NetworkManager-ovs-1:1.39.6-1.el9.x86_64 NetworkManager-ovs-1:1.39.7-2.el9.x86_64
[...]
Looks like it's Facebook's mirror which is behind, and that's what we're pulling from...
wget -qO- http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packag... | grep NetworkManager-ovs | sed 's/.*href="([^"]*)".*/\1/'
NetworkManager-ovs-1.39.3-1.el9.x86_64.rpm NetworkManager-ovs-1.39.5-1.el9.x86_64.rpm NetworkManager-ovs-1.39.6-1.el9.x86_64.rpm NetworkManager-ovs-1.39.7-2.el9.x86_64.rpm NetworkManager-ovs-1.39.10-1.el9.x86_64.rpm
On Wed, Aug 10, 2022 at 7:00 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2022-08-10 18:05:45 +0200 (+0200), Alfredo Moralejo Alonso wrote:
Standard CentOS repo:
[root@77b891bf9fd8 ~]# dnf repoquery -q NetworkManager-ovs NetworkManager-ovs-1:1.39.10-1.el9.x86_64 NetworkManager-ovs-1:1.39.12-1.el9.x86_64 NetworkManager-ovs-1:1.39.5-1.el9.x86_64 NetworkManager-ovs-1:1.39.6-1.el9.x86_64 NetworkManager-ovs-1:1.39.7-2.el9.x8
Checking AFS:
# dnf repoquery --repofrompath=afs,
http://mirror.regionone.vexxhost-nodepool-sf.rdoproject.org/centos-stream/9-...
--disablerepo="*" --enablerepo=afs -q NetworkManager-ovs
NetworkManager-ovs-1:1.39.10-1.el9.x86_64 NetworkManager-ovs-1:1.39.3-1.el9.x86_64 NetworkManager-ovs-1:1.39.5-1.el9.x86_64 NetworkManager-ovs-1:1.39.6-1.el9.x86_64 NetworkManager-ovs-1:1.39.7-2.el9.x86_64
[...]
Looks like it's Facebook's mirror which is behind, and that's what we're pulling from...
wget -qO- http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packag... | grep NetworkManager-ovs | sed 's/.*href="([^"]*)".*/\1/'
NetworkManager-ovs-1.39.3-1.el9.x86_64.rpm NetworkManager-ovs-1.39.5-1.el9.x86_64.rpm NetworkManager-ovs-1.39.6-1.el9.x86_64.rpm NetworkManager-ovs-1.39.7-2.el9.x86_64.rpm NetworkManager-ovs-1.39.10-1.el9.x86_64.rpm
Yep, that seems to be the actual issue.
As you noted, the mirror I mentioned is from RDO but it's backed by opendev AFS.
-- Jeremy Stanley
On Wed, Aug 10, 2022 at 7:16 PM Alfredo Moralejo Alonso amoralej@redhat.com wrote:
On Wed, Aug 10, 2022 at 7:00 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2022-08-10 18:05:45 +0200 (+0200), Alfredo Moralejo Alonso wrote:
Standard CentOS repo:
[root@77b891bf9fd8 ~]# dnf repoquery -q NetworkManager-ovs NetworkManager-ovs-1:1.39.10-1.el9.x86_64 NetworkManager-ovs-1:1.39.12-1.el9.x86_64 NetworkManager-ovs-1:1.39.5-1.el9.x86_64 NetworkManager-ovs-1:1.39.6-1.el9.x86_64 NetworkManager-ovs-1:1.39.7-2.el9.x8
Checking AFS:
# dnf repoquery --repofrompath=afs,
http://mirror.regionone.vexxhost-nodepool-sf.rdoproject.org/centos-stream/9-...
--disablerepo="*" --enablerepo=afs -q NetworkManager-ovs
NetworkManager-ovs-1:1.39.10-1.el9.x86_64 NetworkManager-ovs-1:1.39.3-1.el9.x86_64 NetworkManager-ovs-1:1.39.5-1.el9.x86_64 NetworkManager-ovs-1:1.39.6-1.el9.x86_64 NetworkManager-ovs-1:1.39.7-2.el9.x86_64
[...]
Looks like it's Facebook's mirror which is behind, and that's what we're pulling from...
wget -qO- http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packag... | grep NetworkManager-ovs | sed 's/.*href="([^"]*)".*/\1/'
NetworkManager-ovs-1.39.3-1.el9.x86_64.rpm NetworkManager-ovs-1.39.5-1.el9.x86_64.rpm NetworkManager-ovs-1.39.6-1.el9.x86_64.rpm NetworkManager-ovs-1.39.7-2.el9.x86_64.rpm NetworkManager-ovs-1.39.10-1.el9.x86_64.rpm
Yep, that seems to be the actual issue.
Sent https://review.opendev.org/c/opendev/system-config/+/852793 to switch to rackspace mirror
As you noted, the mirror I mentioned is from RDO but it's backed by opendev AFS.
-- Jeremy Stanley
thanks very much for your help Alfredo and Jeremy
Seems the facebook mirror didn't refresh/sync yet - at least the last run in the build history is from ~1 UTC this morning and it is still hitting the same issue
https://0eee8cd31d7046899357-239a29fa7add7ba7ba8e1040fbce5f75.ssl.cf2.rackcd...
On Thu, Aug 11, 2022 at 1:12 AM Alfredo Moralejo Alonso amoralej@redhat.com wrote:
On Wed, Aug 10, 2022 at 7:16 PM Alfredo Moralejo Alonso amoralej@redhat.com wrote:
On Wed, Aug 10, 2022 at 7:00 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2022-08-10 18:05:45 +0200 (+0200), Alfredo Moralejo Alonso wrote:
Standard CentOS repo:
[root@77b891bf9fd8 ~]# dnf repoquery -q NetworkManager-ovs NetworkManager-ovs-1:1.39.10-1.el9.x86_64 NetworkManager-ovs-1:1.39.12-1.el9.x86_64 NetworkManager-ovs-1:1.39.5-1.el9.x86_64 NetworkManager-ovs-1:1.39.6-1.el9.x86_64 NetworkManager-ovs-1:1.39.7-2.el9.x8
Checking AFS:
# dnf repoquery --repofrompath=afs, http://mirror.regionone.vexxhost-nodepool-sf.rdoproject.org/centos-stream/9-... --disablerepo="*" --enablerepo=afs -q NetworkManager-ovs
NetworkManager-ovs-1:1.39.10-1.el9.x86_64 NetworkManager-ovs-1:1.39.3-1.el9.x86_64 NetworkManager-ovs-1:1.39.5-1.el9.x86_64 NetworkManager-ovs-1:1.39.6-1.el9.x86_64 NetworkManager-ovs-1:1.39.7-2.el9.x86_64
[...]
Looks like it's Facebook's mirror which is behind, and that's what we're pulling from...
wget -qO- http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packag... | grep NetworkManager-ovs | sed 's/.*href="([^"]*)".*/\1/'
NetworkManager-ovs-1.39.3-1.el9.x86_64.rpm NetworkManager-ovs-1.39.5-1.el9.x86_64.rpm NetworkManager-ovs-1.39.6-1.el9.x86_64.rpm NetworkManager-ovs-1.39.7-2.el9.x86_64.rpm NetworkManager-ovs-1.39.10-1.el9.x86_64.rpm
Yep, that seems to be the actual issue.
Sent https://review.opendev.org/c/opendev/system-config/+/852793 to switch to rackspace mirror
As you noted, the mirror I mentioned is from RDO but it's backed by opendev AFS.
-- Jeremy Stanley
On Wed, Aug 10, 2022, at 10:16 AM, Alfredo Moralejo Alonso wrote:
Yep, that seems to be the actual issue.
As you noted, the mirror I mentioned is from RDO but it's backed by opendev AFS.
Note, we can and do change how our mirrors are configured, and we do not treat this as an externally stable interface. A good example of this is when we changed the pypi bandersnatch mirror in AFS to a caching proxy. I'm not aware of any plans that would break this currently, and don't expect it to change anytime soon. But be aware that if we need to change it, we can and will.
Another example is how we recently dropped source packages from our mirrors. Our jobs don't rely on them and if they do they can fetch them from upstream.
Clark
Hello All,
As per [1], Issue on Centos stream 9 build infra got solved last night.
The Facebook mirror is in sync now[2] for the earlier affected package. The upgrade job is back to green.[3]
[1] https://lists.centos.org/pipermail/centos-devel/2022-August/120525.html [2] ~~~ $ wget -qO- http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packag... | grep NetworkManager-ovs | sed 's/.*href="([^"]*)".*/\1/'
NetworkManager-ovs-1.39.5-1.el9.x86_64.rpm NetworkManager-ovs-1.39.6-1.el9.x86_64.rpm NetworkManager-ovs-1.39.7-2.el9.x86_64.rpm NetworkManager-ovs-1.39.10-1.el9.x86_64.rpm NetworkManager-ovs-1.39.12-1.el9.x86_64.rpm ~~~
[3] https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-und...
Thank You Sandeep
On Thu, Aug 11, 2022 at 11:24 PM Clark Boylan cboylan@sapwetik.org wrote:
On Wed, Aug 10, 2022, at 10:16 AM, Alfredo Moralejo Alonso wrote:
Yep, that seems to be the actual issue.
As you noted, the mirror I mentioned is from RDO but it's backed by
opendev AFS.
Note, we can and do change how our mirrors are configured, and we do not treat this as an externally stable interface. A good example of this is when we changed the pypi bandersnatch mirror in AFS to a caching proxy. I'm not aware of any plans that would break this currently, and don't expect it to change anytime soon. But be aware that if we need to change it, we can and will.
Another example is how we recently dropped source packages from our mirrors. Our jobs don't rely on them and if they do they can fetch them from upstream.
Clark
On 2022-08-12 11:15:43 +0530 (+0530), Sandeep Yadav wrote:
As per [1], Issue on Centos stream 9 build infra got solved last night.
The Facebook mirror is in sync now[2] for the earlier affected package. The upgrade job is back to green.[3]
[...]
Unfortunately, that's about the same time Ian logged that we completed the work to shift synchronization off Facebook's mirror and back to Rackspace's again.
participants (5)
-
Alfredo Moralejo Alonso
-
Clark Boylan
-
Jeremy Stanley
-
Marios Andreou
-
Sandeep Yadav