Any ideas on fixing bug 1827083 so we can merge code?
I'm not sure what is causing the bug [1] but it's failing at a really high rate for about week now. Do we have ideas on the issue? Do we have thoughts on a workaround? Or should we disable the vexxhost-sjc1 provider until it's solved?
[1] http://status.openstack.org/elastic-recheck/#1827083
Have we tried changing the URI to https://opendev.org/openstack/requirements/raw/branch/master/upper-constrain... to avoid the redirecting?
On 5/9/19 8:02 AM, Matt Riedemann wrote:
I'm not sure what is causing the bug [1] but it's failing at a really high rate for about week now. Do we have ideas on the issue? Do we have thoughts on a workaround? Or should we disable the vexxhost-sjc1 provider until it's solved?
On 2019-05-09 08:49:35 -0500 (-0500), Eric Fried wrote:
Have we tried changing the URI to https://opendev.org/openstack/requirements/raw/branch/master/upper-constrain... to avoid the redirecting?
On 5/9/19 8:02 AM, Matt Riedemann wrote:
I'm not sure what is causing the bug [1] but it's failing at a really high rate for about week now. Do we have ideas on the issue? Do we have thoughts on a workaround? Or should we disable the vexxhost-sjc1 provider until it's solved?
I have to assume the bug report itself is misleading. Jobs should be using the on-disk copy of the requirements repository provided by Zuul for this and not retrieving that file over the network. However the problem is presumably DNS resolution not working at all on those nodes, so something is going to break at some point in the job in those cases regardless.
is it possible that this is because `mirror.sjc1.vexxhost.openstack.org` does not actually have AAAA records that causes this?
On Thu, May 9, 2019 at 9:58 AM Jeremy Stanley fungi@yuggoth.org wrote:
On 2019-05-09 08:49:35 -0500 (-0500), Eric Fried wrote:
Have we tried changing the URI to https://opendev.org/openstack/requirements/raw/branch/master/upper-constrain... to avoid the redirecting?
On 5/9/19 8:02 AM, Matt Riedemann wrote:
I'm not sure what is causing the bug [1] but it's failing at a really high rate for about week now. Do we have ideas on the issue? Do we have thoughts on a workaround? Or should we disable the vexxhost-sjc1 provider until it's solved?
I have to assume the bug report itself is misleading. Jobs should be using the on-disk copy of the requirements repository provided by Zuul for this and not retrieving that file over the network. However the problem is presumably DNS resolution not working at all on those nodes, so something is going to break at some point in the job in those cases regardless. -- Jeremy Stanley
On 5/11/2019 6:57 PM, Mohammed Naser wrote:
is it possible that this is because `mirror.sjc1.vexxhost.openstack.org` does not actually have AAAA records that causes this?
On Thu, May 9, 2019 at 9:58 AM Jeremy Stanleyfungi@yuggoth.org wrote:
On 2019-05-09 08:49:35 -0500 (-0500), Eric Fried wrote:
Have we tried changing the URI to https://opendev.org/openstack/requirements/raw/branch/master/upper-constrain... to avoid the redirecting?
On 5/9/19 8:02 AM, Matt Riedemann wrote:
I'm not sure what is causing the bug [1] but it's failing at a really high rate for about week now. Do we have ideas on the issue? Do we have thoughts on a workaround? Or should we disable the vexxhost-sjc1 provider until it's solved?
I have to assume the bug report itself is misleading. Jobs should be using the on-disk copy of the requirements repository provided by Zuul for this and not retrieving that file over the network. However the problem is presumably DNS resolution not working at all on those nodes, so something is going to break at some point in the job in those cases regardless. -- Jeremy Stanley
Just an update on this for those watching at home. Clark switched the vexxhost-sjc1 nodes to IPv4:
https://review.opendev.org/#/c/659140/
https://review.opendev.org/#/c/659181/
And early results in logstash [1] show that might have worked around the problem for now.
[1] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A...
participants (4)
-
Eric Fried
-
Jeremy Stanley
-
Matt Riedemann
-
Mohammed Naser