<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 29, 2020 at 2:25 AM Bogdan Dobrelya <<a href="mailto:bdobreli@redhat.com">bdobreli@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 7/28/20 6:09 PM, Wesley Hayutin wrote:<br>
> <br>
> <br>
> On Tue, Jul 28, 2020 at 7:24 AM Emilien Macchi <<a href="mailto:emilien@redhat.com" target="_blank">emilien@redhat.com</a> <br>
> <mailto:<a href="mailto:emilien@redhat.com" target="_blank">emilien@redhat.com</a>>> wrote:<br>
> <br>
> <br>
> <br>
> On Tue, Jul 28, 2020 at 9:20 AM Alex Schultz <<a href="mailto:aschultz@redhat.com" target="_blank">aschultz@redhat.com</a><br>
> <mailto:<a href="mailto:aschultz@redhat.com" target="_blank">aschultz@redhat.com</a>>> wrote:<br>
> <br>
> On Tue, Jul 28, 2020 at 7:13 AM Emilien Macchi<br>
> <<a href="mailto:emilien@redhat.com" target="_blank">emilien@redhat.com</a> <mailto:<a href="mailto:emilien@redhat.com" target="_blank">emilien@redhat.com</a>>> wrote:<br>
> ><br>
> ><br>
> ><br>
> > On Mon, Jul 27, 2020 at 5:27 PM Wesley Hayutin<br>
> <<a href="mailto:whayutin@redhat.com" target="_blank">whayutin@redhat.com</a> <mailto:<a href="mailto:whayutin@redhat.com" target="_blank">whayutin@redhat.com</a>>> wrote:<br>
> >><br>
> >> FYI...<br>
> >><br>
> >> If you find your jobs are failing with an error similar to<br>
> [1], you have been rate limited by <a href="http://docker.io" rel="noreferrer" target="_blank">docker.io</a> <<a href="http://docker.io" rel="noreferrer" target="_blank">http://docker.io</a>><br>
> via the upstream mirror system and have hit [2]. I've been<br>
> discussing the issue w/ upstream infra, rdo-infra and a few CI<br>
> engineers.<br>
> >><br>
> >> There are a few ways to mitigate the issue however I don't<br>
> see any of the options being completed very quickly so I'm<br>
> asking for your patience while this issue is socialized and<br>
> resolved.<br>
> >><br>
> >> For full transparency we're considering the following options.<br>
> >><br>
> >> 1. move off of <a href="http://docker.io" rel="noreferrer" target="_blank">docker.io</a> <<a href="http://docker.io" rel="noreferrer" target="_blank">http://docker.io</a>> to <a href="http://quay.io" rel="noreferrer" target="_blank">quay.io</a><br>
> <<a href="http://quay.io" rel="noreferrer" target="_blank">http://quay.io</a>><br>
> ><br>
> ><br>
> > <a href="http://quay.io" rel="noreferrer" target="_blank">quay.io</a> <<a href="http://quay.io" rel="noreferrer" target="_blank">http://quay.io</a>> also has API rate limit:<br>
> > <a href="https://docs.quay.io/issues/429.html" rel="noreferrer" target="_blank">https://docs.quay.io/issues/429.html</a><br>
> ><br>
> > Now I'm not sure about how many requests per seconds one can<br>
> do vs the other but this would need to be checked with the quay<br>
> team before changing anything.<br>
> > Also <a href="http://quay.io" rel="noreferrer" target="_blank">quay.io</a> <<a href="http://quay.io" rel="noreferrer" target="_blank">http://quay.io</a>> had its big downtimes as well,<br>
> SLA needs to be considered.<br>
> ><br>
> >> 2. local container builds for each job in master, possibly<br>
> ussuri<br>
> ><br>
> ><br>
> > Not convinced.<br>
> > You can look at CI logs:<br>
> > - pulling / updating / pushing container images from<br>
> <a href="http://docker.io" rel="noreferrer" target="_blank">docker.io</a> <<a href="http://docker.io" rel="noreferrer" target="_blank">http://docker.io</a>> to local registry takes ~10 min on<br>
> standalone (OVH)<br>
> > - building containers from scratch with updated repos and<br>
> pushing them to local registry takes ~29 min on standalone (OVH).<br>
> ><br>
> >><br>
> >> 3. parent child jobs upstream where rpms and containers will<br>
> be build and host artifacts for the child jobs<br>
> ><br>
> ><br>
> > Yes, we need to investigate that.<br>
> ><br>
> >><br>
> >> 4. remove some portion of the upstream jobs to lower the<br>
> impact we have on 3rd party infrastructure.<br>
> ><br>
> ><br>
> > I'm not sure I understand this one, maybe you can give an<br>
> example of what could be removed?<br>
> <br>
> We need to re-evaulate our use of scenarios (e.g. we have two<br>
> scenario010's both are non-voting). There's a reason we<br>
> historically<br>
> didn't want to add more jobs because of these types of resource<br>
> constraints. I think we've added new jobs recently and likely<br>
> need to<br>
> reduce what we run. Additionally we might want to look into reducing<br>
> what we run on stable branches as well.<br>
> <br>
> <br>
> Oh... removing jobs (I thought we would remove some steps of the jobs).<br>
> Yes big +1, this should be a continuous goal when working on CI, and<br>
> always evaluating what we need vs what we run now.<br>
> <br>
> We should look at:<br>
> 1) services deployed in scenarios that aren't worth testing (e.g.<br>
> deprecated or unused things) (and deprecate the unused things)<br>
> 2) jobs themselves (I don't have any example beside scenario010 but<br>
> I'm sure there are more).<br>
> -- <br>
> Emilien Macchi<br>
> <br>
> <br>
> Thanks Alex, Emilien<br>
> <br>
> +1 to reviewing the catalog and adjusting things on an ongoing basis.<br>
> <br>
> All.. it looks like the issues with <a href="http://docker.io" rel="noreferrer" target="_blank">docker.io</a> <<a href="http://docker.io" rel="noreferrer" target="_blank">http://docker.io</a>> were <br>
> more of a flare up than a change in <a href="http://docker.io" rel="noreferrer" target="_blank">docker.io</a> <<a href="http://docker.io" rel="noreferrer" target="_blank">http://docker.io</a>> policy <br>
> or infrastructure [2]. The flare up started on July 27 8am utc and <br>
> ended on July 27 17:00 utc, see screenshots.<br>
<br>
The numbers of image prepare workers and its exponential fallback <br>
intervals should be also adjusted. I've analysed the log snippet [0] for <br>
the connection reset counts by workers versus the times the rate <br>
limiting was triggered. See the details in the reported bug [1].<br>
<br>
tl;dr -- for an example 5 sec interval 03:55:31,379 - 03:55:36,110:<br>
<br>
Conn Reset Counts by a Worker PID:<br>
3 58412<br>
2 58413<br>
3 58415<br>
3 58417<br>
<br>
which seems too much of (workers*reconnects) and triggers rate limiting <br>
immediately.<br>
<br>
[0] <br>
<a href="https://13b475d7469ed7126ee9-28d4ad440f46f2186fe3f98464e57890.ssl.cf1.rackcdn.com/741228/6/check/tripleo-ci-centos-8-undercloud-containers/8e47836/logs/undercloud/var/log/tripleo-container-image-prepare.log" rel="noreferrer" target="_blank">https://13b475d7469ed7126ee9-28d4ad440f46f2186fe3f98464e57890.ssl.cf1.rackcdn.com/741228/6/check/tripleo-ci-centos-8-undercloud-containers/8e47836/logs/undercloud/var/log/tripleo-container-image-prepare.log</a><br>
<br>
[1] <a href="https://bugs.launchpad.net/tripleo/+bug/1889372" rel="noreferrer" target="_blank">https://bugs.launchpad.net/tripleo/+bug/1889372</a><br>
<br>
-- <br>
Best regards,<br>
Bogdan Dobrelya,<br>
Irc #bogdando<br>
<br></blockquote><div><br></div><div>FYI..</div><div><br></div><div>The issue w/ "too many requests" is back. Expect delays and failures in attempting to merge your patches upstream across all branches. The issue is being tracked as a critical issue.</div></div></div>