<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 29, 2020 at 4:33 PM Alex Schultz <<a href="mailto:aschultz@redhat.com">aschultz@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Wed, Jul 29, 2020 at 7:13 AM Wesley Hayutin <<a href="mailto:whayutin@redhat.com" target="_blank">whayutin@redhat.com</a>> wrote:<br>
><br>
><br>
><br>
> On Wed, Jul 29, 2020 at 2:25 AM Bogdan Dobrelya <<a href="mailto:bdobreli@redhat.com" target="_blank">bdobreli@redhat.com</a>> wrote:<br>
>><br>
>> On 7/28/20 6:09 PM, Wesley Hayutin wrote:<br>
>> ><br>
>> ><br>
>> > On Tue, Jul 28, 2020 at 7:24 AM Emilien Macchi <<a href="mailto:emilien@redhat.com" target="_blank">emilien@redhat.com</a><br>
>> > <mailto:<a href="mailto:emilien@redhat.com" target="_blank">emilien@redhat.com</a>>> wrote:<br>
>> ><br>
>> ><br>
>> ><br>
>> > On Tue, Jul 28, 2020 at 9:20 AM Alex Schultz <<a href="mailto:aschultz@redhat.com" target="_blank">aschultz@redhat.com</a><br>
>> > <mailto:<a href="mailto:aschultz@redhat.com" target="_blank">aschultz@redhat.com</a>>> wrote:<br>
>> ><br>
>> > On Tue, Jul 28, 2020 at 7:13 AM Emilien Macchi<br>
>> > <<a href="mailto:emilien@redhat.com" target="_blank">emilien@redhat.com</a> <mailto:<a href="mailto:emilien@redhat.com" target="_blank">emilien@redhat.com</a>>> wrote:<br>
>> > ><br>
>> > ><br>
>> > ><br>
>> > > On Mon, Jul 27, 2020 at 5:27 PM Wesley Hayutin<br>
>> > <<a href="mailto:whayutin@redhat.com" target="_blank">whayutin@redhat.com</a> <mailto:<a href="mailto:whayutin@redhat.com" target="_blank">whayutin@redhat.com</a>>> wrote:<br>
>> > >><br>
>> > >> FYI...<br>
>> > >><br>
>> > >> If you find your jobs are failing with an error similar to<br>
>> > [1], you have been rate limited by <a href="http://docker.io" rel="noreferrer" target="_blank">docker.io</a> <<a href="http://docker.io" rel="noreferrer" target="_blank">http://docker.io</a>><br>
>> > via the upstream mirror system and have hit [2]. I've been<br>
>> > discussing the issue w/ upstream infra, rdo-infra and a few CI<br>
>> > engineers.<br>
>> > >><br>
>> > >> There are a few ways to mitigate the issue however I don't<br>
>> > see any of the options being completed very quickly so I'm<br>
>> > asking for your patience while this issue is socialized and<br>
>> > resolved.<br>
>> > >><br>
>> > >> For full transparency we're considering the following options.<br>
>> > >><br>
>> > >> 1. move off of <a href="http://docker.io" rel="noreferrer" target="_blank">docker.io</a> <<a href="http://docker.io" rel="noreferrer" target="_blank">http://docker.io</a>> to <a href="http://quay.io" rel="noreferrer" target="_blank">quay.io</a><br>
>> > <<a href="http://quay.io" rel="noreferrer" target="_blank">http://quay.io</a>><br>
>> > ><br>
>> > ><br>
>> > > <a href="http://quay.io" rel="noreferrer" target="_blank">quay.io</a> <<a href="http://quay.io" rel="noreferrer" target="_blank">http://quay.io</a>> also has API rate limit:<br>
>> > > <a href="https://docs.quay.io/issues/429.html" rel="noreferrer" target="_blank">https://docs.quay.io/issues/429.html</a><br>
>> > ><br>
>> > > Now I'm not sure about how many requests per seconds one can<br>
>> > do vs the other but this would need to be checked with the quay<br>
>> > team before changing anything.<br>
>> > > Also <a href="http://quay.io" rel="noreferrer" target="_blank">quay.io</a> <<a href="http://quay.io" rel="noreferrer" target="_blank">http://quay.io</a>> had its big downtimes as well,<br>
>> > SLA needs to be considered.<br>
>> > ><br>
>> > >> 2. local container builds for each job in master, possibly<br>
>> > ussuri<br>
>> > ><br>
>> > ><br>
>> > > Not convinced.<br>
>> > > You can look at CI logs:<br>
>> > > - pulling / updating / pushing container images from<br>
>> > <a href="http://docker.io" rel="noreferrer" target="_blank">docker.io</a> <<a href="http://docker.io" rel="noreferrer" target="_blank">http://docker.io</a>> to local registry takes ~10 min on<br>
>> > standalone (OVH)<br>
>> > > - building containers from scratch with updated repos and<br>
>> > pushing them to local registry takes ~29 min on standalone (OVH).<br>
>> > ><br>
>> > >><br>
>> > >> 3. parent child jobs upstream where rpms and containers will<br>
>> > be build and host artifacts for the child jobs<br>
>> > ><br>
>> > ><br>
>> > > Yes, we need to investigate that.<br>
>> > ><br>
>> > >><br>
>> > >> 4. remove some portion of the upstream jobs to lower the<br>
>> > impact we have on 3rd party infrastructure.<br>
>> > ><br>
>> > ><br>
>> > > I'm not sure I understand this one, maybe you can give an<br>
>> > example of what could be removed?<br>
>> ><br>
>> > We need to re-evaulate our use of scenarios (e.g. we have two<br>
>> > scenario010's both are non-voting). There's a reason we<br>
>> > historically<br>
>> > didn't want to add more jobs because of these types of resource<br>
>> > constraints. I think we've added new jobs recently and likely<br>
>> > need to<br>
>> > reduce what we run. Additionally we might want to look into reducing<br>
>> > what we run on stable branches as well.<br>
>> ><br>
>> ><br>
>> > Oh... removing jobs (I thought we would remove some steps of the jobs).<br>
>> > Yes big +1, this should be a continuous goal when working on CI, and<br>
>> > always evaluating what we need vs what we run now.<br>
>> ><br>
>> > We should look at:<br>
>> > 1) services deployed in scenarios that aren't worth testing (e.g.<br>
>> > deprecated or unused things) (and deprecate the unused things)<br>
>> > 2) jobs themselves (I don't have any example beside scenario010 but<br>
>> > I'm sure there are more).<br>
>> > --<br>
>> > Emilien Macchi<br>
>> ><br>
>> ><br>
>> > Thanks Alex, Emilien<br>
>> ><br>
>> > +1 to reviewing the catalog and adjusting things on an ongoing basis.<br>
>> ><br>
>> > All.. it looks like the issues with <a href="http://docker.io" rel="noreferrer" target="_blank">docker.io</a> <<a href="http://docker.io" rel="noreferrer" target="_blank">http://docker.io</a>> were<br>
>> > more of a flare up than a change in <a href="http://docker.io" rel="noreferrer" target="_blank">docker.io</a> <<a href="http://docker.io" rel="noreferrer" target="_blank">http://docker.io</a>> policy<br>
>> > or infrastructure [2]. The flare up started on July 27 8am utc and<br>
>> > ended on July 27 17:00 utc, see screenshots.<br>
>><br>
>> The numbers of image prepare workers and its exponential fallback<br>
>> intervals should be also adjusted. I've analysed the log snippet [0] for<br>
>> the connection reset counts by workers versus the times the rate<br>
>> limiting was triggered. See the details in the reported bug [1].<br>
>><br>
>> tl;dr -- for an example 5 sec interval 03:55:31,379 - 03:55:36,110:<br>
>><br>
>> Conn Reset Counts by a Worker PID:<br>
>> 3 58412<br>
>> 2 58413<br>
>> 3 58415<br>
>> 3 58417<br>
>><br>
>> which seems too much of (workers*reconnects) and triggers rate limiting<br>
>> immediately.<br>
>><br>
>> [0]<br>
>> <a href="https://13b475d7469ed7126ee9-28d4ad440f46f2186fe3f98464e57890.ssl.cf1.rackcdn.com/741228/6/check/tripleo-ci-centos-8-undercloud-containers/8e47836/logs/undercloud/var/log/tripleo-container-image-prepare.log" rel="noreferrer" target="_blank">https://13b475d7469ed7126ee9-28d4ad440f46f2186fe3f98464e57890.ssl.cf1.rackcdn.com/741228/6/check/tripleo-ci-centos-8-undercloud-containers/8e47836/logs/undercloud/var/log/tripleo-container-image-prepare.log</a><br>
>><br>
>> [1] <a href="https://bugs.launchpad.net/tripleo/+bug/1889372" rel="noreferrer" target="_blank">https://bugs.launchpad.net/tripleo/+bug/1889372</a><br>
>><br>
>> --<br>
>> Best regards,<br>
>> Bogdan Dobrelya,<br>
>> Irc #bogdando<br>
>><br>
><br>
> FYI..<br>
><br>
> The issue w/ "too many requests" is back. Expect delays and failures in attempting to merge your patches upstream across all branches. The issue is being tracked as a critical issue.<br>
<br>
Working with the infra folks and we have identified the authorization<br>
header as causing issues when we're rediected from <a href="http://docker.io" rel="noreferrer" target="_blank">docker.io</a> to<br>
cloudflare. I'll throw up a patch tomorrow to handle this case which<br>
should improve our usage of the cache. It needs some testing against<br>
other registries to ensure that we don't break authenticated fetching<br>
of resources.<br>
<br></blockquote><div>Thanks Alex! </div></div></div>