Open Stack

On Tue, Jul 28, 2020, 18:59 Emilien Macchi <emilien at redhat.com> wrote:

>
>
> On Tue, Jul 28, 2020 at 9:20 AM Alex Schultz <aschultz at redhat.com> wrote:
>
>> On Tue, Jul 28, 2020 at 7:13 AM Emilien Macchi <emilien at redhat.com>
>> wrote:
>> >
>> >
>> >
>> > On Mon, Jul 27, 2020 at 5:27 PM Wesley Hayutin <whayutin at redhat.com>
>> wrote:
>> >>
>> >> FYI...
>> >>
>> >> If you find your jobs are failing with an error similar to [1], you
>> have been rate limited by docker.io via the upstream mirror system and
>> have hit [2].  I've been discussing the issue w/ upstream infra, rdo-infra
>> and a few CI engineers.
>> >>
>> >> There are a few ways to mitigate the issue however I don't see any of
>> the options being completed very quickly so I'm asking for your patience
>> while this issue is socialized and resolved.
>> >>
>> >> For full transparency we're considering the following options.
>> >>
>> >> 1. move off of docker.io to quay.io
>> >
>> >
>> > quay.io also has API rate limit:
>> > https://docs.quay.io/issues/429.html
>> >
>> > Now I'm not sure about how many requests per seconds one can do vs the
>> other but this would need to be checked with the quay team before changing
>> anything.
>> > Also quay.io had its big downtimes as well, SLA needs to be considered.
>> >
>> >> 2. local container builds for each job in master, possibly ussuri
>> >
>> >
>> > Not convinced.
>> > You can look at CI logs:
>> > - pulling / updating / pushing container images from docker.io to
>> local registry takes ~10 min on standalone (OVH)
>> > - building containers from scratch with updated repos and pushing them
>> to local registry takes ~29 min on standalone (OVH).
>> >
>> >>
>> >> 3. parent child jobs upstream where rpms and containers will be build
>> and host artifacts for the child jobs
>> >
>> >
>> > Yes, we need to investigate that.
>> >
>> >>
>> >> 4. remove some portion of the upstream jobs to lower the impact we
>> have on 3rd party infrastructure.
>> >
>> >
>> > I'm not sure I understand this one, maybe you can give an example of
>> what could be removed?
>>
>> We need to re-evaulate our use of scenarios (e.g. we have two
>> scenario010's both are non-voting).  There's a reason we historically
>> didn't want to add more jobs because of these types of resource
>> constraints.  I think we've added new jobs recently and likely need to
>> reduce what we run. Additionally we might want to look into reducing
>> what we run on stable branches as well.
>>
>
> Oh... removing jobs (I thought we would remove some steps of the jobs).
> Yes big +1, this should be a continuous goal when working on CI, and
> always evaluating what we need vs what we run now.
>
> We should look at:
> 1) services deployed in scenarios that aren't worth testing (e.g.
> deprecated or unused things) (and deprecate the unused things)
> 2) jobs themselves (I don't have any example beside scenario010 but I'm
> sure there are more).
>
Isn't scenario010 testing octavia? Though I've seen toggling between
voting/non-voting due to different issues for a long time.

> --
> Emilien Macchi
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200729/d05b0f2b/attachment.html>