Open Stack

On Wed, Aug 19, 2020 at 8:59 AM Jeremy Stanley <fungi at yuggoth.org> wrote:
>
> On 2020-08-19 15:40:08 +0200 (+0200), Cédric Jeanneret wrote:
> > On 8/19/20 3:23 PM, Alex Schultz wrote:
> [...]
> > > 1) stop using mirrors (not ideal but likely makes this go away).
> > > Alternatively switch stable branches off the mirrors due to a reduced
> > > number of executions and leave mirrors configured on master only (or
> > > vice versa).
> >
> > might be good, but it might lead to some other issues - docker might
> > want to rate-limit on container owner. I wouldn't be surprised if they
> > go that way in the future. Could be OK as a first "unlocking step".
> [...]
>
> Be aware that there is another side effect: right now the images are
> being served from a cache within the same environment as the test
> nodes, and instead your jobs will begin fetching them over the
> Internet. This may mean longer average job run time, and a higher
> percentage of download failures due to network hiccups (whether
> these will be of a greater frequency than the API rate limit
> blocking, it's hard to guess). It also necessarily means
> significantly more bandwidth utilization for our resource donors,
> particularly as TripleO consumes far more job resources than any
> other project already.
>

Yea I know so we're trying to find a solution that doesn't make it
worse.  It would be great if we could have any visibility into the
cache hit ratio/requests going through these mirrors to know if we
have changes that are improving things or making it worse.

> I wonder if there's a middle ground: finding a way to use the cache
> for fetching images, but connecting straight to Dockerhub when
> you're querying metadata? It sounds like the metadata requests
> represent a majority of the actual Dockerhub API calls anyway, and
> can't be cached regardless.

Maybe, but at the moment i'm working on not even doing the requests at
all which would be better.  Next i'll look into that but the mirror
config is handled before we even start requesting things

> --
> Jeremy Stanley