On Wed, Aug 19, 2020 at 8:59 AM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2020-08-19 15:40:08 +0200 (+0200), Cédric Jeanneret wrote:
On 8/19/20 3:23 PM, Alex Schultz wrote: [...]
1) stop using mirrors (not ideal but likely makes this go away). Alternatively switch stable branches off the mirrors due to a reduced number of executions and leave mirrors configured on master only (or vice versa).
might be good, but it might lead to some other issues - docker might want to rate-limit on container owner. I wouldn't be surprised if they go that way in the future. Could be OK as a first "unlocking step". [...]
Be aware that there is another side effect: right now the images are being served from a cache within the same environment as the test nodes, and instead your jobs will begin fetching them over the Internet. This may mean longer average job run time, and a higher percentage of download failures due to network hiccups (whether these will be of a greater frequency than the API rate limit blocking, it's hard to guess). It also necessarily means significantly more bandwidth utilization for our resource donors, particularly as TripleO consumes far more job resources than any other project already.
Yea I know so we're trying to find a solution that doesn't make it worse. It would be great if we could have any visibility into the cache hit ratio/requests going through these mirrors to know if we have changes that are improving things or making it worse.
I wonder if there's a middle ground: finding a way to use the cache for fetching images, but connecting straight to Dockerhub when you're querying metadata? It sounds like the metadata requests represent a majority of the actual Dockerhub API calls anyway, and can't be cached regardless.
Maybe, but at the moment i'm working on not even doing the requests at all which would be better. Next i'll look into that but the mirror config is handled before we even start requesting things
-- Jeremy Stanley