Open Stack

On Fri, May 24, 2019, at 12:17 PM, Alfredo Moralejo Alonso wrote:
> 
> 
> On Fri, May 24, 2019 at 9:00 PM Jeremy Stanley <fungi at yuggoth.org> wrote:
> > On 2019-05-24 20:41:58 +0200 (+0200), Alfredo Moralejo Alonso wrote:
> >  > On Fri, May 24, 2019 at 8:33 PM Alfredo Moralejo Alonso <amoralej at redhat.com>
> >  > wrote:
> >  [...]
> >  > > For periodic jobs we run every 4 hours i see:
> >  > > - Last successful jobs started at 23-May-2019 21:06:42 UTC
> >  > > - First failed jobs with this issue started at 24-May-2019 03:06:37 UTC
> >  > 
> >  > We just got one that failed and sarted at 24-May-2019 02:25:00 UTC
> >  [...]
> > 
> >  Thanks, between the timing and symptoms, we're starting to suspect
> >  this could be related to how older Git clients' requests are being
> >  distributed between our load-balanced backends with inconsistent
> >  packing. We're weighing some ideas for how to improve things there,
> >  so stay tuned. The good news is that within the next 24 hours the
> >  backends will be mostly in sync most of the time, so the symptoms
> >  you're observing may subside for the most part once that's the case
> >  (but to be entirely honest, we're still not quite sure just yet).
> 
> ok, thanks for the update. I'll switch some jobs to use github mirrors 
> as workaround until we find it more stable.

Can you test opendev.org again? We've updated the load balancer method from least connections to a source IP hash. We think this will address the state mismatch problems that you noticed previously. I have since been able to run a git clone --mirror and git fetch origin --prune myself (when it didn't work reliably in the past).

Please let us know if it works now or if it doesn't. That data will be useful either way.

Clark