Open Stack

On Tue, May 28, 2019 at 11:01 PM Clark Boylan <cboylan at sapwetik.org> wrote:

> On Fri, May 24, 2019, at 12:17 PM, Alfredo Moralejo Alonso wrote:
> >
> >
> > On Fri, May 24, 2019 at 9:00 PM Jeremy Stanley <fungi at yuggoth.org>
> wrote:
> > > On 2019-05-24 20:41:58 +0200 (+0200), Alfredo Moralejo Alonso wrote:
> > >  > On Fri, May 24, 2019 at 8:33 PM Alfredo Moralejo Alonso <
> amoralej at redhat.com>
> > >  > wrote:
> > >  [...]
> > >  > > For periodic jobs we run every 4 hours i see:
> > >  > > - Last successful jobs started at 23-May-2019 21:06:42 UTC
> > >  > > - First failed jobs with this issue started at 24-May-2019
> 03:06:37 UTC
> > >  >
> > >  > We just got one that failed and sarted at 24-May-2019 02:25:00 UTC
> > >  [...]
> > >
> > >  Thanks, between the timing and symptoms, we're starting to suspect
> > >  this could be related to how older Git clients' requests are being
> > >  distributed between our load-balanced backends with inconsistent
> > >  packing. We're weighing some ideas for how to improve things there,
> > >  so stay tuned. The good news is that within the next 24 hours the
> > >  backends will be mostly in sync most of the time, so the symptoms
> > >  you're observing may subside for the most part once that's the case
> > >  (but to be entirely honest, we're still not quite sure just yet).
> >
> > ok, thanks for the update. I'll switch some jobs to use github mirrors
> > as workaround until we find it more stable.
>
> Can you test opendev.org again? We've updated the load balancer method
> from least connections to a source IP hash. We think this will address the
> state mismatch problems that you noticed previously. I have since been able
> to run a git clone --mirror and git fetch origin --prune myself (when it
> didn't work reliably in the past).
>
> Please let us know if it works now or if it doesn't. That data will be
> useful either way.
>
>
I've tested it both locally and in CI jobs and it's working fine in all
cases.

Thanks for your support,

Alfredo


> Clark
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20190529/6f231e9d/attachment.html>