Open Stack

On Tue, Apr 18, 2017 at 2:28 PM, Emilien Macchi <emilien at redhat.com> wrote:

> On Mon, Apr 17, 2017 at 3:52 PM, Justin Kilpatrick <jkilpatr at redhat.com>
> wrote:
> > Because CI jobs tend to max out about 5 nodes there's a whole class of
> > minor bugs that make it into releases.
> >
> > What happens is that they never show up in small clouds, then when
> > they do show up in larger testing clouds the people deploying those
> > simply work around the issue and get onto what they where supposed to
> > be testing. These workarounds do get documented/BZ'd but since they
> > don't block anyone and only show up in large environments they become
> > hard for developers to fix.
> >
> > So the issue gets stuck in limbo, with nowhere to test a patchset and
> > no one owning the issue.
> >
> > These issues pile up and pretty soon there is a significant difference
> > between the default documented workflow and the 'scale' workflow which
> > is filled with workarounds which may or may not be documented
> > upstream.
> >
> > I'd like to propose getting these issues more visibility to having a
> > periodic upstream job that uses 20-30 ovb instances to do a larger
> > deployment. Maybe at 3am on a Sunday or some other time where there's
> > idle execution capability to exploit. The goal being to make these
> > sorts of issues more visible and hopefully get better at fixing them.
>
> Wait no, I know some folks at 3am on a Saturday night who use TripleO
> CI (ok that was a joke).
>
> > To be honest I'm not sure this is the best solution, but I'm seeing
> > this anti pattern across several issues and I think we should try and
> > come up with a solution.
> >
>
> Yes this proposal is really cool. There is an alternative to run this
> periodic scenario outside TripleO CI and send results via email maybe.
> But it is something we need to discuss with RDO Cloud people and see
> if we would have such resources to make it on a weekly frequency.
>

+1
I think with RDO Cloud it's possible to run a test of that scale either in
the
tripleo system or just report results, either would be great.  Until RDO
Cloud
is full production we might as well begin by running a job internally with
the master-tripleo-ci release config file.  The browbeat jobs are logging
[1] here
will be fairly simple step to run w/ the upstream content.

Adding Arx Cruz as he is point on a tool that distrubutes test results from
the tripleo periodic jobs that may come in handy for this scale test.  I'll
probably put you two in touch tomorrow.

I'm still looking for opportunities to run browbeat in upstream tripleo as
well.
Could be a productive sync up :)

[1] https://thirdparty-logs.rdoproject.org/

Thanks!



>
> Thanks for bringing this up, it's crucial for us to have this kind of
> feedback, now let's take actions.
> --
> Emilien Macchi
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170418/1e5b4b1f/attachment.html>