Open Stack

----- Original Message -----
> From: "Bob Ball" <bob.ball at citrix.com>
> To: "Sean Dague" <sean at dague.net>, "Dan Prince" <dprince at redhat.com>
> Cc: openstack-infra at lists.openstack.org
> Sent: Saturday, August 10, 2013 10:20:40 AM
> Subject: RE: [OpenStack-Infra] Smokestack posting +/-2 votes
> 
> Thanks for all the comments.  I understand the concerns around the infra team
> needing to manage the gating process, and I completely agree with that.
> 
> I do, however, think that there are ways to ensure that the gate is under
> infra's control yet can still accept a veto from a third party system.

I do think you are unto something here. I expect there are many groups out there with extra special (internal) testing setups that would love to gate various projects in some sort of manner. Network company X might be interested in running a 3rd party testing setup against neutron branches for example... and as that setup matures it might generally be well accepted enough that we'd want to gate on it in some manner.

I haven't quite put my head around all these problems but I'm not sure gating in this manner would look like what we call the gate today (Zuul's gate). Perhaps we call it a time limited gating feedback loop or something. We'd certainly need to put some thought into how this would work (and scale) if we wanted to allow multiple 3rd parties to participate in such a manner.

> 
> My understanding at the moment is that there is a check immediately prior to
> merging to make sure that there aren't any -2's added by core reviewers
> between the merge being approved and the gating jobs completing - is that
> right?
> If so, I'm not sure I understand the differences between a reviewer
> preventing a merge in this fashion and a known test failure from preventing
> it.
> Is the main concern there that smokestack may find breakages more often than
> core reviewers prevent a merge; and as such there might be more work for the
> gating jobs?

For that matter anyone who has "core" privs (and can -2 something) could write a script to do this. When I first wrote bellows for example I actually tested some things with my personal account so I had to be really careful not to allow an unwanted -2's to get posted to gerrit. Today what we use for SmokeStack only allows a -1 in the verified column.

> 
> In terms of ensuring the actual gate is not affected, it should be relatively
> easy to make Jenkins ignore SmokeStack if it ever broke in such a way that
> it prevented merging at the gate in an unwarranted fashion - either with a
> commit to change to the rules that are followed or it could even check a
> more dynamic switch prior to honouring the -2 (or even temporarily
> downgrading SmokeStack's permissions to prevent it from adding -2's).
> 
> Of course, we will continue to improve smokestack in its current role - but
> one of the reasons that we re-test in the gate is to catch any new failures
> (particularly since it may be several weeks since the last patch was
> uploaded).  I'm keen for those scenarios to also be tested by SmokeStack,
> which is why I'm trying to push for a veto at merge-time.
> 
> In terms of Xen running virtualised, yes it can be done and I have instances
> running on the Rackspace cloud at the moment.  Performance, however, is not
> great because SmokeStack runs in a VM within that virtualised XenServer.
> 
> Bob
> ________________________________________
> From: Sean Dague [sean at dague.net]
> Sent: 10 August 2013 13:12
> To: Dan Prince
> Cc: openstack-infra at lists.openstack.org
> Subject: Re: [OpenStack-Infra] Smokestack posting +/-2 votes
> 
> I'm basically with Jim 100% on this.
> 
> That being said, -1 on Verify is typically enough to prevent reviewers
> from pushing the Approve button unless they are sure that the -1 is
> bogus. I think the only time I've overriden a SmokeStack -1 was when
> it went crazy one weekend and -1ed everything due to a packaging
> issue. SmokeStack logs were good enough to be able to realize this
> wasn't a functional problem on the OpenStack side.
> 
> I think that we as a project are pretty respectful of -1s as long as
> they are consistently not false negatives. So I'd just say focus on
> getting +1/-1 voting back quickly.
> 
> On Fri, Aug 9, 2013 at 5:07 PM, Dan Prince <dprince at redhat.com> wrote:
> > In general I agree with what has been outline in Jim's reply below. Jim, I
> > have a couple replies inline though :)
> >
> > Bob, I think calling what we talked about a "gate" is probably the wrong
> > word. This is third party testing running as a pre-gate check which we are
> > trying to make a bit more formal. Can you prevent bad code from landing by
> > running tests as soon as someone prevents a branch: Yes!. Can you gate
> > everything and guarantee nothing slips in: No. Is it worth doing? I think
> > so because at the end of the day preventing bad code from landing is
> > always good. Much of the SmokeStack improvements we talked about (like
> > isolating package build failures from torpedo/test failures) is going to
> > happen anyway. So what we end up with is basically something where we can
> > confidently at pre-gate time run tests and provide feedback in the form of
> > a -1 in the verified column without someone like me having to approve it.
> > I understand you'd like that to be a -2 if the tests fail... but is
> > automatically posting a -1 good enough for now.
> >
> >
> > ----- Original Message -----
> >> From: "James E. Blair" <jeblair at openstack.org>
> >> To: "Bob Ball" <bob.ball at citrix.com>
> >> Cc: openstack-infra at lists.openstack.org
> >> Sent: Friday, August 9, 2013 11:56:28 AM
> >> Subject: Re: [OpenStack-Infra] Smokestack posting +/-2 votes
> >>
> >> Bob Ball <bob.ball at citrix.com> writes:
> >>
> >> > What are the thoughts on how an external system can add gating
> >> > comments?
> >>
> >> This is very unlikely to happen.
> >>
> >> Multiple gating systems for one project (or set of projects in our case)
> >> will not work.  Zuul does quite a bit of work to try to merge changes as
> >> quickly as possible, including speculative execution (where it builds
> >> git trees for several projects that represent the expected repo states
> >> in the future after many changes merge).  It also recognizes updates to
> >> the current state of changes in review, determining whether they can
> >> merge based on current review criteria, dependencies, etc.
> >>
> >> In other words, Zuul needs a global and authoritative view of what is
> >> happening.  If there is an outside influence that may cause an
> >> unpredictable change (such as a random -2 showing up), Zuul will not be
> >> able to make the correct choices.
> >>
> >> There is another important issue -- we can not gate on something that is
> >> not run by the infrastructure team.  We have a responsibility to the
> >> developers of the project to keep this system running.  If something
> >> breaks, any one of several people on the (open; anyone is welcome!)
> >> infrastructure team needs to be able to fix it.  If we started running
> >> gate tests on an external system and it stopped working, development on
> >> the entire project would stop, and we would be powerless to fix it.
> >>
> >> On a constructive note, much of what smokestack does could be
> >> incorporated into the current infrastructure.  The "devstack" jenkins
> >> nodes are becoming increasingly inaccurately named, as they are already
> >> running tests that have nothing to do with devstack.  Think of them as
> >> general purpose single-use jenkins slaves.  It would not be too
> >> difficult to have a jenkins job build packages, and then have further
> >> jenkins jobs use those packages to stand up a multi-node openstack (if
> >> you need three nodes, write three jobs that each run on a single-use
> >> full-access slave).  Obviously this would be quite resource intensive,
> >> but I believe it's possible.
> >
> >
> > Obviously I'm very interested in this idea. Using packages and
> > configuration management in multi-node environments to test upstream is
> > really the reason SmokeStack exists. Once upon a time we did have some
> > Jenkins jobs that used packages and once devstack came around interest
> > just wasn't there for us to maintain those (in the upstream openstack
> > Jenkins even). If things had gone differently... well lets just say there
> > may not be a SmokeStack around today. Perhaps that job was before its time
> > though and at the time the community seem violently against using packages
> > of any sort in upstream. Two years later: if there is an open door here
> > I'd like to pursue it... it is a lot of work though and I certainly think
> > there is a good bit to learn from the workflow we use in SmokeStack now.
> > In the meantime there is still a good bit of ground to hold with
> > SmokeStack which isn't easily swapped out.
> >
> > So this is encouraging... and we have lots of work to do. :)
> >
> >
> >>
> >> The problem, as I understand it, is that we can't run Xen nested on any
> >> of our current cloud providers.  If there is no way to do that, then I
> >> believe we would need a new source of test resources for this.  I think
> >> we talked about this a bit at the summit, but I think in general if
> >> someone wants to provide new testing resources, they would need to be
> >> sufficient to support our resource usage (which is large and growing),
> >> supported by a real ops team (because we aren't one) with something like
> >> an SLA.  We discussed some ideas around bare metal testing at the summit
> >> here: https://etherpad.openstack.org/havana-bare-metal-testing
> >>
> >> Even without gating, the advisory reporting from smokestack is hugely
> >> valuable.  I think any increase in reliability and speed (such as
> >> automating the failure detection as you were talking about) will be
> >> perceived by the review community and they will act accordingly, giving
> >> it significant weight in their reviews.
> >
> > Exactly.
> >
> >>
> >> -Jim
> >>
> >> _______________________________________________
> >> OpenStack-Infra mailing list
> >> OpenStack-Infra at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> >>
> >
> > _______________________________________________
> > OpenStack-Infra mailing list
> > OpenStack-Infra at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> 
> 
> 
> --
> Sean Dague
> http://dague.net
> 
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>