Open Stack

On 04/25/2013 01:27 PM, Brian Lamar wrote:
> 
> On Apr 24, 2013, at 1:24 PM, Monty Taylor <mordred at inaugust.com>
> wrote:
> 
>> 
>> 
>> On 04/24/2013 12:18 PM, David Kranz wrote:
>>> Monty, I think there are really two issues here:
>>> 
>>> 1. How can we include a wider variety of tests (different
>>> configs, multi-node, performance, stress,  etc.) within the
>>> system that the infra group maintains? 2. Once we can run more
>>> kinds of tests, which of them are actually in the gate, and is
>>> the answer the same for all projects?
>> 
>> I agree with both of these points.
>> 
>>> There was a lot of discussion at the summit about progress on (1)
>>> and it seems you guys have real plans to address it, perhaps not
>>> without some controversy.
>> 
>> I think 1 is easier - both in terms of the things I want to drive
>> in the infra systems, and also in terms of things that other people
>> want to drive. It's an open system - so having more people (like
>> bodepd) add more tests that do more things is straightforward and
>> limited only by the resources people are interested in putting in
>> to it
>> 
>>> But there was no real consensus about (2). I continue to believe
>>> that as the number of projects and configurations that we can
>>> test explodes, we will need a more principled way to make such
>>> decisions and run some tests as periodic or advisory instead. The
>>> more complex and "real" the testing gets, the harder it is to
>>> make the gate as reliable as it needs to be.
>> 
>> 2 is the trick - because being in the gate itself means it's 
>> prescriptive and effects other people - and it means that the infra
>> team needs to be able to address it.
>> 
>> The way you've framed this issue I think makes something clearer
>> for me that I've been trying to say- so I'm going to try a blunter
>> approach on part of it:
>> 
>> Because the infra team has to be able to support it, the infra
>> team needs/has veto power on what goes into the gate.
> 
> Ok. I'm going to suggest something. It's probably been tried before,
> so please feel free to tear the idea apart.
> 
> I agree that the infra team has veto power on what goes into the
> official gate. Why does that need to be the only gate? We've been
> focusing a lot on testing changing before they get in to
> openstack/master but not a lot about what happens after they get in.

Nothing at all is stopping there from being downstream gates. The main
thing is though - my concern is still there - resources that could be
spent improving the gate that gates the main branch of the project
instead wind up being diverted to work on a branch that isn't the branch
the project works on.

> Why can't SmokeStack provide an openstack/smokestack branch which is
> always going to be slightly behind openstack/master? It slows things
> down, but so what? If you want something more stable you're going to
> have to wait longer, makes sense to me.
> 
> Think of it as cascading gates:
> 
> openstack/master (zuul) --> openstack/zuul+smokestack -->
> openstack/zuul+smokestack+cloudcafe -->
> openstack/zuul+smokestack+cloudcafe+rax
> 
> This branch chain would indicate that changes have undergone testing
> by zuul, smokestack, cloudcafe, and RAX internal testing (for
> example). Obviously branch names would get out of hand but we don't
> have to use that syntax and if naming is our biggest concern we're in
> good shape.
> 
> The best part about this is that everyone can choose what branch they
> want to pick from. Do you want to make sure your code is checked
> against performance regressions? Check out openstack/zuul+performance
> -- you're not going to EVER get *additional patches* in any of these
> branches. If tests fail, for example in our hypothetical performance
> test suite, the patch which caused the issue will either need to be
> reverted or fixed before the branch can move forward.
> 
> Once again, this will significantly slow down things for some
> deployers that want to make sure a lot of testing is done...but it
> won't necessarily hamper development time. Devs will just need to be
> aware that reverts are going to be more common?
> 
> I think with some tooling around this concept it could be pretty nice
> to see multiple download options for OpenStack -- although a
> canonical one should be chosen for basic users as to not add to
> confusion.
> 
> Thoughts?

I think this would destroy OpenStack, and I do not think it solves
anything. What this does is codifies in a quasi-upstream the terrible
divergent behavior that most of the current downstreams are trying to
get away from at the moment.

That Rackspace tests things additionally in private is a bug that both
we and Rackspace have expressed interest in fixing.

The more branches there are, the more possibility there is for developer
confusion. Should a dev start dev work on openstack/master? Or
openstack/smokestack? Then where do they send their patch to land?

What I really want to see is for all of the effort that would go in  to
that go in to the infra team and the gate instead. OR, for things that
really and trully cannot gate master, developing a solid set of
post-merge tests that people find important.

I want to see more people join the infra team and therefor be able to
drive more testing of divergent needs in the area where people are
focusing their development. I feel that divergent branches and divergent
testing resources for those branches will reduce even further the
likelihood of this happening.

Let's work on getting you guys to be an integrated part of the infra
team so that it doesn't feel like you need to build your own solutions
to the side. We want your solutions in the main system! We like you! :)

>> 
>> Now - that's a negative statement - but there's a corollary to
>> that:
>> 
>> The infra team is open just like all of the other project teams.
>> 
>> If gating on a thing is important enough to someone that they
>> actually add enough staff to the project that become members of the
>> infra team so that the infra team as a whole feels appropriately
>> staffed to deal with it, then I don't think it'll be a problem.
>> It's the same as with any other team - if someone wants a thing in
>> nova, they have to pony up the folks to do it.
>> 
>> The current team barely has enough resources to handle the current
>> workload.
>> 
>> The reason I get irate about parallel testing efforts is that it
>> does nothing to actually increase the bandwidth of the infra team
>> itself.
>> 
>> So - to sum up
>> 
>> - We should definitely test more things and more combinations of
>> things - We need people to become involved enough to actually take
>> an active hand in accomplishing whichever of those things they find
>> important - We would love more people to become involved enough
>> that they become part of the infra core team. - Same as all the
>> other projects - working with others instead of doing something
>> locally is always harder at first - but also sees a greater 
>> long-term benefit for everyone involved.
>> 
>> Monty
>> 
>>> On 4/24/2013 11:19 AM, Monty Taylor wrote:
>>>> 
>>>> On 04/24/2013 11:07 AM, Brian Lamar wrote:
>>>>> On Apr 24, 2013, at 10:50 AM, Dan Prince <dprince at redhat.com>
>>>>> wrote:
>>>>> 
>>>>>> ----- Original Message -----
>>>>>>> From: Brian Lamar <brian.lamar at rackspace.com> So the
>>>>>>> question again becomes how can we get XenServer into the 
>>>>>>> gate.
>>>>>> Hi Lamar,
>>>>>> 
>>>>>> I understand there is a long term goal to move towards
>>>>>> larger scale testing on bare metal w/ TripleO, etc. I'm
>>>>>> very interested in that myself too. In the short/mid term
>>>>>> however perhaps using SmokeStack (which supports testing
>>>>>> multiple/flexible configurations) would help cover the
>>>>>> scenario you described below via pre-gate testing?
>>>>> I knew I was missing something big when I had finished this
>>>>> email. Absolutely. Maybe our first fight really is how we can
>>>>> get SmokeStack (and thus XenServer testing) as part of the
>>>>> gate.
>>>>> 
>>>>> My understanding is that the main gripe is that it needs to
>>>>> be easy for people to see what failed and for them to have
>>>>> access to the nodes to troubleshoot failures. Is this what is
>>>>> holding things back? I'd love to get as many specifics as
>>>>> possible so we can work on getting things integrated. This is
>>>>> probably a question for Jim/Monty.
>>>> There are a few issues around this:
>>>> 
>>>> a) puppet/packages in the gate is a HUGE issue that we do not
>>>> have a solution for yet b) reliability engineering - running
>>>> the amount of tests that we run with the other system is a
>>>> really big undertaking and there is a full time team of people
>>>> working basically just on making it work. c) duplication of
>>>> effort. This was the thing that made me yell at Dan originally
>>>> - although I think we're friends now. But we don't want two 
>>>> things in the gate driving testing. We have zuul, and we have
>>>> jenkins, and we add new features to them constantly to keep up
>>>> with the needs of the gate. We don't want to add a second
>>>> parallel system to a collection of tools that's already a huge
>>>> undertaking to manage, because it would mean that everything we
>>>> do to the zuul system would have to be duplicated in the
>>>> non-zuul system.
>>>> 
>>>> Now don't get me wrong - I think smokestack has been a very
>>>> useful tool for the community and I think it's great that Dan
>>>> has provided that service for the community. But if we were to
>>>> "integrate" it in to the gate, I'd be much more interested in
>>>> porting the things that it runs on machines to the system that
>>>> the OpenStack project already runs.
>>>> 
>>>> That still doesn't address the issues with adding packaging and
>>>> config management to the gate though.
>>>> 
>>>> If smokestack is useful to you to get started on this task,
>>>> then by all means use it! But in terms of gating, I'd really
>>>> like for us to engage in the hard work of meeting all of the
>>>> requirements that are out there in a way that's integrated with
>>>> the work that's already being done.
>>>> 
>>>>>>> I imagine it going something like this:
>>>>>>> 
>>>>>>> 
>>>>>>> 1) Hardware is provided with XenServer 6.1 installed
>>>>>> SmokeStack currently runs a single node (all in one)
>>>>>> XenServer 5.6 test on upstream branches and comments on
>>>>>> them (pre-gate testing). It does this using 4 Dell R710's
>>>>>> (provided by RAX). Why not upgrade these machines to
>>>>>> XenServer 6.1 to cover this?
>>>>> Sounds good to me. I think I should be able to handle this.
>>>>> I'm going to send some emails internally to get this process
>>>>> kicked off. If I don't reply with a timeline in a week can
>>>>> you ping me again?
>>>>> 
>>>>>>> 2) Script for Jenkins will boot XenServer VM on hardware 
>>>>>>> provided
>>>>>> SmokeStack uses this approach as well (via something
>>>>>> called Kytoon).
>>>>>> 
>>>>>>> 3) VM will be configured using...devstack?
>>>>>> SmokeStack uses Puppet w/ Fedora packages I maintain :). I 
>>>>>> understand people have various opinions on how to deploy
>>>>>> things... but if it works w/ packages I would argue that is
>>>>>> probably "close enough" to the test coverage you are
>>>>>> looking for. The actual configuration that is used for
>>>>>> XenServer testing is easily modifiable though so long as
>>>>>> the puppet modules support it.
>>>>> Agreed.
>>>>> 
>>>>>>> 4) This script will be an unofficial gate (aka 3rd party
>>>>>>> gate) until it is considered stable 5) This script will
>>>>>>> be integrated into the official process either as a
>>>>>>> gate-gate or a periodic-gate
>>>>>> SmokeStack comments on reviews pre-commit (via Bellows). It
>>>>>> isn't a gate (that ship sailed at the Diablo conference)
>>>>>> but it is a well defined 3rd party review tool ATM.
>>>>>> 
>>>>>> ----
>>>>>> 
>>>>>> Not trying to derail this into a conversation about
>>>>>> SmokeStack. But it does seem like it wouldn't take too much
>>>>>> effort to be able to cover your concerns in the short/mid
>>>>>> term via something that is already running and commenting
>>>>>> on reviews upstream.
>>>>> Yup! My sincere apologies for forgetting about SmokeStack.
>>>>> I've used it enough to know that it has caught tons of stuff
>>>>> for us in the past. I very much would like to work with you
>>>>> to at least get XenServer gating. Once again if we can get an
>>>>> enumerated list of apprehensions from others it will go a
>>>>> long way towards success.
>>>> I hope I was clear above on apprehensions above. If not, PLEASE
>>>> loop back with me - it's not my intent to be negative or snarky
>>>> or anything here.
>>>> 
>>>> Monty
>>>> 
>>>> _______________________________________________ OpenStack-Infra
>>>> mailing list OpenStack-Infra at lists.openstack.org 
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>
>>>
>>>
>>>> 
_______________________________________________
>>> OpenStack-Infra mailing list OpenStack-Infra at lists.openstack.org 
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>
>>
>>
>>> 
_______________________________________________
>> OpenStack-Infra mailing list OpenStack-Infra at lists.openstack.org 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
>> 
>