Open Stack

On Apr 24, 2013, at 1:24 PM, Monty Taylor <mordred at inaugust.com> wrote:

> 
> 
> On 04/24/2013 12:18 PM, David Kranz wrote:
>> Monty, I think there are really two issues here:
>> 
>> 1. How can we include a wider variety of tests (different configs,
>> multi-node, performance, stress,  etc.) within the system that the infra
>> group maintains?
>> 2. Once we can run more kinds of tests, which of them are actually in
>> the gate, and is the answer the same for all projects?
> 
> I agree with both of these points.
> 
>> There was a lot of discussion at the summit about progress on (1) and it
>> seems you guys have real plans to address it, perhaps not without some
>> controversy.
> 
> I think 1 is easier - both in terms of the things I want to drive in the
> infra systems, and also in terms of things that other people want to
> drive. It's an open system - so having more people (like bodepd) add
> more tests that do more things is straightforward and limited only by
> the resources people are interested in putting in to it
> 
>> But there was no real consensus about (2). I continue to believe that as
>> the number of projects and configurations that we can test explodes, we
>> will need a more principled way to make such decisions and run some
>> tests as periodic or advisory instead. The more complex and "real" the
>> testing gets, the harder it
>> is to make the gate as reliable as it needs to be.
> 
> 2 is the trick - because being in the gate itself means it's
> prescriptive and effects other people - and it means that the infra team
> needs to be able to address it.
> 
> The way you've framed this issue I think makes something clearer for me
> that I've been trying to say- so I'm going to try a blunter approach on
> part of it:
> 
> Because the infra team has to be able to support it, the infra team
> needs/has veto power on what goes into the gate.

Ok. I'm going to suggest something. It's probably been tried before, so please feel free to tear the idea apart.

I agree that the infra team has veto power on what goes into the official gate. Why does that need to be the only gate? We've been focusing a lot on testing changing before they get in to openstack/master but not a lot about what happens after they get in.

Why can't SmokeStack provide an openstack/smokestack branch which is always going to be slightly behind openstack/master? It slows things down, but so what? If you want something more stable you're going to have to wait longer, makes sense to me.

Think of it as cascading gates:

openstack/master (zuul) --> openstack/zuul+smokestack --> openstack/zuul+smokestack+cloudcafe --> openstack/zuul+smokestack+cloudcafe+rax

This branch chain would indicate that changes have undergone testing by zuul, smokestack, cloudcafe, and RAX internal testing (for example). Obviously branch names would get out of hand but we don't have to use that syntax and if naming is our biggest concern we're in good shape.

The best part about this is that everyone can choose what branch they want to pick from. Do you want to make sure your code is checked against performance regressions? Check out openstack/zuul+performance -- you're not going to EVER get *additional patches* in any of these branches. If tests fail, for example in our hypothetical performance test suite, the patch which caused the issue will either need to be reverted or fixed before the branch can move forward.

Once again, this will significantly slow down things for some deployers that want to make sure a lot of testing is done...but it won't necessarily hamper development time. Devs will just need to be aware that reverts are going to be more common?

I think with some tooling around this concept it could be pretty nice to see multiple download options for OpenStack -- although a canonical one should be chosen for basic users as to not add to confusion.

Thoughts?


> 
> Now - that's a negative statement - but there's a corollary to that:
> 
> The infra team is open just like all of the other project teams.
> 
> If gating on a thing is important enough to someone that they actually
> add enough staff to the project that become members of the infra team so
> that the infra team as a whole feels appropriately staffed to deal with
> it, then I don't think it'll be a problem. It's the same as with any
> other team - if someone wants a thing in nova, they have to pony up the
> folks to do it.
> 
> The current team barely has enough resources to handle the current workload.
> 
> The reason I get irate about parallel testing efforts is that it does
> nothing to actually increase the bandwidth of the infra team itself.
> 
> So - to sum up
> 
> - We should definitely test more things and more combinations of things
> - We need people to become involved enough to actually take an active
> hand in accomplishing whichever of those things they find important
> - We would love more people to become involved enough that they become
> part of the infra core team.
> - Same as all the other projects - working with others instead of doing
> something locally is always harder at first - but also sees a greater
> long-term benefit for everyone involved.
> 
> Monty
> 
>> On 4/24/2013 11:19 AM, Monty Taylor wrote:
>>> 
>>> On 04/24/2013 11:07 AM, Brian Lamar wrote:
>>>> On Apr 24, 2013, at 10:50 AM, Dan Prince <dprince at redhat.com> wrote:
>>>> 
>>>>> ----- Original Message -----
>>>>>> From: Brian Lamar <brian.lamar at rackspace.com>
>>>>>> So the question again becomes how can we get XenServer into the
>>>>>> gate.
>>>>> Hi Lamar,
>>>>> 
>>>>> I understand there is a long term goal to move towards larger scale
>>>>> testing on bare metal w/ TripleO, etc. I'm very interested in that
>>>>> myself too. In the short/mid term however perhaps using SmokeStack
>>>>> (which supports testing multiple/flexible configurations) would
>>>>> help cover the scenario you described below via pre-gate testing?
>>>> I knew I was missing something big when I had finished this email.
>>>> Absolutely. Maybe our first fight really is how we can get SmokeStack
>>>> (and thus XenServer testing) as part of the gate.
>>>> 
>>>> My understanding is that the main gripe is that it needs to be easy
>>>> for people to see what failed and for them to have access to the
>>>> nodes to troubleshoot failures. Is this what is holding things back?
>>>> I'd love to get as many specifics as possible so we can work on
>>>> getting things integrated. This is probably a question for
>>>> Jim/Monty.
>>> There are a few issues around this:
>>> 
>>> a) puppet/packages in the gate is a HUGE issue that we do not have a
>>> solution for yet
>>> b) reliability engineering - running the amount of tests that we run
>>> with the other system is a really big undertaking and there is a full
>>> time team of people working basically just on making it work.
>>> c) duplication of effort. This was the thing that made me yell at Dan
>>> originally - although I think we're friends now. But we don't want two
>>> things in the gate driving testing. We have zuul, and we have jenkins,
>>> and we add new features to them constantly to keep up with the needs of
>>> the gate. We don't want to add a second parallel system to a collection
>>> of tools that's already a huge undertaking to manage, because it would
>>> mean that everything we do to the zuul system would have to be
>>> duplicated in the non-zuul system.
>>> 
>>> Now don't get me wrong - I think smokestack has been a very useful tool
>>> for the community and I think it's great that Dan has provided that
>>> service for the community. But if we were to "integrate" it in to the
>>> gate, I'd be much more interested in porting the things that it runs on
>>> machines to the system that the OpenStack project already runs.
>>> 
>>> That still doesn't address the issues with adding packaging and config
>>> management to the gate though.
>>> 
>>> If smokestack is useful to you to get started on this task, then by all
>>> means use it! But in terms of gating, I'd really like for us to engage
>>> in the hard work of meeting all of the requirements that are out there
>>> in a way that's integrated with the work that's already being done.
>>> 
>>>>>> I imagine it going something like this:
>>>>>> 
>>>>>> 
>>>>>> 1) Hardware is provided with XenServer 6.1 installed
>>>>> SmokeStack currently runs a single node (all in one) XenServer 5.6
>>>>> test on upstream branches and comments on them (pre-gate testing).
>>>>> It does this using 4 Dell R710's (provided by RAX). Why not upgrade
>>>>> these machines to XenServer 6.1 to cover this?
>>>> Sounds good to me. I think I should be able to handle this. I'm going
>>>> to send some emails internally to get this process kicked off. If I
>>>> don't reply with a timeline in a week can you ping me again?
>>>> 
>>>>>> 2) Script for Jenkins will boot XenServer VM on hardware
>>>>>> provided
>>>>> SmokeStack uses this approach as well (via something called
>>>>> Kytoon).
>>>>> 
>>>>>> 3) VM will be configured using...devstack?
>>>>> SmokeStack uses Puppet w/ Fedora packages I maintain :). I
>>>>> understand people have various opinions on how to deploy things...
>>>>> but if it works w/ packages I would argue that is probably "close
>>>>> enough" to the test coverage you are looking for. The actual
>>>>> configuration that is used for XenServer testing is easily
>>>>> modifiable though so long as the puppet modules support it.
>>>> Agreed.
>>>> 
>>>>>> 4) This script will be an unofficial gate (aka 3rd party gate)
>>>>>> until it is considered stable 5) This script will be integrated
>>>>>> into the official process either as a gate-gate or a
>>>>>> periodic-gate
>>>>> SmokeStack comments on reviews pre-commit (via Bellows). It isn't a
>>>>> gate (that ship sailed at the Diablo conference) but it is a well
>>>>> defined 3rd party review tool ATM.
>>>>> 
>>>>> ----
>>>>> 
>>>>> Not trying to derail this into a conversation about SmokeStack. But
>>>>> it does seem like it wouldn't take too much effort to be able to
>>>>> cover your concerns in the short/mid term via something that is
>>>>> already running and commenting on reviews upstream.
>>>> Yup! My sincere apologies for forgetting about SmokeStack. I've used
>>>> it enough to know that it has caught tons of stuff for us in the
>>>> past. I very much would like to work with you to at least get
>>>> XenServer gating. Once again if we can get an enumerated list of
>>>> apprehensions from others it will go a long way towards success.
>>> I hope I was clear above on apprehensions above. If not, PLEASE loop
>>> back with me - it's not my intent to be negative or snarky or anything
>>> here.
>>> 
>>> Monty
>>> 
>>> _______________________________________________
>>> OpenStack-Infra mailing list
>>> OpenStack-Infra at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>> 
>> 
>> _______________________________________________
>> OpenStack-Infra mailing list
>> OpenStack-Infra at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>> 
> 
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra