Open Stack

On 04/24/2013 12:18 PM, David Kranz wrote:
> Monty, I think there are really two issues here:
> 
> 1. How can we include a wider variety of tests (different configs,
> multi-node, performance, stress,  etc.) within the system that the infra
> group maintains?
> 2. Once we can run more kinds of tests, which of them are actually in
> the gate, and is the answer the same for all projects?

I agree with both of these points.

> There was a lot of discussion at the summit about progress on (1) and it
> seems you guys have real plans to address it, perhaps not without some
> controversy.

I think 1 is easier - both in terms of the things I want to drive in the
infra systems, and also in terms of things that other people want to
drive. It's an open system - so having more people (like bodepd) add
more tests that do more things is straightforward and limited only by
the resources people are interested in putting in to it

> But there was no real consensus about (2). I continue to believe that as
> the number of projects and configurations that we can test explodes, we
> will need a more principled way to make such decisions and run some
> tests as periodic or advisory instead. The more complex and "real" the
> testing gets, the harder it
> is to make the gate as reliable as it needs to be.

2 is the trick - because being in the gate itself means it's
prescriptive and effects other people - and it means that the infra team
needs to be able to address it.

The way you've framed this issue I think makes something clearer for me
that I've been trying to say- so I'm going to try a blunter approach on
part of it:

Because the infra team has to be able to support it, the infra team
needs/has veto power on what goes into the gate.

Now - that's a negative statement - but there's a corollary to that:

The infra team is open just like all of the other project teams.

If gating on a thing is important enough to someone that they actually
add enough staff to the project that become members of the infra team so
that the infra team as a whole feels appropriately staffed to deal with
it, then I don't think it'll be a problem. It's the same as with any
other team - if someone wants a thing in nova, they have to pony up the
folks to do it.

The current team barely has enough resources to handle the current workload.

The reason I get irate about parallel testing efforts is that it does
nothing to actually increase the bandwidth of the infra team itself.

So - to sum up

- We should definitely test more things and more combinations of things
- We need people to become involved enough to actually take an active
hand in accomplishing whichever of those things they find important
- We would love more people to become involved enough that they become
part of the infra core team.
- Same as all the other projects - working with others instead of doing
something locally is always harder at first - but also sees a greater
long-term benefit for everyone involved.

Monty

> On 4/24/2013 11:19 AM, Monty Taylor wrote:
>>
>> On 04/24/2013 11:07 AM, Brian Lamar wrote:
>>> On Apr 24, 2013, at 10:50 AM, Dan Prince <dprince at redhat.com> wrote:
>>>
>>>> ----- Original Message -----
>>>>> From: Brian Lamar <brian.lamar at rackspace.com>
>>>>> So the question again becomes how can we get XenServer into the
>>>>> gate.
>>>> Hi Lamar,
>>>>
>>>> I understand there is a long term goal to move towards larger scale
>>>> testing on bare metal w/ TripleO, etc. I'm very interested in that
>>>> myself too. In the short/mid term however perhaps using SmokeStack
>>>> (which supports testing multiple/flexible configurations) would
>>>> help cover the scenario you described below via pre-gate testing?
>>> I knew I was missing something big when I had finished this email.
>>> Absolutely. Maybe our first fight really is how we can get SmokeStack
>>> (and thus XenServer testing) as part of the gate.
>>>
>>> My understanding is that the main gripe is that it needs to be easy
>>> for people to see what failed and for them to have access to the
>>> nodes to troubleshoot failures. Is this what is holding things back?
>>> I'd love to get as many specifics as possible so we can work on
>>> getting things integrated. This is probably a question for
>>> Jim/Monty.
>> There are a few issues around this:
>>
>> a) puppet/packages in the gate is a HUGE issue that we do not have a
>> solution for yet
>> b) reliability engineering - running the amount of tests that we run
>> with the other system is a really big undertaking and there is a full
>> time team of people working basically just on making it work.
>> c) duplication of effort. This was the thing that made me yell at Dan
>> originally - although I think we're friends now. But we don't want two
>> things in the gate driving testing. We have zuul, and we have jenkins,
>> and we add new features to them constantly to keep up with the needs of
>> the gate. We don't want to add a second parallel system to a collection
>> of tools that's already a huge undertaking to manage, because it would
>> mean that everything we do to the zuul system would have to be
>> duplicated in the non-zuul system.
>>
>> Now don't get me wrong - I think smokestack has been a very useful tool
>> for the community and I think it's great that Dan has provided that
>> service for the community. But if we were to "integrate" it in to the
>> gate, I'd be much more interested in porting the things that it runs on
>> machines to the system that the OpenStack project already runs.
>>
>> That still doesn't address the issues with adding packaging and config
>> management to the gate though.
>>
>> If smokestack is useful to you to get started on this task, then by all
>> means use it! But in terms of gating, I'd really like for us to engage
>> in the hard work of meeting all of the requirements that are out there
>> in a way that's integrated with the work that's already being done.
>>
>>>>> I imagine it going something like this:
>>>>>
>>>>>
>>>>> 1) Hardware is provided with XenServer 6.1 installed
>>>> SmokeStack currently runs a single node (all in one) XenServer 5.6
>>>> test on upstream branches and comments on them (pre-gate testing).
>>>> It does this using 4 Dell R710's (provided by RAX). Why not upgrade
>>>> these machines to XenServer 6.1 to cover this?
>>> Sounds good to me. I think I should be able to handle this. I'm going
>>> to send some emails internally to get this process kicked off. If I
>>> don't reply with a timeline in a week can you ping me again?
>>>
>>>>> 2) Script for Jenkins will boot XenServer VM on hardware
>>>>> provided
>>>> SmokeStack uses this approach as well (via something called
>>>> Kytoon).
>>>>
>>>>> 3) VM will be configured using...devstack?
>>>> SmokeStack uses Puppet w/ Fedora packages I maintain :). I
>>>> understand people have various opinions on how to deploy things...
>>>> but if it works w/ packages I would argue that is probably "close
>>>> enough" to the test coverage you are looking for. The actual
>>>> configuration that is used for XenServer testing is easily
>>>> modifiable though so long as the puppet modules support it.
>>> Agreed.
>>>
>>>>> 4) This script will be an unofficial gate (aka 3rd party gate)
>>>>> until it is considered stable 5) This script will be integrated
>>>>> into the official process either as a gate-gate or a
>>>>> periodic-gate
>>>> SmokeStack comments on reviews pre-commit (via Bellows). It isn't a
>>>> gate (that ship sailed at the Diablo conference) but it is a well
>>>> defined 3rd party review tool ATM.
>>>>
>>>> ----
>>>>
>>>> Not trying to derail this into a conversation about SmokeStack. But
>>>> it does seem like it wouldn't take too much effort to be able to
>>>> cover your concerns in the short/mid term via something that is
>>>> already running and commenting on reviews upstream.
>>> Yup! My sincere apologies for forgetting about SmokeStack. I've used
>>> it enough to know that it has caught tons of stuff for us in the
>>> past. I very much would like to work with you to at least get
>>> XenServer gating. Once again if we can get an enumerated list of
>>> apprehensions from others it will go a long way towards success.
>> I hope I was clear above on apprehensions above. If not, PLEASE loop
>> back with me - it's not my intent to be negative or snarky or anything
>> here.
>>
>> Monty
>>
>> _______________________________________________
>> OpenStack-Infra mailing list
>> OpenStack-Infra at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> 
> 
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>