Open Stack

Monty, I think there are really two issues here:

1. How can we include a wider variety of tests (different configs, 
multi-node, performance, stress,  etc.) within the system that the infra 
group maintains?
2. Once we can run more kinds of tests, which of them are actually in 
the gate, and is the answer the same for all projects?

There was a lot of discussion at the summit about progress on (1) and it 
seems you guys have real plans to address it, perhaps not without some 
controversy.
But there was no real consensus about (2). I continue to believe that as 
the number of projects and configurations that we can test explodes, we
will need a more principled way to make such decisions and run some 
tests as periodic or advisory instead. The more complex and "real" the 
testing gets, the harder it
is to make the gate as reliable as it needs to be.

  -David

On 4/24/2013 11:19 AM, Monty Taylor wrote:
>
> On 04/24/2013 11:07 AM, Brian Lamar wrote:
>> On Apr 24, 2013, at 10:50 AM, Dan Prince <dprince at redhat.com> wrote:
>>
>>> ----- Original Message -----
>>>> From: Brian Lamar <brian.lamar at rackspace.com>
>>>> So the question again becomes how can we get XenServer into the
>>>> gate.
>>> Hi Lamar,
>>>
>>> I understand there is a long term goal to move towards larger scale
>>> testing on bare metal w/ TripleO, etc. I'm very interested in that
>>> myself too. In the short/mid term however perhaps using SmokeStack
>>> (which supports testing multiple/flexible configurations) would
>>> help cover the scenario you described below via pre-gate testing?
>> I knew I was missing something big when I had finished this email.
>> Absolutely. Maybe our first fight really is how we can get SmokeStack
>> (and thus XenServer testing) as part of the gate.
>>
>> My understanding is that the main gripe is that it needs to be easy
>> for people to see what failed and for them to have access to the
>> nodes to troubleshoot failures. Is this what is holding things back?
>> I'd love to get as many specifics as possible so we can work on
>> getting things integrated. This is probably a question for
>> Jim/Monty.
> There are a few issues around this:
>
> a) puppet/packages in the gate is a HUGE issue that we do not have a
> solution for yet
> b) reliability engineering - running the amount of tests that we run
> with the other system is a really big undertaking and there is a full
> time team of people working basically just on making it work.
> c) duplication of effort. This was the thing that made me yell at Dan
> originally - although I think we're friends now. But we don't want two
> things in the gate driving testing. We have zuul, and we have jenkins,
> and we add new features to them constantly to keep up with the needs of
> the gate. We don't want to add a second parallel system to a collection
> of tools that's already a huge undertaking to manage, because it would
> mean that everything we do to the zuul system would have to be
> duplicated in the non-zuul system.
>
> Now don't get me wrong - I think smokestack has been a very useful tool
> for the community and I think it's great that Dan has provided that
> service for the community. But if we were to "integrate" it in to the
> gate, I'd be much more interested in porting the things that it runs on
> machines to the system that the OpenStack project already runs.
>
> That still doesn't address the issues with adding packaging and config
> management to the gate though.
>
> If smokestack is useful to you to get started on this task, then by all
> means use it! But in terms of gating, I'd really like for us to engage
> in the hard work of meeting all of the requirements that are out there
> in a way that's integrated with the work that's already being done.
>
>>>> I imagine it going something like this:
>>>>
>>>>
>>>> 1) Hardware is provided with XenServer 6.1 installed
>>> SmokeStack currently runs a single node (all in one) XenServer 5.6
>>> test on upstream branches and comments on them (pre-gate testing).
>>> It does this using 4 Dell R710's (provided by RAX). Why not upgrade
>>> these machines to XenServer 6.1 to cover this?
>> Sounds good to me. I think I should be able to handle this. I'm going
>> to send some emails internally to get this process kicked off. If I
>> don't reply with a timeline in a week can you ping me again?
>>
>>>> 2) Script for Jenkins will boot XenServer VM on hardware
>>>> provided
>>> SmokeStack uses this approach as well (via something called
>>> Kytoon).
>>>
>>>> 3) VM will be configured using...devstack?
>>> SmokeStack uses Puppet w/ Fedora packages I maintain :). I
>>> understand people have various opinions on how to deploy things...
>>> but if it works w/ packages I would argue that is probably "close
>>> enough" to the test coverage you are looking for. The actual
>>> configuration that is used for XenServer testing is easily
>>> modifiable though so long as the puppet modules support it.
>> Agreed.
>>
>>>> 4) This script will be an unofficial gate (aka 3rd party gate)
>>>> until it is considered stable 5) This script will be integrated
>>>> into the official process either as a gate-gate or a
>>>> periodic-gate
>>> SmokeStack comments on reviews pre-commit (via Bellows). It isn't a
>>> gate (that ship sailed at the Diablo conference) but it is a well
>>> defined 3rd party review tool ATM.
>>>
>>> ----
>>>
>>> Not trying to derail this into a conversation about SmokeStack. But
>>> it does seem like it wouldn't take too much effort to be able to
>>> cover your concerns in the short/mid term via something that is
>>> already running and commenting on reviews upstream.
>> Yup! My sincere apologies for forgetting about SmokeStack. I've used
>> it enough to know that it has caught tons of stuff for us in the
>> past. I very much would like to work with you to at least get
>> XenServer gating. Once again if we can get an enumerated list of
>> apprehensions from others it will go a long way towards success.
> I hope I was clear above on apprehensions above. If not, PLEASE loop
> back with me - it's not my intent to be negative or snarky or anything here.
>
> Monty
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra