Open Stack

On 10/30/2014 11:12 AM, Sean Dague wrote:
> On 10/30/2014 10:47 AM, Eoghan Glynn wrote:
>>> Matthew wrote:
>>>
>>> This would have the advantage of making tempest slimmer for every project
>>> and begin the process of getting projects to take responsibility for their
>>> functional testing rather than relying on tempest.
>> [much snipping]
>>
>>> Sean wrote:
>>>
>>> Ok, so part of this remains to be seen about what the biggest bang for the
>>> buck is. The class of bugs I feel like we need to nail in Nova right now are
>>> going to require tests that bring up pieces of the wsgi stack, but are
>>> probably not runable on a real deploy. Again, this is about debugability.
>> So this notion of the biggest bang for our buck is an aspect of the drive
>> for in-tree functional tests, that's not entirely clear to me as yet.
>>
>> i.e. whether individual projects should be prioritizing within this effort:
>>
>> (a) the creation of net-new coverage for scenarios (especially known or
>>      suspected bugs) that were not previously tested, in a non-unit sense
>>
>> (b) the relocation of existing integration test coverage from Tempest to
>>      the project trees, in order to make the management of Tempest more
>>      tractable
>>
>> It feels like there may be a tension between (a) and (b) in terms of the
>> pay-off for this effort. I'd interested in hearing other opinions on this,
>> on what aspect projects are expecting (and expected) to concentrate on
>> initially.
> For what it's worth I have a bunch of early targets listed for Nova for
> our summit session -
> https://etherpad.openstack.org/p/kilo-nova-functional-testing
>
> My focus in kilo is going to be first about A), as that provides value
> out of the gate (pun intended). Then peel off some stuff from B as makes
> sense.
>
> 	-Sean
>
That seems sensible from the nova point of view and overall health, and 
not all projects have to pursue the same priorities at the same time. 
But a big part of the benefit of (b) is the impact it has on all the 
other projects in that other projects will stop getting as many gate 
failures, and that benefit could be achieved right now by simply 
changing the set of tempest tests that run against each project.

  -David