Open Stack

On 10/5/2012 3:16 PM, Yaniv Kaul wrote:
> On 10/05/2012 07:50 PM, David Kranz wrote:
>> A brief status update:
>>
>> 1. The fixes for the full tempest gate have been merged. Tempest 
>> should run without errors except
>>     that there is still a flakey volume test 
>> https://bugs.launchpad.net/tempest/+bug/1056213.
>>     This is not a bug in tempest and we are struggling with what to 
>> do about it.  Any ideas would be very welcome.
>
> Continue executing, so people will be aware of the issue, and perhaps 
> be able to triage to more specific kernel versions / distributions.
> (The fix proposed in 
> https://bugs.launchpad.net/cinder/+bug/1023755/comments/22 - to not 
> zero out on volume deletion, isn't the greatest from security 
> perspective, but that's a different matter).
>
>>
>> 2. There is now a stable/folsom branch for tempest. If you are 
>> running tempest against a folsom cluster
>>     you should now be using that branch of tempest.
>
> Is there a commit policy to this branch? what about fixes from master?

We will discuss this at the next QA meeting. Obviously it has to track 
stable/folsom for the other projects but
whether to backport new tests from master is a matter of resources. For 
previous stable branches the
policy was that if interested parties wanted to backport fixes they could.

>
>>
>> 3. After discussion with the ci team, we are going to start gating 
>> all core project checkins on full tempest minus flakey tests and 
>> "non-gating" tests.
>>     I hope this will happen next week.
>
> What's the definition of flaky?

For this purpose, flaky means any test that should always succeed but 
fails sometimes in a non-reproducible way.
For example, the volume test mentioned above has been failing perhaps 
5-10% of the time. That may be acceptable for
tempest checkins, but not for other projects. Any external gating on 
projects has to be rock solid.

>
>>
>> 4. We will be starting a nightly run of full tempest plus 
>> "non-gating" tests. We will be adding stress, configuration and 
>> performance
>>     tests to the non-gating suite in the coming weeks.
>
> Great to hear, I hope we'll be doing the same - lets try to coordinate 
> those efforts.

Indeed. There is a session in the QA track at the summit titled 
"Bringing OpenStack QA into the Open": 
http://summit.openstack.org/cfp/details/81
The exact purpose of this session is to encourage more open development 
and execution of tests among the various OpenStack contributors.
I am hoping that people involved in testing from various companies will 
attend the session.

  -David