[openstack-dev] [Openstack-qa-team] Tempest gate and stable/folsom
David Kranz
david.kranz at qrclab.com
Fri Oct 5 19:43:01 UTC 2012
On 10/5/2012 3:16 PM, Yaniv Kaul wrote:
> On 10/05/2012 07:50 PM, David Kranz wrote:
>> A brief status update:
>>
>> 1. The fixes for the full tempest gate have been merged. Tempest
>> should run without errors except
>> that there is still a flakey volume test
>> https://bugs.launchpad.net/tempest/+bug/1056213.
>> This is not a bug in tempest and we are struggling with what to
>> do about it. Any ideas would be very welcome.
>
> Continue executing, so people will be aware of the issue, and perhaps
> be able to triage to more specific kernel versions / distributions.
> (The fix proposed in
> https://bugs.launchpad.net/cinder/+bug/1023755/comments/22 - to not
> zero out on volume deletion, isn't the greatest from security
> perspective, but that's a different matter).
>
>>
>> 2. There is now a stable/folsom branch for tempest. If you are
>> running tempest against a folsom cluster
>> you should now be using that branch of tempest.
>
> Is there a commit policy to this branch? what about fixes from master?
We will discuss this at the next QA meeting. Obviously it has to track
stable/folsom for the other projects but
whether to backport new tests from master is a matter of resources. For
previous stable branches the
policy was that if interested parties wanted to backport fixes they could.
>
>>
>> 3. After discussion with the ci team, we are going to start gating
>> all core project checkins on full tempest minus flakey tests and
>> "non-gating" tests.
>> I hope this will happen next week.
>
> What's the definition of flaky?
For this purpose, flaky means any test that should always succeed but
fails sometimes in a non-reproducible way.
For example, the volume test mentioned above has been failing perhaps
5-10% of the time. That may be acceptable for
tempest checkins, but not for other projects. Any external gating on
projects has to be rock solid.
>
>>
>> 4. We will be starting a nightly run of full tempest plus
>> "non-gating" tests. We will be adding stress, configuration and
>> performance
>> tests to the non-gating suite in the coming weeks.
>
> Great to hear, I hope we'll be doing the same - lets try to coordinate
> those efforts.
Indeed. There is a session in the QA track at the summit titled
"Bringing OpenStack QA into the Open":
http://summit.openstack.org/cfp/details/81
The exact purpose of this session is to encourage more open development
and execution of tests among the various OpenStack contributors.
I am hoping that people involved in testing from various companies will
attend the session.
-David
More information about the OpenStack-dev
mailing list