Open Stack

On 6/16/2017 9:46 AM, Eric Harney wrote:
> On 06/16/2017 10:21 AM, Sean McGinnis wrote:
>>
>> I don't think merging tests that are showing failures, then blacklisting
>> them, is the right approach. And as Eric points out, this isn't
>> necessarily just a failure with Ceph. There is a legitimate logical
>> issue with what this particular test is doing.
>>
>> But in general, to get back to some of the earlier points, I don't think
>> we should be merging tests with known breakages until those breakages
>> can be first addressed.
>>
> 
> As another example, this was the last round of this, in May:
> 
> https://review.openstack.org/#/c/332670/
> 
> which is a new tempest test for a Cinder API that is not supported by
> all drivers.  The Ceph job failed on the tempest patch, correctly, the
> test was merged, then the Ceph jobs broke:
> 
> https://bugs.launchpad.net/glance/+bug/1687538
> https://review.openstack.org/#/c/461625/
> 
> This is really not a sustainable model.
> 
> And this is the _easy_ case, since Ceph jobs run in OpenStack infra and
> are easily visible and trackable.  I'm not sure what the impact is on
> Cinder third-party CI for other drivers.
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

This is generally why we have config options in Tempest to not run tests 
that certain backends don't implement, like all of the backup/snapshot 
volume tests that the NFS job was failing on forever.

I think it's perfectly valid to have tests in Tempest for things that 
not all backends implement as long as they are configurable. It's up to 
the various CI jobs to configure Tempest properly for what they support 
and then work on reducing the number of things they don't support. We've 
been doing that for ages now.

-- 

Thanks,

Matt