[openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

Jordan Pittier jordan.pittier at scality.com
Tue May 2 13:36:20 UTC 2017


On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann <ghanshyammann at gmail.com>
wrote:

> In Cinder, there are many features/APIs which are backend specific and
> will return 405 or 501 if same is not implemented on any backend [1].
> If such tests are implemented in Tempest, then it will break some gate
> where that backend job is voting. like ceph job in glance_store gate.
>
> There been many such cases recently where ceph jobs were broken due to
> such tests and recently it is for force-delete backup feature[2].
> Reverting force-delete tests in [3]. To resolve such cases at some
> extend, Jon is going to add a white/black list of tests which can run
> on ceph job [4] depends on what all feature ceph implemented. But this
> does not resolve it completely due to many reason like
> 1. External use of Tempest become difficult where user needs to know
> what all tests to skip for which backend
> 2. Tempest tests become too specific to backend.
>
> Now there are few options to resolve this:
> 1. Tempest should not tests such API/feature which are backend
> specific like mentioned by api-ref like[1].
>
So basically, if one of the 50 Cinder driver doesn't support a feature, we
should never test that feature ? What about the 49 other drivers ? If a
feature exists and can be tested in the Gate (with whatever default
config/driver is shipped) then I think we should test it.


> 2. Tempest test can be disabled/skip based on backend. - This is not
> good idea as it increase config options and overhead of setting those.
>
Using regex and blacklist, any 3rd party CI can skip any test based on the
test ID. Without introducing a config flag. See:
https://github.com/openstack-infra/project-config/blob/1cea31f402b6b0cccc47cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871


> 3. Tempest test can verify behavior with if else condition as per
> backend. This is bad idea and lose the test strength.
>
Yeah, that's bad.

>
> IMO options 1 is better options. More feedback are welcome.


> ..1 https://developer.openstack.org/api-ref/block-storage/v3/?
> expanded=force-delete-a-backup-detail#force-delete-a-backup
> ..2 https://bugs.launchpad.net/glance/+bug/1687538
> ..3 https://review.openstack.org/#/c/461625/
> ..4 http://lists.openstack.org/pipermail/openstack-dev/2017-
> April/115229.html
>
> -gmann
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170502/582c8b84/attachment.html>


More information about the OpenStack-dev mailing list