Open Stack

In Cinder, there are many features/APIs which are backend specific and
will return 405 or 501 if same is not implemented on any backend [1].
If such tests are implemented in Tempest, then it will break some gate
where that backend job is voting. like ceph job in glance_store gate.

There been many such cases recently where ceph jobs were broken due to
such tests and recently it is for force-delete backup feature[2].
Reverting force-delete tests in [3]. To resolve such cases at some
extend, Jon is going to add a white/black list of tests which can run
on ceph job [4] depends on what all feature ceph implemented. But this
does not resolve it completely due to many reason like
1. External use of Tempest become difficult where user needs to know
what all tests to skip for which backend
2. Tempest tests become too specific to backend.

Now there are few options to resolve this:
1. Tempest should not tests such API/feature which are backend
specific like mentioned by api-ref like[1].
2. Tempest test can be disabled/skip based on backend. - This is not
good idea as it increase config options and overhead of setting those.
3. Tempest test can verify behavior with if else condition as per
backend. This is bad idea and lose the test strength.

IMO options 1 is better options. More feedback are welcome.

..1 https://developer.openstack.org/api-ref/block-storage/v3/?expanded=force-delete-a-backup-detail#force-delete-a-backup
..2 https://bugs.launchpad.net/glance/+bug/1687538
..3 https://review.openstack.org/#/c/461625/
..4 http://lists.openstack.org/pipermail/openstack-dev/2017-April/115229.html

-gmann