Open Stack

On Wed, May 3, 2017 at 21:57 Andrea Frittoli <andrea.frittoli at gmail.com>
wrote:

> On Tue, May 2, 2017 at 2:41 PM Jordan Pittier <jordan.pittier at scality.com>
> wrote:
>
>> On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann <ghanshyammann at gmail.com>
>> wrote:
>>
>>> In Cinder, there are many features/APIs which are backend specific and
>>> will return 405 or 501 if same is not implemented on any backend [1].
>>> If such tests are implemented in Tempest, then it will break some gate
>>> where that backend job is voting. like ceph job in glance_store gate.
>>>
>>> There been many such cases recently where ceph jobs were broken due to
>>> such tests and recently it is for force-delete backup feature[2].
>>> Reverting force-delete tests in [3]. To resolve such cases at some
>>> extend, Jon is going to add a white/black list of tests which can run
>>> on ceph job [4] depends on what all feature ceph implemented. But this
>>> does not resolve it completely due to many reason like
>>> 1. External use of Tempest become difficult where user needs to know
>>> what all tests to skip for which backend
>>> 2. Tempest tests become too specific to backend.
>>>
>>> Now there are few options to resolve this:
>>> 1. Tempest should not tests such API/feature which are backend
>>> specific like mentioned by api-ref like[1].
>>>
>> So basically, if one of the 50 Cinder driver doesn't support a feature,
>> we should never test that feature ? What about the 49 other drivers ? If a
>> feature exists and can be tested in the Gate (with whatever default
>> config/driver is shipped) then I think we should test it.
>>
>>
>>> 2. Tempest test can be disabled/skip based on backend. - This is not
>>> good idea as it increase config options and overhead of setting those.
>>>
>> Using regex and blacklist, any 3rd party CI can skip any test based on
>> the test ID. Without introducing a config flag. See:
>> https://github.com/openstack-infra/project-config/blob/1cea31f402b6b0cccc47cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871
>>
>
> This way each 3rd party system has to maintain its own list, which has the
> advantage that
> different teams maintain their own list (which is nice from an ownership
> and scale pov).
>
> However I think such list of tests are not as re-usable as having a
> devstack plugin (or an
> ansible or puppet module) changing a few tempest config options.
>

Humm, I am little bit hesitate to go that way. For gate and CI it might be
good solution but for production cloud testing it makes Tenpest difficult
to use.

If I use Tempest to test my cloud, few tests going to fail as those were
not supported by cinder driver my cloud has or going to have.
I do not have any way to configure something so that test can be disabled.
Instead I need to maintain list of tests to run or skip. And that list is
not static, it grows dynamically.
This makes Tempest difficult to use.




>
>>
>>> 3. Tempest test can verify behavior with if else condition as per
>>> backend. This is bad idea and lose the test strength.
>>>
>> Yeah, that's bad.
>>
>>>
>>> IMO options 1 is better options. More feedback are welcome.
>>
>>
>>> ..1
>>> https://developer.openstack.org/api-ref/block-storage/v3/?expanded=force-delete-a-backup-detail#force-delete-a-backup
>>> ..2 https://bugs.launchpad.net/glance/+bug/1687538
>>> ..3 https://review.openstack.org/#/c/461625/
>>> ..4
>>> http://lists.openstack.org/pipermail/openstack-dev/2017-April/115229.html
>>>
>>> -gmann
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 
-gmann
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170504/661a29a1/attachment.html>