Open Stack

On 04/05/17 15:01, Ghanshyam Mann wrote:
>
> On Wed, May 3, 2017 at 21:57 Andrea Frittoli
> <andrea.frittoli at gmail.com <mailto:andrea.frittoli at gmail.com>> wrote:
>
>     On Tue, May 2, 2017 at 2:41 PM Jordan Pittier
>     <jordan.pittier at scality.com <mailto:jordan.pittier at scality.com>>
>     wrote:
>
>         On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann
>         <ghanshyammann at gmail.com <mailto:ghanshyammann at gmail.com>> wrote:
>
>             In Cinder, there are many features/APIs which are backend
>             specific and
>             will return 405 or 501 if same is not implemented on any
>             backend [1].
>             If such tests are implemented in Tempest, then it will
>             break some gate
>             where that backend job is voting. like ceph job in
>             glance_store gate.
>
>             There been many such cases recently where ceph jobs were
>             broken due to
>             such tests and recently it is for force-delete backup
>             feature[2].
>             Reverting force-delete tests in [3]. To resolve such cases
>             at some
>             extend, Jon is going to add a white/black list of tests
>             which can run
>             on ceph job [4] depends on what all feature ceph
>             implemented. But this
>             does not resolve it completely due to many reason like
>             1. External use of Tempest become difficult where user
>             needs to know
>             what all tests to skip for which backend
>             2. Tempest tests become too specific to backend.
>
>             Now there are few options to resolve this:
>             1. Tempest should not tests such API/feature which are backend
>             specific like mentioned by api-ref like[1].
>
>         So basically, if one of the 50 Cinder driver doesn't support a
>         feature, we should never test that feature ? What about the 49
>         other drivers ? If a feature exists and can be tested in the
>         Gate (with whatever default config/driver is shipped) then I
>         think we should test it.
>          
>
>             2. Tempest test can be disabled/skip based on backend. -
>             This is not
>             good idea as it increase config options and overhead of
>             setting those.
>
>         Using regex and blacklist, any 3rd party CI can skip any test
>         based on the test ID. Without introducing a config flag.
>         See: https://github.com/openstack-infra/project-config/blob/1cea31f402b6b0cccc47cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871
>
>
>     This way each 3rd party system has to maintain its own list, which
>     has the advantage that
>     different teams maintain their own list (which is nice from an
>     ownership and scale pov).
>
>     However I think such list of tests are not as re-usable as having
>     a devstack plugin (or an
>     ansible or puppet module) changing a few tempest config options. 
>
>
> Humm, I am little bit hesitate to go that way. For gate and CI it
> might be good solution but for production cloud testing it makes
> Tenpest difficult to use.
>
> If I use Tempest to test my cloud, few tests going to fail as those
> were not supported by cinder driver my cloud has or going to have. 
> I do not have any way to configure something so that test can be
> disabled. Instead I need to maintain list of tests to run or skip. And
> that list is not static, it grows dynamically. 
> This makes Tempest difficult to use. 
>
>

Agree. We (Catalyst Cloud based in NZ) are using Tempest as the
monitoring tool and CI/CD gate for our cloud. But we do have to maintain
a white list of test cases because there are some cases are not fitting
our cloud.

>
>
>          
>
>             3. Tempest test can verify behavior with if else condition
>             as per
>             backend. This is bad idea and lose the test strength.
>
>         Yeah, that's bad. 
>
>
>             IMO options 1 is better options. More feedback are welcome. 
>
>
>             ..1
>             https://developer.openstack.org/api-ref/block-storage/v3/?expanded=force-delete-a-backup-detail#force-delete-a-backup
>             ..2 https://bugs.launchpad.net/glance/+bug/1687538
>             ..3 https://review.openstack.org/#/c/461625/
>             ..4
>             http://lists.openstack.org/pipermail/openstack-dev/2017-April/115229.html
>
>             -gmann
>
>             __________________________________________________________________________
>             OpenStack Development Mailing List (not for usage questions)
>             Unsubscribe:
>             OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>             <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>             http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>         __________________________________________________________________________
>         OpenStack Development Mailing List (not for usage questions)
>         Unsubscribe:
>         OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>         <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> -- 
> -gmann
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
Cheers & Best regards,
Feilong Wang (王飞龙)
--------------------------------------------------------------------------
Senior Cloud Software Engineer
Tel: +64-48032246
Email: flwang at catalyst.net.nz
Catalyst IT Limited
Level 6, Catalyst House, 150 Willis Street, Wellington
-------------------------------------------------------------------------- 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170504/758a9760/attachment.html>