Open Stack

On Wed, May 03, 2017 at 11:09:32PM -0400, Monty Taylor wrote:
> On 05/02/2017 11:49 AM, Sean McGinnis wrote:
> > On Tue, May 02, 2017 at 03:36:20PM +0200, Jordan Pittier wrote:
> > > On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann <ghanshyammann at gmail.com>
> > > wrote:
> > > 
> > > > In Cinder, there are many features/APIs which are backend specific and
> > > > will return 405 or 501 if same is not implemented on any backend [1].
> > > > If such tests are implemented in Tempest, then it will break some gate
> > > > where that backend job is voting. like ceph job in glance_store gate.
> > > > 
> > > > There been many such cases recently where ceph jobs were broken due to
> > > > such tests and recently it is for force-delete backup feature[2].
> > > > Reverting force-delete tests in [3]. To resolve such cases at some
> > > > extend, Jon is going to add a white/black list of tests which can run
> > > > on ceph job [4] depends on what all feature ceph implemented. But this
> > > > does not resolve it completely due to many reason like
> > > > 1. External use of Tempest become difficult where user needs to know
> > > > what all tests to skip for which backend
> > > > 2. Tempest tests become too specific to backend.
> > > > 
> > > > Now there are few options to resolve this:
> > > > 1. Tempest should not tests such API/feature which are backend
> > > > specific like mentioned by api-ref like[1].
> > > > 
> > > So basically, if one of the 50 Cinder driver doesn't support a feature, we
> > > should never test that feature ? What about the 49 other drivers ? If a
> > > feature exists and can be tested in the Gate (with whatever default
> > > config/driver is shipped) then I think we should test it.
> > > 
> > 50? Over 100 as of Ocata.
> > 
> > Well, is tempest's purpose in life to provide complete gate test coverage,
> > or is tempest's purpose in life to give operators a tool to validate that
> > their deployment is working as expected?
> 
> I'd actually like to suggest that such a scenario actually points out a
> thing that is ultimately potential pain passed to the end user in the real
> world, so this question about what/how to test this in tempest is a good one
> to have.
> 
> If there is a feature which is only provisionally available depending on the
> backend driver such that it's hard to test in tempest without an out of band
> config - then it's a feature that a user will have no clue whether it works
> on a given cloud.
> 
> As we find these, I'd love it if we could expose discovery in the API for
> viability of the feature. Like:
> 
> GET /capabilities
> 
> {
>   "capabilities": {
>     "has_force_delete": true
>   }
> }
> 
> (I know we've talked about that concept generally, but this is a specific
> example)
> 
> If such a thing existed, then the user can know whether they can use a thing
> .. and so can tempest. A tempest test to validate force_delete working could
> check the capability reported by the API and validate that if the API says
> "true" that the feature work as expected, and if it says "false" validate
> that attempting to call it returns a 405 (or whatever is appropriate)
> 
> Ultimately, every config we need to feed to tempest is potentially a place
> where an end user is unable to know whether or not to expect a call to work
> - and an opportunity for us to provide our API consumers with a richer
> experience.

Heh, well I've been saying all of these things for years. In fact I got so tired
of repeating it all the time I just put it in the tempest configuration guide:
(although I remember it being a lot snarkier)

https://docs.openstack.org/developer/tempest/configuration.html#service-feature-configuration

So, I'll be very happy when the capabilities work actually becomes a thing you
can use. But, it feels like we've been talking about around the problem for a
very long time...

-Matt Treinish


> 
> > In attempting to do things in the past, I've received push back based on
> > the argument that it was the latter. For this reason, in-tree tempest tests
> > were added to Cinder to give us a way to get better test coverage for our
> > own sake.
> > 
> > Now that this is all in place, I think it's working well and I would like
> > to see it continue that way. IMO, tempest proper should not have anything
> > that isn't universally applicable to real world deployments. Not just for
> > things like Ceph, but things like the manage/unmanage backend specific
> > tests that were added and broke a large majority of third party CI.
> > 
> > Backend specific things should not be part of tempest in my opinion. We
> > should cover those things through in-tree tempest plugins and our own
> > testing.
> > > 
> > > > 2. Tempest test can be disabled/skip based on backend. - This is not
> > > > good idea as it increase config options and overhead of setting those.
> > > > 
> > > Using regex and blacklist, any 3rd party CI can skip any test based on the
> > > test ID. Without introducing a config flag. See:
> > > https://github.com/openstack-infra/project-config/blob/1cea31f402b6b0cccc47cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871
> > > 
> > > 
> > > > 3. Tempest test can verify behavior with if else condition as per
> > > > backend. This is bad idea and lose the test strength.
> > > > 
> > > Yeah, that's bad.
> > > 
> > > > 
> > > > IMO options 1 is better options. More feedback are welcome.
> > > 
> > > 
> > > > ..1 https://developer.openstack.org/api-ref/block-storage/v3/?
> > > > expanded=force-delete-a-backup-detail#force-delete-a-backup
> > > > ..2 https://bugs.launchpad.net/glance/+bug/1687538
> > > > ..3 https://review.openstack.org/#/c/461625/
> > > > ..4 http://lists.openstack.org/pipermail/openstack-dev/2017-
> > > > April/115229.html
> > > > 
> > > > -gmann
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170504/ed00e0a0/attachment.sig>