[openstack-dev] [qa] Selecting Compute tests by driver/hypervisor

Russell Bryant rbryant at redhat.com
Tue Apr 22 12:33:03 UTC 2014


On 04/22/2014 07:23 AM, Sean Dague wrote:
> On 04/22/2014 06:55 AM, Daniel P. Berrange wrote:
>> On Mon, Apr 21, 2014 at 10:52:39PM +0000, Daryl Walleck wrote:
>>> I nearly opened a spec for this, but I'd really like to get
>>> some feedback first. One of the challenges I've seen lately for
>>> Nova teams not using KVM or Xen (Ironic and LXC are just a few)
>>> is how to properly run the subset of Compute tests that will
>>> run for their hypervisor or driver. Regexes are what Ironic
>>> went with, but I'm not sure how well that will work long term
>>> since it's very much dependent on naming conventions. The good
>>> thing is that the capabilities for each hypervisor/driver are
>>> well defined 
>>> (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so 
>>> it's just a matter of how to convey that information. I see a 
>>> few ways forward from here:
>>> 
>>> 1.       Expand the compute_features_group config section to
>>> include all Compute actions and make sure all tests that
>>> require specific capabilities have skipIfs or raise a
>>> skipException. This options seems it would require the least
>>> work within Tempest, but the size of the config will continue
>>> to grow as more Nova actions are added.
>>> 
>>> 2.       Create a new decorator class like was done with
>>> service tags that defines what drivers the test does or does
>>> not work for, and have the definitions of the different driver
>>> capabilities be referenced by the decorator. This is nice
>>> because it gets rid of the config creep, but it's also yet
>>> another decorator, which may not be desirable.
>>> 
>>> I'm going to continue working through both of these
>>> possibilities, but any feedback either solution would be
>>> appreciated.
>> 
>> It strikes me that if the test suites have a problem determining 
>> support status of APIs with the currently active driver, then
>> the applications using OpenStack will likely suffer the same
>> problem. Given that it'd be desirable to ensure we can solve it
>> in a general way, rather than only consider the test suite
>> needs.
>> 
>> On the Nova side of things, I think it would be important to
>> ensure that there is a single "OperationNotSupported" exception
>> that is always raised when the API tries to exercise a feature
>> that is not available with a specific hypervisor driver. If a
>> test case in the test suite ever receives "OperationNotSupported"
>> it could then just mark that test case as "skipped" rather than
>> having exception propagate to result in a "fail".  To me the nice
>> thing about such an approach is that you do not need to ever
>> maintain a "matrix" of supported features, as the test suite
>> would just do the right thing whenever the driver is updated.
> 
> Agreed. Though I think we probably want the Nova API to be
> explicit about what parts of the API it's ok to throw a Not
> Supported. Because I don't think it's a blanket ok. On API
> endpoints where this is ok, we can convert not supported to a
> skip.

Definitely agreed with Dan's points here.

We already raise NotImplementedError for this in the code.  Assuming
it makes it all the way back up to the API, it should be converted to
a 501 Not Implemented response.

It doesn't look like this is handled in a general way in the API code.
 Each API extension that supports this is handling it manually.
However, it is used a lot.  A quick grep in nova/api shows 50 cases of
raising webob.exc.HTTPNotImplemented.

It would be nice to identify the specific cases where this isn't
working as expected.

-- 
Russell Bryant



More information about the OpenStack-dev mailing list