Open Stack

On 04/22/2014 08:35 AM, Daniel P. Berrange wrote:
> On Tue, Apr 22, 2014 at 07:23:20AM -0400, Sean Dague wrote:
>> On 04/22/2014 06:55 AM, Daniel P. Berrange wrote:
>>> On Mon, Apr 21, 2014 at 10:52:39PM +0000, Daryl Walleck wrote:
>>>> I nearly opened a spec for this, but I'd really like to get some
>>>> feedback first. One of the challenges I've seen lately for Nova
>>>> teams not using KVM or Xen (Ironic and LXC are just a few) is
>>>> how to properly run the subset of Compute tests that will run for
>>>> their hypervisor or driver. Regexes are what Ironic went with,
>>>> but I'm not sure how well that will work long term since it's
>>>> very much dependent on naming conventions. The good thing is
>>>> that the capabilities for each hypervisor/driver are well defined
>>>> (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so
>>>> it's just a matter of how to convey that information. I see a
>>>> few ways forward from here:
>>>>
>>>> 1.       Expand the compute_features_group config section to include
>>>> all Compute actions and make sure all tests that require specific
>>>> capabilities have skipIfs or raise a skipException. This options seems
>>>> it would require the least work within Tempest, but the size of the
>>>> config will continue to grow as more Nova actions are added.
>>>>
>>>> 2.       Create a new decorator class like was done with service tags
>>>> that defines what drivers the test does or does not work for, and have
>>>> the definitions of the different driver capabilities be referenced by
>>>> the decorator. This is nice because it gets rid of the config creep,
>>>> but it's also yet another decorator, which may not be desirable.
>>>>
>>>> I'm going to continue working through both of these possibilities,
>>>> but any feedback either solution would be appreciated.
>>>
>>> It strikes me that if the test suites have a problem determining
>>> support status of APIs with the currently active driver, then the
>>> applications using OpenStack will likely suffer the same problem.
>>> Given that it'd be desirable to ensure we can solve it in a general
>>> way, rather than only consider the test suite needs.
>>>
>>> On the Nova side of things, I think it would be important to ensure
>>> that there is a single "OperationNotSupported" exception that is
>>> always raised when the API tries to exercise a feature that is not
>>> available with a specific hypervisor driver. If a test case in the
>>> test suite ever receives "OperationNotSupported" it could then just
>>> mark that test case as "skipped" rather than having exception propagate
>>> to result in a "fail".  To me the nice thing about such an approach is
>>> that you do not need to ever maintain a "matrix" of supported features,
>>> as the test suite would just do the right thing whenever the driver is
>>> updated.
>>
>> Agreed. Though I think we probably want the Nova API to be explicit
>> about what parts of the API it's ok to throw a Not Supported. Because I
>> don't think it's a blanket ok. On API endpoints where this is ok, we can
>> convert not supported to a skip.
> 
> Yep, this ties into a discussion I recall elsewhere about specifying
> exactly which parts of the Nova API are considered mandatory features
> for drivers to implement.

I put that in as a proposal to discuss at the design summit.

http://summit.openstack.org/cfp/details/55

-- 
Russell Bryant