Open Stack

On 04/22/2014 06:55 AM, Daniel P. Berrange wrote:
> On Mon, Apr 21, 2014 at 10:52:39PM +0000, Daryl Walleck wrote:
>> I nearly opened a spec for this, but I'd really like to get some
>> feedback first. One of the challenges I've seen lately for Nova
>> teams not using KVM or Xen (Ironic and LXC are just a few) is
>> how to properly run the subset of Compute tests that will run for
>> their hypervisor or driver. Regexes are what Ironic went with,
>> but I'm not sure how well that will work long term since it's
>> very much dependent on naming conventions. The good thing is
>> that the capabilities for each hypervisor/driver are well defined
>> (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so
>> it's just a matter of how to convey that information. I see a
>> few ways forward from here:
>>
>> 1.       Expand the compute_features_group config section to include
>> all Compute actions and make sure all tests that require specific
>> capabilities have skipIfs or raise a skipException. This options seems
>> it would require the least work within Tempest, but the size of the
>> config will continue to grow as more Nova actions are added.
>>
>> 2.       Create a new decorator class like was done with service tags
>> that defines what drivers the test does or does not work for, and have
>> the definitions of the different driver capabilities be referenced by
>> the decorator. This is nice because it gets rid of the config creep,
>> but it's also yet another decorator, which may not be desirable.
>>
>> I'm going to continue working through both of these possibilities,
>> but any feedback either solution would be appreciated.
> 
> It strikes me that if the test suites have a problem determining
> support status of APIs with the currently active driver, then the
> applications using OpenStack will likely suffer the same problem.
> Given that it'd be desirable to ensure we can solve it in a general
> way, rather than only consider the test suite needs.
> 
> On the Nova side of things, I think it would be important to ensure
> that there is a single "OperationNotSupported" exception that is
> always raised when the API tries to exercise a feature that is not
> available with a specific hypervisor driver. If a test case in the
> test suite ever receives "OperationNotSupported" it could then just
> mark that test case as "skipped" rather than having exception propagate
> to result in a "fail".  To me the nice thing about such an approach is
> that you do not need to ever maintain a "matrix" of supported features,
> as the test suite would just do the right thing whenever the driver is
> updated.

Agreed. Though I think we probably want the Nova API to be explicit
about what parts of the API it's ok to throw a Not Supported. Because I
don't think it's a blanket ok. On API endpoints where this is ok, we can
convert not supported to a skip.

	-Sean

-- 
Sean Dague
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140422/da2bf79d/attachment.pgp>