Open Stack

On Mon, Apr 21, 2014 at 10:52:39PM +0000, Daryl Walleck wrote:
> I nearly opened a spec for this, but I'd really like to get some
> feedback first. One of the challenges I've seen lately for Nova
> teams not using KVM or Xen (Ironic and LXC are just a few) is
> how to properly run the subset of Compute tests that will run for
> their hypervisor or driver. Regexes are what Ironic went with,
> but I'm not sure how well that will work long term since it's
> very much dependent on naming conventions. The good thing is
> that the capabilities for each hypervisor/driver are well defined
> (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so
> it's just a matter of how to convey that information. I see a
> few ways forward from here:
> 
> 1.       Expand the compute_features_group config section to include
> all Compute actions and make sure all tests that require specific
> capabilities have skipIfs or raise a skipException. This options seems
> it would require the least work within Tempest, but the size of the
> config will continue to grow as more Nova actions are added.
> 
> 2.       Create a new decorator class like was done with service tags
> that defines what drivers the test does or does not work for, and have
> the definitions of the different driver capabilities be referenced by
> the decorator. This is nice because it gets rid of the config creep,
> but it's also yet another decorator, which may not be desirable.
> 
> I'm going to continue working through both of these possibilities,
> but any feedback either solution would be appreciated.

It strikes me that if the test suites have a problem determining
support status of APIs with the currently active driver, then the
applications using OpenStack will likely suffer the same problem.
Given that it'd be desirable to ensure we can solve it in a general
way, rather than only consider the test suite needs.

On the Nova side of things, I think it would be important to ensure
that there is a single "OperationNotSupported" exception that is
always raised when the API tries to exercise a feature that is not
available with a specific hypervisor driver. If a test case in the
test suite ever receives "OperationNotSupported" it could then just
mark that test case as "skipped" rather than having exception propagate
to result in a "fail".  To me the nice thing about such an approach is
that you do not need to ever maintain a "matrix" of supported features,
as the test suite would just do the right thing whenever the driver is
updated.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|