Open Stack

Hi Adam,

I think the steps you've completed so far are the foundation for testers like myself to be able to really dive in and start kicking the tires of Ironic. Having an easily repeatable way to generate test environments makes a huge impact in where I spend my time (environments vs actual testing). As for functional coverage, to be completely honest I'm still working on fully context switching from Nova, but I've looked at your etherpad and I definitely have some thoughts and additions to share.

I also wouldn't rule the idea completely of communicating with other management resources. While this used to be an idea I strongly and loudly opposed, one of the realizations I had during my time testing Nova was that ignoring the backend was somewhat of a mistake, at least from a root cause analysis of failures perspective. While working with Nova and XenServer, I ran into enough situations where XenServer had a much clearer perspective of faults and failures that I began to add optional duplicate validations to the XenServer API to help better track where the real source of a failure was. While I'm still not sure if this would make sense to implement in any Ironic tests, I don't think completely closing the door on the idea would be wise yet.

As for dealing with narrowing the scope of Tempest tests by Nova driver, I think there's a few easy options here. What I've done in my own testing outside of Tempest is to use and expand on the Hypervisor/Driver Feature Matrix [1] in combination with a driver configuration value to determine at run time which tests should be executed. When I first started developing Tempest, I implemented a much weaker solution, which was to provide a per feature flag to enable certain tests that use individual features, which still exists today. The concept of feature sets isn't a problem limited to Ironic (other hypervisors and drivers and similar problems), so I'd like to reengage the Tempest community with this concept and see how it's received.

Daryl

[1] https://wiki.openstack.org/wiki/HypervisorSupportMatrix#Hypervisor_feature_support_matrix

On Mar 20, 2014, at 9:18 PM, "Adam Gandelman" <adamg at ubuntu.com<mailto:adamg at ubuntu.com>> wrote:

Hi--

We've made some progress over the last week or so toward more thorough Ironic CI:

* Initial devstack support has been merged enabling an all-in-one Ironic environment that uses libvirt+VMs+OVS to support baremetal deployments [1]

* Devananda and myself have been getting patches pushed into infra. that add an experimental devstack gate check using the new support

While we continue to work out the kinks in the devstack gate, I've started thinking about what functional Ironic tests in Tempest would look like.  I've been capturing my thoughts in an etherpad [2] and invite anyone interested to add theirs.

I've put together a Tempest scenario test that validates instance spawn and corresponding Ironic orchestration  [3].  This initial test assumes its being tested against the pxe_ssh driver (which devstack enrolls nodes with) and verifies assumptions accordingly.  A longer term goal would be for this same test to be run against other non-devstack environments (ie, TripleO undercloud) and verify other things specific to the drivers in use there.

I'm curious to know what others think should be, or even can be, stressed and tested from the outside by Tempest.  Since Tempest cannot assume it has access to poke at management resources like libvirt, IPMI, etc. it can really only inspect what is provided by the APIs.  Validating that nova/neutron/etc injects the correct data into Ironic node/port/etc objects seems like the most that can happen beyond simply spawning an instance and waiting for it to show up on the network.

On a related note, running Tempest in the gate against Ironic presents some interesting challenges that we'll need to work with the Tempest team to solve.  The API and smoke tests that are run assume many things about the supported features of the running compute driver. Many of these are not supported in Ironic (eg,, boot from volume).  This not specific to Ironic (eg, lxc) but will require some discussion and work before Tempest is just point-and-shoot against Ironic.   I'm wondering if it would make sense to lean on the soon-to-be-deprecated devstack exercises for verification in the short-term, while we work through the larger issues in Tempest.

-Adam


[1] https://review.openstack.org/#/c/70348/
[2] https://etherpad.openstack.org/p/IronicCI
[2] https://review.openstack.org/#/c/81958/
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140323/a0db7f16/attachment.html>