[openstack-dev] [Ironic] Third-party drivers and testing

Devananda van der Veen devananda.vdv at gmail.com
Sun Jan 19 19:47:14 UTC 2014

Hi all,

I've been thinking about how we should treat third-party drivers in Ironic
for a while, and had several discussions at the Hong Kong summit and last
week at LCA. Cinder, Nova, Neutron, and TripleO are all having similar
discussions, too. What follows is a summary of my thoughts and a proposal
for our project's guidelines to vendors.

At Ironic's core is a set of reference drivers that use PXE and IPMI, which
are broadly supported by most hardware, to provide the minimum necessary
set of features for provisioning hardware. Vendors naturally want to
improve upon those features, whether by adding new features or making their
hardware support more robust or better performing. This is good and worth
encouraging. However, I do not want Ironic to get get into a position where
third-party drivers are in trunk but not tested, and thus potentially of a
lower quality code, inadvertently broken by changes to the common code, and
a burden upon the core contributors. I want to encourage vendors'
contributions while guarding the shared codebase to ensure that it works
well for everyone, and without increasing the burden unfairly on core
contributors and reviewers. We're in the unique position at the moment, as
a new project, of not having any vendor drivers in trunk today; we have the
opportunity to set the bar without having to do a lot of clean up first.

To this end, I believe we need third-party functional testing on real
hardware for each third-party driver, and I believe we need this for every
commit to Ironic. This is the bar that we're going to hold the PXE driver
to, and I would like our users to have the same level of confidence in
third-party drivers.

Before requiring that degree of testing, I would like to be able to direct
vendors at a working test suite which they can copy. I expect us to have
functional testing for the PXE and SSH drivers within Tempest and devstack
/ devstack-gate either late in this cycle or early next cycle. Around the
same time, I believe TripleO will switch to using Ironic in their test
suite, so we'll have coverage of the IPMI driver on real hardware as well
(this may be periodic coverage rather than per-test feedback initially).

With that timeline in mind, it would not be fair to require vendors to
complete their own third-party testing this cycle with no example to work
from. On the other hand, I don't want to block the various vendors who have
already expressed a strong interest in contributing their drivers to the
Icehouse release.

I am proposing that we provisionally allow in vendor drivers this cycle
with the following requirements, and that we draw the line at the J3
milestone to give everyone ample time to get testing up on real hardware --
without blocking innovation now. At that time, we may kick out third-party
drivers if these criteria haven't been met.

1. each driver must adhere to the existing driver interfaces.
2. each driver must have comprehensive unit test coverage and sufficient
inline documentation.
3. vendors are responsible for fixing bugs in their driver in a timely
4. vendors commit to have third-party testing on a supported hardware
platform implemented by the J3 milestone.
5. vendors contribute a portion of at least one developer's time to
upstream participation.

Items 1 and 2 are criteria for any code landing in trunk; these are not
specific to this discussion, but I feel they're worth calling out

Items 3 and 4 are crucial to maintaining code quality in the third-party
drivers. See http://ci.openstack.org/third_party.html for a general
description of third-party testing. My goal is that by J3, we should have
smoke testing (vote +1/-1 but not gate blocking) for every commit to Ironic
on supported hardware for each third-party driver.

Item 5 is meant to avoid what I call the "throwing code over the wall"
problem. I believe this will ensure that there is ongoing vendor
participation in the Ironic developer community. To clarify, broadly
speaking, I mean that this person should:
- be a developer from the internal team responsible for the contributed
- subscribe to the openstack-dev mailing list and pay attention to the
[Ironic] discussions,
- attend and participate in the weekly IRC meeting,
- participate in Ironic code reviews (1 per day would be sufficient),
- occasionally make code contributions that are not directly related to the
vendor driver (eg, help with general bug fixes),
- and be present in #openstack-ironic IRC channel on Freenode during their
working hours so that other developers can reach them easily when necessary.

This doesn't have to consume much of a developer's time, roughly 5 - 10
hrs/week should be enough.

I recognize that the costs to do this (both human and hardware) are
non-trivial, but I believe they are not unreasonable and are crucial to
ensuring a consistent quality across all the drivers in Ironic.

Inviting comments and feedback...

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140119/4174a02a/attachment.html>

More information about the OpenStack-dev mailing list