Open Stack

On Sun, Jan 19, 2014 at 2:47 PM, Devananda van der Veen <
devananda.vdv at gmail.com> wrote:

> Hi all,
>
> I've been thinking about how we should treat third-party drivers in Ironic
> for a while, and had several discussions at the Hong Kong summit and last
> week at LCA. Cinder, Nova, Neutron, and TripleO are all having similar
> discussions, too. What follows is a summary of my thoughts and a proposal
> for our project's guidelines to vendors.
>

I applaud the effort. I'm actually currently in the process of writing up
instructions for Cinder and Neutron vendors interested in constructing a
3rd party testing platform that uses the openstack-infra tooling as much as
possible. (Yes, I know there is existing documentation on ci.openstack.org,
but based on discussions this past week with the Neutron vendors
implementing these test platforms, there's a number of areas that are
poorly understood and some more detail is clearly needed).

I would hope the docs I'm putting together for Cinder and Neutron will
require little, if any, changes for similar instructions for Ironic 3rd
party testers.


> Before requiring that degree of testing, I would like to be able to direct
> vendors at a working test suite which they can copy. I expect us to have
> functional testing for the PXE and SSH drivers within Tempest and devstack
> / devstack-gate either late in this cycle or early next cycle. Around the
> same time, I believe TripleO will switch to using Ironic in their test
> suite, so we'll have coverage of the IPMI driver on real hardware as well
> (this may be periodic coverage rather than per-test feedback initially).
>

I think using Tempest as that working test suite would be the best way to
go. Cinder 3rd party testing is going in this direction (the cinder_cert/
directory in devstack simply sets up Tempest, sets the appropriate Cinder
driver properly in the cinder.conf and then runs the Tempest Volume API
tests. A similar approach would work for Ironic, I believe, once the Ironic
API tests are complete for Tempest.


> I am proposing that we provisionally allow in vendor drivers this cycle
> with the following requirements, and that we draw the line at the J3
> milestone to give everyone ample time to get testing up on real hardware --
> without blocking innovation now. At that time, we may kick out third-party
> drivers if these criteria haven't been met.
>
> 1. each driver must adhere to the existing driver interfaces.
> 2. each driver must have comprehensive unit test coverage and sufficient
> inline documentation.
> 3. vendors are responsible for fixing bugs in their driver in a timely
> fashion.
> 4. vendors commit to have third-party testing on a supported hardware
> platform implemented by the J3 milestone.
> 5. vendors contribute a portion of at least one developer's time to
> upstream participation.
>

All good things. However, specificity is critical here. What does
"sufficient inline documentation" entail? Who is the arbiter? What does
"comprehensive unit test coverage" mean? 90%? 100%? What does "timely
fashion" mean? Within 2 days? By X milestone?

The more specificity, the less miscommunication will occur.

And, BTW, the above goes for all the driver verification programs currently
being fleshed out... not just Ironic, of course! :)

Best,
-jay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140120/54db74fd/attachment.html>