Open Stack

On Mon, Jan 20, 2014 at 8:29 AM, Jay Pipes <jaypipes at gmail.com> wrote:

> On Sun, Jan 19, 2014 at 2:47 PM, Devananda van der Veen <
> devananda.vdv at gmail.com> wrote:
>
>> Hi all,
>>
>> I've been thinking about how we should treat third-party drivers in
>> Ironic for a while, and had several discussions at the Hong Kong summit and
>> last week at LCA. Cinder, Nova, Neutron, and TripleO are all having similar
>> discussions, too. What follows is a summary of my thoughts and a proposal
>> for our project's guidelines to vendors.
>>
>
> I applaud the effort. I'm actually currently in the process of writing up
> instructions for Cinder and Neutron vendors interested in constructing a
> 3rd party testing platform that uses the openstack-infra tooling as much as
> possible. (Yes, I know there is existing documentation on ci.openstack.org,
> but based on discussions this past week with the Neutron vendors
> implementing these test platforms, there's a number of areas that are
> poorly understood and some more detail is clearly needed).
>
> I would hope the docs I'm putting together for Cinder and Neutron will
> require little, if any, changes for similar instructions for Ironic 3rd
> party testers.
>

Awesome! I would hope so, too. I would love to be included in that process
(eg, tag me on the review if there is one, or what ever).

The Infra team is also working on some broad guidelines too:
https://review.openstack.org/#/c/63478/



>
>
>> Before requiring that degree of testing, I would like to be able to
>> direct vendors at a working test suite which they can copy. I expect us to
>> have functional testing for the PXE and SSH drivers within Tempest and
>> devstack / devstack-gate either late in this cycle or early next cycle.
>> Around the same time, I believe TripleO will switch to using Ironic in
>> their test suite, so we'll have coverage of the IPMI driver on real
>> hardware as well (this may be periodic coverage rather than per-test
>> feedback initially).
>>
>
> I think using Tempest as that working test suite would be the best way to
> go. Cinder 3rd party testing is going in this direction (the cinder_cert/
> directory in devstack simply sets up Tempest, sets the appropriate Cinder
> driver properly in the cinder.conf and then runs the Tempest Volume API
> tests. A similar approach would work for Ironic, I believe, once the Ironic
> API tests are complete for Tempest.
>

++

Right now, we've got Ironic API tests in Tempest for CRUD operations using
our "fake" driver. These will get extended to support the pxe_ssh driver,
with devstack spinning up some small VMs and feeding their credentials to
tempest, so we can mock bare metal within devstack-gate and our local test
environments. Tempest will then perform functional tests on the driver
interfaces (eg, power on/off, deploy/undeploy, etc).

Using the same approach, but on real hardware, 3rd party testers could feed
in a .csv of their hardware credentials to tempest and do the same tests.
At least, that's my plan :)


>
>> I am proposing that we provisionally allow in vendor drivers this cycle
>> with the following requirements, and that we draw the line at the J3
>> milestone to give everyone ample time to get testing up on real hardware --
>> without blocking innovation now. At that time, we may kick out third-party
>> drivers if these criteria haven't been met.
>>
>> 1. each driver must adhere to the existing driver interfaces.
>> 2. each driver must have comprehensive unit test coverage and sufficient
>> inline documentation.
>> 3. vendors are responsible for fixing bugs in their driver in a timely
>> fashion.
>> 4. vendors commit to have third-party testing on a supported hardware
>> platform implemented by the J3 milestone.
>> 5. vendors contribute a portion of at least one developer's time to
>> upstream participation.
>>
>
> All good things. However, specificity is critical here. What does
> "sufficient inline documentation" entail? Who is the arbiter? What does
> "comprehensive unit test coverage" mean? 90%? 100%? What does "timely
> fashion" mean? Within 2 days? By X milestone?
>

Point 2 should be getting enforced by the existing Ironic-core review team
when the driver code is submitted. It's ultimately up to our reviewers to
determine whether the inline documentation is, at a minimum, up to the
standard that we hold our own code to, and the same is true for the unit
tests. I think we're doing a pretty good job as a team of that today, and
my point in calling it out here is to let vendors know the expectation --
they don't just get to submit a driver and have it land. It will get the
same scrutiny as all other code in the project.

This is also why I want vendors to contribute a reviewer - their driver
increases our review load, so I feel they should help review other patches,
too. Though said person doesn't have to be[come] a core reviewer, that'd be
great if it happens, and I suspect vendors will encourage their employees
in that direction if they know that it's possible and beneficial to the
project.


>
> The more specificity, the less miscommunication will occur.
>
> And, BTW, the above goes for all the driver verification programs
> currently being fleshed out... not just Ironic, of course! :)
>
>
Indeed!

Cheers,
-Deva
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140121/bdc90739/attachment.html>