Open Stack

John Griffith wrote:
> So we've talked about this a bit and had a number of ideas regarding
> how to test and show compatibility for third-party drivers in Cinder.
> This has been an eye opening experience (the number of folks that have
> NEVER run tempest before, as well as the problems uncovered now that
> they're trying it).
> 
> I'm even more convinced now that having vendors run these tests is a
> good thing and should be required.  That being said there's a ton of
> push back from my proposal to require that results from a successful
> run of the tempest tests to accompany any new drivers submitted to
> Cinder.

Could you describe the nature of the pushback ? Is it that the tests are
too deep and reject valid drivers ? Is it that it's deemed unfair to
block new drivers while the existing ones aren't better ? Is it that
it's difficult for them to run those tests and get a report ? Or is it
because they care more about having their name covered in mainline and
not so much about having the code working properly ?

> The consensus from the Cinder community for now is that we'll
> log a bug for each driver after I3, stating that it hasn't passed
> certification tests.  We'll then have a public record showing
> drivers/vendors that haven't demonstrated functional compatibility,
> and in order to close those bugs they'll be required to run the tests
> and submit the results to the bug in Launchpad.
> 
> So, this seems to be the approach we're taking for Icehouse at least,
> it's far from ideal IMO, however I think it's still progress and it's
> definitely exposed some issues with how drivers are currently
> submitted to Cinder so those are positive things that we can learn
> from and improve upon in future releases.
> 
> To add some controversy and keep the original intent of having only
> known tested and working drivers in the Cinder release, I am going to
> propose that any driver that has not submitted successful functional
> testing by RC1 that that driver be removed.  I'd at least like to see
> driver maintainers try... if the test fails a test or two that's
> something that can be discussed, but it seems that until now most
> drivers just flat out are not even being tested.

I think there are multiple stages here.

Stage 0: noone knows if drivers work
Stage 1: we know the (potentially sad) state of the drivers that are in
the release
Stage 2: only drivers that pass tests are added, drivers that don't pass
tests have a gap analysis and a plan to fix it
Stage 3: drivers that fail tests are removed before release
Stage 4: 3rd-party testing rigs must run tests on every change in order
to stay in tree

At the very minimum you should be at stage 1 for the Icehouse release,
so I agree with your last paragraph. I'd recommend that you start the
Juno cycle at stage 2 (for new drivers), and try to reach stage 3 for
the end of the Juno release.

-- 
Thierry Carrez (ttx)