Open Stack

On 02/13/2014 02:51 AM, Thierry Carrez wrote:
> John Griffith wrote:
>> So we've talked about this a bit and had a number of ideas regarding
>> how to test and show compatibility for third-party drivers in Cinder.
>> This has been an eye opening experience (the number of folks that have
>> NEVER run tempest before, as well as the problems uncovered now that
>> they're trying it).
>>
>> I'm even more convinced now that having vendors run these tests is a
>> good thing and should be required.  That being said there's a ton of
>> push back from my proposal to require that results from a successful
>> run of the tempest tests to accompany any new drivers submitted to
>> Cinder.
> Could you describe the nature of the pushback ? Is it that the tests are
> too deep and reject valid drivers ? Is it that it's deemed unfair to
> block new drivers while the existing ones aren't better ? Is it that
> it's difficult for them to run those tests and get a report ? Or is it
> because they care more about having their name covered in mainline and
> not so much about having the code working properly ?
>
>> The consensus from the Cinder community for now is that we'll
>> log a bug for each driver after I3, stating that it hasn't passed
>> certification tests.  We'll then have a public record showing
>> drivers/vendors that haven't demonstrated functional compatibility,
>> and in order to close those bugs they'll be required to run the tests
>> and submit the results to the bug in Launchpad.
>>
>> So, this seems to be the approach we're taking for Icehouse at least,
>> it's far from ideal IMO, however I think it's still progress and it's
>> definitely exposed some issues with how drivers are currently
>> submitted to Cinder so those are positive things that we can learn
>> from and improve upon in future releases.
>>
>> To add some controversy and keep the original intent of having only
>> known tested and working drivers in the Cinder release, I am going to
>> propose that any driver that has not submitted successful functional
>> testing by RC1 that that driver be removed.  I'd at least like to see
>> driver maintainers try... if the test fails a test or two that's
>> something that can be discussed, but it seems that until now most
>> drivers just flat out are not even being tested.
> I think there are multiple stages here.
>
> Stage 0: noone knows if drivers work
> Stage 1: we know the (potentially sad) state of the drivers that are in
> the release
> Stage 2: only drivers that pass tests are added, drivers that don't pass
> tests have a gap analysis and a plan to fix it
> Stage 3: drivers that fail tests are removed before release
> Stage 4: 3rd-party testing rigs must run tests on every change in order
> to stay in tree
>
> At the very minimum you should be at stage 1 for the Icehouse release,
> so I agree with your last paragraph. I'd recommend that you start the
> Juno cycle at stage 2 (for new drivers), and try to reach stage 3 for
> the end of the Juno release.
>
I have to agree with Thierry here.  I think if we can get drivers to 
pass the tests
in the Juno timeframe, then it's fine to remove then during Juno.
I think the idea of having drivers run their code through tempest and work
towards passing all of those tests is a great thing for Cinder and 
OpenStack in general.

What I would do different for the Icehouse release is this:

If a driver doesn't pass the certification test by IceHouse RC1, then we 
have a bug filed
against the driver.   I would also put a warning message in the log for 
that driver that it
doesn't pass the certification test.  I would not remove it from the 
codebase.

Also:
    if a driver hasn't even run the certification test by RC1, then we 
mark the driver as
uncertified and deprecated in the code and throw an error at driver init 
time.
We can have a option in cinder.conf that says 
ignore_uncertified_drivers=False.
If an admin wants to ignore the error, they set the flag to True, and we 
let the driver init at next startup.
The admin then takes full responsibility for running uncertified code.

   I think removing the drivers outright is premature for Icehouse, 
since the certification process is a new thing.
For Juno, we remove any drivers that are still marked as uncertified and 
haven't run the tests.

I think the purpose of the tests is to get vendors to actually run their 
code through tempest and
prove to the community that they are willing to show that they are 
fixing their code.  At the end of the day,
it better serves the community and Cinder if we have many working drivers.

My $0.02,
Walt