Open Stack

On Wed, Jan 15, 2014 at 6:03 PM, Michael Still <mikal at stillhq.com> wrote:
> On Thu, Jan 16, 2014 at 6:28 AM, John Griffith
> <john.griffith at solidfire.com> wrote:
>> Hey Everyone,
>>
>> A while back I started talking about this idea of requiring Cinder
>> driver contributors to run a super simple cert script (some info here:
>> [1]).  Since then I've been playing with introduction of a third party
>> gate check here in my own lab.  My proposal was to have a non-voting
>> check that basically duplicates the base devstack gate test in my lab,
>> but uses different back-end devices that I have available configured
>> in Cinder to run periodic tests against.  Long term I'd like to be
>> able to purpose this gear to also do something "more useful" for the
>> over all OpenStack gating effort but to start it's strictly an
>> automated verification of my Cinder driver/backend.
>>
>> What I'm questioning is how to report this information and the
>> results.  Currently patches and reviews are our mechanism for
>> triggering tests and providing feedback.  Myself and many other
>> vendors that might like to participate in something like this
>> obviously don't have the infrastructure to try and run something like
>> this on every single commit.  Also since it would be non-voting it's
>> difficult to capture and track the results.
>>
>> One idea that I had was to set something like what I've described
>> above to run locally on a periodic basis (weekly, nightly etc) and
>> publish results to something like a "third party verification
>> dashboard".  So the idea would be that results from various third
>> party tests would all adhere to a certain set of criteria WRT what
>> they do and what they report  and those results would be logged and
>> tracked publicly for anybody in the OpenStack community to access and
>> view?
>
> My concern here is how to identify what patch broke the third party
> thing. If you run this once a week, then there are possible hundreds
> of patches which might be responsible. How do you identify which one
> is the winner?

To be honest I'd like to see more than once a week, however the main
point of this is to have public testing of third party drivers.
Currently we say "it's in trunk and passed review and unit tests" so
you're good to go.  Frankly that's not sufficient, there needs to be
some sort of testing publicly that shows that a product/config
actually works in the minimum sense at least.  This won't address
things like a bad patch breaking things, but again in Cinder's case
this is a bit different, it is designed more to show compatibility and
integration completeness.  If a patch goes in and breaks a vendors
driver but not the reference implementation, that means the vendor has
work to do bring their driver up to date.

Cinder is not a dumping ground, the drivers in the code base should no
be static but require continued maintenance and development as the
project grows.

Non-Voting tests on every patch seems unrealistic, however there's no
reason that if vendors have the resources they couldn't do that if
they so choose.

>
>> Does this seem like something that others would be interested in
>> participating in?  I think it's extremely valuable for projects like
>> Cinder that have dozens of backend devices, and regardless of other
>> interest or participation in the community I intend to implement
>> something like this on my own regardless.  It would just be
>> interesting to see if we could have an organized and official effort
>> to gather this sort of information and run these types of tests.
>>
>> Open to suggestions and thoughts as well as any of you that may
>> already be doing this sort of thing.  By the way, I've been looking at
>> things like SmokeStack and other third party gating checks to get some
>> ideas as well.
>
> Michael
>
> --
> Rackspace Australia
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev