[openstack-dev] [OpenStack-Dev] Third party testing

Robert Collins robertc at robertcollins.net
Sat Jan 18 01:24:14 UTC 2014

On 18 January 2014 06:42, John Griffith <john.griffith at solidfire.com> wrote:
> On Fri, Jan 17, 2014 at 1:15 AM, Robert Collins
> <robertc at robertcollins.net> wrote:

> Maybe this is going a bit sideways, but my point was that making a
> first step of getting periodic runs on vendor gear and publicly
> submitting those results would be a good starting point and a
> SIGNIFICANT improvement over what we have today.
> It seems to me that "requiring" every vendor to have a deployment in
> house dedicated and reserved 24/7 might be a tough order right out of
> the gate.  That being said, of course I'm willing and able to do that
> for my employer, but feedback from others hasn't been quite so
> amiable.
> The feedback here seems significant enough that maybe gating every
> change is the way to go though.  I'm certainly willing to opt in to
> that model and get things off the ground.  I do have a couple of
> concerns (number 3 begin the most significant):
> 1. I don't want ANY commit/patch waiting for a Vendors infrastructure
> to run a test.  We would definitely need a timeout mechanism or
> something along those lines to ensure none of this disrupts the gate
> 2. Isolating this to changes in Cinder seems fine, the intent was
> mostly a compatability / features check.  This takes it up a notch and
> allows us to detect when something breaks right away which is
> certainly a good thing.
> 3. Support and maintenance is a concern here.  We have a first rate
> community that ALL pull together to make our gating and infrastructure
> work in OpenStack.  Even with that it's still hard for everybody to
> keep up due to number of project and simply the volume of patches that
> go in on a daily basis.  There's no way I could do my regular jobs
> that I'm already doing AND maintain my own fork/install of the
> OpenStack gating infrastructure.
> 4. Despite all of the heavy weight corporation throwing resource after
> resource at OpenStack, keep in mind that it is an Open Source
> community still.  I don't want to do ANYTHING that would make it some
> unfriendly to folks who would like to commit.  Keep in mind that
> vendors here aren't necessarily all large corporations, or even all
> paid for proprietary products.  There are open source storage drivers
> for example in Cinder and they may or may not have any of the
> resources to make this happen but that doesn't mean they should not be
> allowed to have code in OpenStack.
> The fact is that the problem I see is that there are drivers/devices
> that flat out don't work and end users (heck even some vendors that
> choose not to test) don't know this until they've purchased a bunch of
> gear and tried to deploy their cloud.  What I was initially proposing
> here was just a more formal public and community representation of
> whether a device works as it's advertised or not.
> Please keep in mind that my proposal here was a first step sort of
> test case.  Rather than start with something HUGE like deploying the
> OpenStack CI in every vendors lab to test every commit (and I"m sorry
> for those that don't agree but that does seem like a SIGNIFICANT
> undertaking), why not take incremental steps to make things better and
> learn as we go along?

Certainly - I totally agree that anything >> nothing. I was asking
about your statement of not having enough infra to get a handle on
what would block things. As you know, tripleo is running up a
production quality test cloud to test tripleo, Ironic and once we get
everything in place - multinode gating jobs. We're *super* interested
in making the bar to increased validation as low as possible.

I broadly agree with your points 1 through 4, of course!


Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

More information about the OpenStack-dev mailing list