Open Stack

On Mon, Mar 30, 2015 at 4:06 PM, Doug Wiegley <dougwig at parksidesoftware.com>
wrote:

> A few reasons, I’m sure there are others:
>
> - Broken tests that hardcode something about the ref implementation. The
> test needs to be fixed, of course, but in the meantime, a constantly
> failing CI is worthless (hello, lbaas scenario test.)
>
​Certainly... but that's relatively easy to fix (bug/patch to Tempest).
Although that's not actually the case in this particular context as there
are a handful of third party devices that run the full set of tests that
the ref driver runs with no additional skips or modifications.
​


> - Test relies on some “optional” feature, like overlapping IP subnets that
> the backend doesn’t support.  I’d argue it’s another case of broken tests
> if they require an optional feature, but it still needs skipping in the
> meantime.
>

​This may be something specific to Neutron perhaps?  In Cinder LVM is
pretty much the "lowest common denominator".  I'm not aware of any volume
tests in Tempest that rely on optional features that don't pick this up
automatically out of the config (like multi-backend for example).
​


> - Some new feature added to an interface, in the presence of
> shims/decomposed drivers/plugins (e.g. adding TLS termination support to
> lbaas.) Those implementations will lag the feature commit, by definition.
>

​Yeah, certainly I think this highlights some of the differences between
Cinder and Neutron perhaps and the differences in complexity.
Thanks for the feedback... I don't disagree per say, however Cinder is set
up a bit different here in terms of expectations for base functionality
requirements and compatibility but your points are definitely well taken. ​

>
> Thanks,
> doug
>
>
> On Mar 30, 2015, at 2:54 PM, John Griffith <john.griffith8 at gmail.com>
> wrote:
>
> This may have already been raised/discussed, but I'm kinda confused so
> thought I'd ask on the ML here.  The whole point of third party CI as I
> recall was to run the same tests that we run in the official Gate against
> third party drivers.  To me that would imply that a CI system/device that
> marks itself as "GOOD" doesn't do things like add skips locally that aren't
> in the tempest code already?
>
> In other words, seems like cheating to say "My CI passes and all is good,
> except for the tests that don't work which I skip... but pay no attention
> to those please".
>
> Did I miss something, isn't the whole point of Third Party CI to
> demonstrate that a third parties backend is tested and functions to the
> same degree that the reference implementations do? So the goal (using
> Cinder for example) was to be able to say that any API call that works on
> the LVM reference driver will work on the drivers listed in driverlog; and
> that we know this because they run the same Tempest API tests?
>
> Don't get me wrong, certainly not saying there's malice or things should
> be marked as no good... but if the practice is to skip what you can't do
> then maybe that should be documented in the driverlog submission, as
> opposed to just stating "Yeah, we run CI successfully".
>
> Thanks,
> John
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150330/923475fa/attachment.html>