[openstack-dev] [cinder][neutron][all] New third-party-ci testing requirements for OpenStack Compatible mark

Armando M. armamig at gmail.com
Mon Oct 5 23:22:26 UTC 2015


On 29 September 2015 at 08:28, Chris Hoge <chris at openstack.org> wrote:

> On Sep 29, 2015, at 8:04 AM, Erlon Cruz <sombrafam at gmail.com> wrote:
>
>
> Hi Cris,
>
> There are some questions that came to my mind.
>
> Cinder has near zero tolerance to backends that does not have a CI
> running. So, can one assume that all drivers in Cinder will have the
> "OpenStack Compatible" seal?
>
>
> One of the reasons we started with Cinder was because they have
> have an existing program that is well maintained. Any driver passing
> CI becomes eligible for the "OpenStack Compatible” mark. It’s not
> automatic, and still needs a signed agreement with the Foundation.
>

> When you say that the driver have to 'pass' the integration tests, what
> tests do you consider? All tests in tempest? All patches? Do you have any
> criteria to determine if a backend is passing or not?
>
>
> We’re letting the project drive what tests need to be passed. So,
> taking a look at this dashboard[1] (it’s one of many that monitor
> our test systems) the drivers are running the dsvm-tempest-full
> tests. One of the things that the tests exercise, and we’re interested
> in from the driver standpoint, are both the user-facing Cinder APIs
> as well as the driver-facing APIs.
>

> For Neutron, which we would like to help roll out in the coming year,
> this would be a CI run that is defined by the Neutron development
> team. We have no interest in dictating to the developers what should
> be run. Instead, we want to adopt what the community considers
> to be the best-practices and standards for drivers.
>

We've experienced that tracking the CI's outcome on a per-commit basis can
only lead to insanity, at least as far as Neutron is concerned.

I have been mulling over the idea (I am sure I am not the only one) that
compliance should be sought and validated at specific milestones of
interest (when the stable branch is cut? When the milestone RC is ready?
etc). We can then collect the outcome of all the CI's reporting back,
vetting the output, verifying who is bogus and who isn't etc. The
post-processing will inevitably require some degree of manual intervention.
We can also ask for those results to be persistent for longer.

Is it something that would be acceptable?

Obviously the first step for us, Neutron, is to come up with a testing
suite(s) that's representative of all the various flavors of support that a
networking solution can provide when it claims to be integrated with
Neutron.


>
> About this "OpenStack Compatible" flag, how does it work? Will you hold a
> list with the Compatible vendors? Is anything a vendor need to to in order
> to use this?
>
>
> “OpenStack Compatible” is one of the trademark programs that is
> administered by the Foundation. A company that want to apply the
> OpenStack logo to their product needs to sign a licensing agreement,
> which gives them the right to use the logo in their marketing materials.
>
> We also create an entry in the OpenStack Marketplace for their
> product, which has information about the company and the product, but
> also information about tests that the product may have passed. The
> best example I can give right now is with the “OpenStack Powered”
> program, where we display which Defcore guideline a product has
> successfully passed[2].
>
> Chris
>
> [1] http://ci-watch.tintri.com/project?project=cinder&time=24+hours
> [2] For example:
> http://www.openstack.org/marketplace/public-clouds/unitedstack/uos-cloud
>
> Thanks,
> Erlon
>
> On Mon, Sep 28, 2015 at 5:55 PM, Kyle Mestery <mestery at mestery.com> wrote:
>
>> The Neutron team also discussed this in Vancouver, you can see the
>> etherpad here [1]. We talked about the idea of creating a validation suite,
>> and it sounds like that's something we should again discuss in Tokyo for
>> the Mitaka cycle. I think a validation suite would be a great step forward
>> for Neutron third-party CI systems to use to validate they work with a
>> release.
>>
>> [1] https://etherpad.openstack.org/p/YVR-neutron-third-party-ci-liberty
>>
>> On Sun, Sep 27, 2015 at 11:39 AM, Armando M. <armamig at gmail.com> wrote:
>>
>>>
>>>
>>> On 25 September 2015 at 15:40, Chris Hoge <chris at openstack.org> wrote:
>>>
>>>> In November, the OpenStack Foundation will start requiring vendors
>>>> requesting
>>>> new "OpenStack Compatible" storage driver licenses to start passing the
>>>> Cinder
>>>> third-party integration tests.
>>>
>>> The new program was approved by the Board at
>>>> the July meeting in Austin and follows the improvement of the testing
>>>> standards
>>>> and technical requirements for the "OpenStack Powered" program. This is
>>>> all
>>>> part of the effort of the Foundation to use the OpenStack brand to
>>>> guarantee a
>>>> base-level of interoperability and consistency for OpenStack users and
>>>> to
>>>> protect the work of our community of developers by applying a trademark
>>>> backed
>>>> by their technical efforts.
>>>>
>>>> The Cinder driver testing is the first step of a larger effort to apply
>>>> community determined standards to the Foundation marketing programs.
>>>> We're
>>>> starting with Cinder because it has a successful testing program in
>>>> place, and
>>>> we have plans to extend the program to network drivers and OpenStack
>>>> applications. We're going require CI testing for new "OpenStack
>>>> Compatible"
>>>> storage licenses starting on November 1, and plan to roll out network
>>>> and
>>>> application testing in 2016.
>>>>
>>>> One of our goals is to work with project leaders and developers to help
>>>> us
>>>> define and implement these test programs. The standards for third-party
>>>> drivers and applications should be determined by the developers and
>>>> users
>>>> in our community, who are experts in how to maintain the quality of the
>>>> ecosystem.
>>>>
>>>> We welcome and feedback on this program, and are also happy to answer
>>>> any
>>>> questions you might have.
>>>>
>>>
>>> Thanks for spearheading this effort.
>>>
>>> Do you have more information/pointers about the program, and how Cinder
>>> in particular is
>>> paving the way for other projects to follow?
>>>
>>> Thanks,
>>> Armando
>>>
>>>
>>>> Thanks!
>>>>
>>>> Chris Hoge
>>>> Interop Engineer
>>>> OpenStack Foundation
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> <http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe>
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> <http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151005/de1a3453/attachment.html>


More information about the OpenStack-dev mailing list