Open Stack

On Tue, Jan 24, 2017 at 12:46 PM, Jeremy Stanley <fungi at yuggoth.org> wrote:
> On 2017-01-24 10:51:39 -0800 (-0800), Ihar Hrachyshka wrote:
>> On Tue, Jan 24, 2017 at 12:50 AM, Kevin Benton <kevin at benton.pub> wrote:
>> > I'm on board with getting visibility into the drivers with improvements to
>> > driverlog, etc. What I'm uncertain of is providing much in the lines of
>> > 'validation'. Core reviewers don't frequently have access to the hardware or
>> > software required to validate these drivers so we can't be sure if the
>> > features really are working as expected.
>> >
>> > If validation is as flexible as you highlighted in the email, we can at
>> > least get it to a point where all recent CI runs are linkable from driverlog
>> > and people can see recent tempest runs. I don't foresee the Neutron team
>> > getting to a point soon where we vouch for certain drivers though just
>> > because it is so hard to keep up with their changes (even ignoring changes
>> > in the vendor hardware itself).
>>
>> Good point. We may guide plugins and drivers on how to set up CI; we
>> may help Foundation to set up Marketplace in such a way that would
>> allow to automatically consume test artifacts from driver owners; we
>> may provide guidance to Foundation about which features are more
>> important to reflect that in Marketplace; but I would hope we don't
>> put the Neutron team on the hook to validate each driver, or even
>> police CI owners to produce consumable results. (The stick in the
>> latter case would be driver not showing up in Marketplace, or showing
>> up with no feature support information.)
>
> I guess the question I have is who, then, can tell our
> operators/users what Neutron drivers are reasonably supported? It
> sounds like you're saying Neutron developers are not well-placed to
> determine that, which leaves us with these other options:
>
> A. Have the OpenStack Foundation as maintainers of the Marketplace
>    decide which Neutron drivers are usable (they don't really staff
>    for this purpose so would be throwing darts I think)
>
> B. Trust the driver authors to declare whether they're supported and
>    what features they provide (maybe that works better than I
>    expect?)
>
> C. Identify another party with a vested interest in validating
>    driver support (a board of operators from different organizations
>    maybe?)
>
> D. Provide links/aggregation of QA/CI and let the consumers attempt
>    to divine supportability for themselves (seems a bit downstream
>    hostile)
>
> Are any of those options preferable?

I think it's B, but under Neutron team supervision. As a first step,
we would need to come up with an objective way to compare existing
drivers, and anarchy in how they execute tests, set up environment,
and publish artifacts won't make it easy.

I believe that as long as vendor CI setups are in line with our
guidelines (execute all tests and pass them, configure api_extensions
in tempest.conf to explicitly enumerate supported extensions), then we
can feed test artifacts into Marketplace, where a driver would get a
'supported'/'tested' badge for each extension enumerated in
tempest.conf as long as the latest run is success. If test run failed,
or an extension was not enumerated, then we don't have information
about whether it works with the driver, and hence will show N/A or
Unsupported in corresponding per-feature column.

Of course I assume here that support information useful to users is of
extension granularity. I think that's a reasonable assumption and is
in line with how Networking API is supposed to be consumed. We may
introduce additional tempest flags for special cases like ipv6 support
(already present) if better granularity is needed.

Ihar