Open Stack

On Mon, Jan 22, 2018 at 3:05 PM Graham Hayes <gr at ham.ie> wrote:

>
>
> On 19/01/18 16:27, Andrea Frittoli wrote:
> >
> > Thanks for the summary!
> >
> > To be honest I don't see why this decision has to be difficult to take.
>
> I think the confusion comes from one thing being decided already, and
> now conflicting direction is being given to teams, without anyone
> updating the Governance repo.
>
> (It is not as if there was not plenty of warning, I have been raising it
> as an issue for over a year)
>
> > Nothing we decide today is written in stone and the main risk ahead of
> > us is to take
> > a decision that requires a lot of upfront work and that it ends up
> > providing no
> > significant benefit, or even making things worst in some aspect. So we
> > may try one way
> > today and if we hit some significant issue we can still change.
> >
> > TL;DR my preferred option would be number (2) - it's the least initial
> > effort, so the
> > least risk, and deciding for (2) now won't make it any difficult in the
> > future to switch
> > to option (1) or option (3). I'm not pushing back on (2), I just think
> > (1) is more convenient.
> > Details below each option.
> >
> >
> >
> >     So far the patch proposes three options:
> >
> >     1) All trademark-related tests should go in the tempest repo, in
> >     accordance
> >        with the original resolution. This would mean that even projects
> >     that have
> >        never had tests in tempest would now have to add at least some of
> >     their
> >        black-box tests to tempest.
> >
> >
> > This option is a valid one, but I think it introduces too much extra
> > work and
> > testing complications for too little benefit.
>
> What it does do is *guarantee* that the InterOp suite will work, as it
> will be CI'd. I see these programs as important enough that we should CI
> the tooling used for them, but I seem to be in a minority.
>

Add-ons intreoperability tests will be CI'd for every change in Tempest as
long
as they are executed in a job that runs on every change in Tempest.

This can be achieve regardless of the location of the tests and having the
tests
in the Tempest tree is not by itself a guarantee that they will be executed
against
every change.



>
> >
> >
> >     The value of this option is that centralizes tests used for the
> >     Interop program
> >     in a location where interop-minded folks from the QA team can
> >     control them.
> >
> >
> > There are other ways this can be achieved - it is possible to mark tests
> > so that
> > team may require a +1 from interop/qa when specific tests are modified.
>
> Is there? AFAIK gerrit does not do per file path permissions, so unless
> we have a job that just checks the votes on a patch, and passes or fails
> if a test changes (which would be awful for the teams) we cannot do
> that.
>

If we really want to enforce having a vote from someone it may be tricky,
yes, but I don't think enforcement is what we need, rather awareness,
For governance and project-config patches reviewed always ask for a +1
from the project PTL where relevant, and add-on project reviewers could
do the same.

To help building awareness we could have automation in place to post a
comment to Gerrit, like we do for elastic recheck. We could do that
on every change to the plugin in the beginning and include  a link to the
interoperability recommendation to help reviewers in their job.


>
> >
> >
> >     The
> >     downside is that projects that so far have avoided having a
> >     dependency on
> >     tempest will now lose some control over the black-box tests that
> >     they use for
> >     functional and integration that would now also be used for trademark
> >     certification.
> >     There's also concern for the review bandwidth of the QA team - we
> >     can't expect
> >     the QA team to be continually responsible for an ever-growing list
> >     of projects
> >     and their trademark tests.
> >
> >
> > If we restrict to interop tests, the review bandwidth issue is probably
> > not so bad.
> > The QA team would have to request the domain knowledge required for
> proper
> > review from the respective teams anyways.
> >
> > There are other complications introduced though:
> >
> > - service clients and other common bits (config and so) would have to
> > move to
> >   Tempest since we cannot have tempest depend on plugins. But then
> modifying
> >   those common bits on Tempest side would risk to break non-interop
> tests.
> >   Solution for that is to make all those bits stable interfaces for
> plugins
>
> Is this not already the case? e.g. the neutron plugin uses the nova
> service client already.
>

What I was saying is that Tempest cannot depend on a Tempest plugin,
so service clients should be in Tempest. Which is absolutely fine, I'm was
just listing out the work that needs to be done if we move add-on
interoperability tests into Tempest.


>
> This would also help for the neutron plugin which is currently importing
> DNS service clients from the designate-tempest-plugin repo - having them
> in the tempest repo would allow them to to be more stable, and remove
> the extra dependency.
>
> >
> > - tempest would have to add new CI jobs to run the interop tests from
> add-on
> >   program on every tempest change so that the new code is tested on a
> > regular
> >   basis.
>
> That is a good thing, and we should probably do that for the other 2
> options as well...
>
> >
> > - heat tests are wrapped in a Tempest plugin but actually written in
> > Gabbi so we
> >   would need to add Gabbi as a dependency to Tempest
> >
> > Nothing too terrible really, but I think it might not be worth the extra
> > effort, especially
> > now that teams available resources are getting thinner and thinner.
> >
> >
> >     2) All trademark-related tests for *add-on projects* should be
> >     sourced from
> >        plugins external to tempest.
> >
> > I wouldn't go as far as saying they "should" be sourced. I think saying
> > that they
> > *may* be sourced from a plugin is enough. Apart from that this is my
> > favourite
> > option. The only thing required really is updating the resolution and we
> are
> > ready to go.
> >
> > With all the plugins no in own branchless repositories, the usability
> > concern is
> > not so strong anymore really.
> >
> >
> >     The value of this option is it allows project teams to retain
> >     control over
> >     these tests.
> >
> >
> > Other value is given by simplicity, least changes to implement and low
> risk.
>
> I do think if we do this for add-ons, we should be doing it for other
> programs as well, so that resolution will just be deleted,and it will
> allow other teams to have the same control, and further reduce the
> review load for QA.
>

If you mean to move interoperability tests only, this would create an
artificial split in tests and it would not have a real impact on the review
load of the QA team.

If you mean moving all tests, that would take a while since we have
a number of stable interfaces (including config) that we cannot just
remove. Besides tests are an integral part of Tempest I would not
move them out.


> >
> >
> >     The potential problem with it is that individual project teams are
> >     not necessarily reviewing test changes with an eye for interop
> >     concerns and so
> >     could inadvertently change the behavior of the
> >     trademark-verification tools.
>
> This is a big issue, and I think it is overlooked.
>
> <snip/>
>
> So, thus far, we have had 3 responses from people working on the QA
> tooling, with one for Option 1, one for Option 2, and one for Option 1
> if Heat + Designate are now part of "core".
>
> If I start migrating tooling to tempest from designate, will it be -2'd?
>

I still don't understand why you'd want to invest much of your and of the
QA team time in moving things around now - I think it would be too much
of trying to solve a problem before it exists.

But if the majority agrees this is the best course of action I won't oppose
it.
I would start with an etherpad or spec or something with a checklist of
things
to be done and expected target state in terms of what's where and what's
tested where.

The plan of what needs to be tested where has to be done regardless of
the location of the tests.

Andrea Frittoli (andreaf)


>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180122/0280e980/attachment.html>