Open Stack

On 19/01/18 16:27, Andrea Frittoli wrote:
> 
> Thanks for the summary!
> 
> To be honest I don't see why this decision has to be difficult to take.

I think the confusion comes from one thing being decided already, and
now conflicting direction is being given to teams, without anyone
updating the Governance repo.

(It is not as if there was not plenty of warning, I have been raising it
as an issue for over a year)

> Nothing we decide today is written in stone and the main risk ahead of
> us is to take
> a decision that requires a lot of upfront work and that it ends up
> providing no
> significant benefit, or even making things worst in some aspect. So we
> may try one way
> today and if we hit some significant issue we can still change.
> 
> TL;DR my preferred option would be number (2) - it's the least initial
> effort, so the
> least risk, and deciding for (2) now won't make it any difficult in the
> future to switch
> to option (1) or option (3). I'm not pushing back on (2), I just think
> (1) is more convenient.
> Details below each option.
>  
> 
> 
>     So far the patch proposes three options:
> 
>     1) All trademark-related tests should go in the tempest repo, in
>     accordance
>        with the original resolution. This would mean that even projects
>     that have
>        never had tests in tempest would now have to add at least some of
>     their
>        black-box tests to tempest.
> 
> 
> This option is a valid one, but I think it introduces too much extra
> work and
> testing complications for too little benefit.

What it does do is *guarantee* that the InterOp suite will work, as it
will be CI'd. I see these programs as important enough that we should CI
the tooling used for them, but I seem to be in a minority.

>  
> 
>     The value of this option is that centralizes tests used for the
>     Interop program
>     in a location where interop-minded folks from the QA team can
>     control them. 
> 
> 
> There are other ways this can be achieved - it is possible to mark tests
> so that
> team may require a +1 from interop/qa when specific tests are modified.

Is there? AFAIK gerrit does not do per file path permissions, so unless
we have a job that just checks the votes on a patch, and passes or fails
if a test changes (which would be awful for the teams) we cannot do
that.

>  
> 
>     The
>     downside is that projects that so far have avoided having a
>     dependency on
>     tempest will now lose some control over the black-box tests that
>     they use for
>     functional and integration that would now also be used for trademark
>     certification.
>     There's also concern for the review bandwidth of the QA team - we
>     can't expect
>     the QA team to be continually responsible for an ever-growing list
>     of projects
>     and their trademark tests.
> 
> 
> If we restrict to interop tests, the review bandwidth issue is probably
> not so bad.
> The QA team would have to request the domain knowledge required for proper
> review from the respective teams anyways.
> 
> There are other complications introduced though:
> 
> - service clients and other common bits (config and so) would have to
> move to
>   Tempest since we cannot have tempest depend on plugins. But then modifying
>   those common bits on Tempest side would risk to break non-interop tests.
>   Solution for that is to make all those bits stable interfaces for plugins

Is this not already the case? e.g. the neutron plugin uses the nova
service client already.

This would also help for the neutron plugin which is currently importing
DNS service clients from the designate-tempest-plugin repo - having them
in the tempest repo would allow them to to be more stable, and remove
the extra dependency.

>  
> - tempest would have to add new CI jobs to run the interop tests from add-on
>   program on every tempest change so that the new code is tested on a
> regular
>   basis.

That is a good thing, and we should probably do that for the other 2
options as well...

> 
> - heat tests are wrapped in a Tempest plugin but actually written in
> Gabbi so we
>   would need to add Gabbi as a dependency to Tempest
> 
> Nothing too terrible really, but I think it might not be worth the extra
> effort, especially
> now that teams available resources are getting thinner and thinner.
> 
> 
>     2) All trademark-related tests for *add-on projects* should be
>     sourced from
>        plugins external to tempest.
> 
> I wouldn't go as far as saying they "should" be sourced. I think saying
> that they
> *may* be sourced from a plugin is enough. Apart from that this is my
> favourite
> option. The only thing required really is updating the resolution and we are
> ready to go.
> 
> With all the plugins no in own branchless repositories, the usability
> concern is
> not so strong anymore really.
>  
> 
>     The value of this option is it allows project teams to retain
>     control over
>     these tests. 
> 
> 
> Other value is given by simplicity, least changes to implement and low risk.

I do think if we do this for add-ons, we should be doing it for other
programs as well, so that resolution will just be deleted,and it will
allow other teams to have the same control, and further reduce the
review load for QA.

>  
> 
>     The potential problem with it is that individual project teams are
>     not necessarily reviewing test changes with an eye for interop
>     concerns and so
>     could inadvertently change the behavior of the
>     trademark-verification tools.

This is a big issue, and I think it is overlooked.

<snip/>

So, thus far, we have had 3 responses from people working on the QA
tooling, with one for Option 1, one for Option 2, and one for Option 1
if Heat + Designate are now part of "core".

If I start migrating tooling to tempest from designate, will it be -2'd?


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180122/b4ae9141/attachment.sig>