Open Stack

On 08/03/18 19:31, Doug Hellmann wrote:
> If these new plugins might contain "candidate" tests and all tests
> are potentially candidates, how are these new repos different from
> the existing repos that already contain all of the tests?

We have to allow candidate tests because otherwise there's a 
chicken-and-egg type problem where we can never add tests to an existing 
trademark program. But the definition of 'candidate' that I'm thinking 
of is along the lines of "under active discussion to be added to the 
next iteration of refstack". Perhaps that should be documented.

To answer your question directly, the main way they're different is that 
by putting tests in one of these repos, teams are committing to not 
really change them, and certainly to ~never break backwards compat.

> It seems
> like at least part of the problem with the current system was
> triggered by confusion about when to move tests around to satisfy
> the policy. Can we avoid that problem with the new system? If we're
> not going to move the tests into Tempest itself and have the QA
> team manage them, why not simply take the tests from the repos where
> they already live?

I think a lesson we've learned from Tempest is that there are two parts 
to successfully reviewing a change:

1) Determine whether the change affects a test that is part of the 
trademark program.
2) Don't make the change if it does.

2 is easy, but 1 is hard. By separating the trademark tests into a 
separate repo, we make 1 easy as well. This reduces risk and increases 
review throughput.

> I thought the QA team no longer wanted to be responsible for these
> extra tests. Has that changed again? I've lost track of everyone's
> positions, I'm afraid.

Their position as I understand it is:

* They're not going to write tests
* They're happy to document the process, offer advice, and review tests 
as time allows
* They don't want tests to be thrown over the transom and made their 
problem for the rest of time

That doesn't conflict with anybody's goals here, so it's mostly a matter 
of documenting it clearly so that somebody reading without all this 
context won't get the wrong idea.

cheers,
Zane.