Open Stack

Excerpts from Thierry Carrez's message of 2017-06-01 11:51:50 +0200:
> Graham Hayes wrote:
> > On 01/06/17 01:30, Matthew Treinish wrote:
> >> TBH, it's a bit premature to have the discussion. These additional programs do
> >> not exist yet, and there is a governance road block around this. Right now the
> >> set of projects that can be used defcore/interopWG is limited to the set of 
> >> projects in:
> >>
> >> https://governance.openstack.org/tc/reference/tags/tc_approved-release.html
> > 
> > Sure - but that is a solved problem, when the interop committee is
> > ready to propose them, they can add projects into that tag. Or am I
> > misunderstanding [1] (again)?
> 
> I think you understand it well. The Board/InteropWG should propose
> additions/removals of this tag, which will then be approved by the TC:
> 
> https://governance.openstack.org/tc/reference/tags/tc_approved-release.html#tag-application-process
> 
> > [...]
> >> We had a forum session on it (I can't find the etherpad for the session) which
> >> was pretty speculative because it was about planning the new programs. Part of
> >> that discussion was around the feasibility of using tests in plugins and whether
> >> that would be desirable. Personally, I was in favor of doing that for some of
> >> the proposed programs because of the way they were organized it was a good fit.
> >> This is because the proposed new programs were extra additions on top of the
> >> base existing interop program. But it was hardly a definitive discussion.
> > 
> > Which will create 2 classes of testing for interop programs.
> 
> FWIW I would rather have a single way of doing "tests used in trademark
> programs" without differentiating between old and new trademark programs.
> 
> I fear that we are discussing solutions before defining the problem. We
> want:
> 
> 1- Decentralize test maintenance, through more tempest plugins, to
> account for limited QA resources
> 2- Additional codereview constraints and approval rules for tests that
> happen to be used in trademark programs
> 3- Discoverability/ease-of-install of the set of tests that happen to be
> used in trademark programs
> 4- A git repo layout that can be simply explained, for new teams to
> understand
> 
> It feels like the current git repo layout (result of that 2016-05-04
> resolution) optimizes for 2 and 3, which kind of works until you add
> more trademark programs, at which point it breaks 1 and 4.
> 
> I feel like you could get 2 and 3 without necessarily using git repo
> boundaries (using Gerrit approval rules and some tooling to install/run
> subset of tests across multiple git repos), which would allow you to
> optimize git repo layout to get 1 and 4...
> 
> Or am I missing something ?
> 

Right. The point of having the trademark tests "in tempest" was not
to have them "in the tempest repo", that was just an implementation
detail of the policy of "put them in a repository managed by people
who understand the expanded review rules".

There were a lot of unexpected issues when we started treating the
test suite as a production tool for validating a cloud.  We have
to be careful about how we change the behavior of tests, for example,
even if the API responses are expected to be the same.  It's not
fair to vendors or operators who get trademark approval with one
release to have significant changes in behavior in the exact same
tests for the next release.

At the early stage, when the DefCore team was still figuring out
these issues, it made sense to put all of the tests in one place
with a review team that was actively participating in establishing
the process. If we better understand the "rules" for these tests
now, we can document them and distribute the work of maintaining the
test suites.

And yes, I agree with the argument that we should be fair and treat
all projects the same way. If we're going to move tests out of the
tempest repository, we should move all of them. The QA team can
still help maintain the test suites for whatever projects they want,
even if those tests are in plugins.

Doug