Open Stack

On Fri, Jun 2, 2017, at 09:51 AM, Matthew Treinish wrote:
> On Thu, Jun 01, 2017 at 11:57:00AM -0400, Doug Hellmann wrote:
> > Excerpts from Thierry Carrez's message of 2017-06-01 11:51:50 +0200:
> > > Graham Hayes wrote:
> > > > On 01/06/17 01:30, Matthew Treinish wrote:
> > > >> TBH, it's a bit premature to have the discussion. These additional programs do
> > > >> not exist yet, and there is a governance road block around this. Right now the
> > > >> set of projects that can be used defcore/interopWG is limited to the set of 
> > > >> projects in:
> > > >>
> > > >> https://governance.openstack.org/tc/reference/tags/tc_approved-release.html
> > > > 
> > > > Sure - but that is a solved problem, when the interop committee is
> > > > ready to propose them, they can add projects into that tag. Or am I
> > > > misunderstanding [1] (again)?
> > > 
> > > I think you understand it well. The Board/InteropWG should propose
> > > additions/removals of this tag, which will then be approved by the TC:
> > > 
> > > https://governance.openstack.org/tc/reference/tags/tc_approved-release.html#tag-application-process
> > > 
> > > > [...]
> > > >> We had a forum session on it (I can't find the etherpad for the session) which
> > > >> was pretty speculative because it was about planning the new programs. Part of
> > > >> that discussion was around the feasibility of using tests in plugins and whether
> > > >> that would be desirable. Personally, I was in favor of doing that for some of
> > > >> the proposed programs because of the way they were organized it was a good fit.
> > > >> This is because the proposed new programs were extra additions on top of the
> > > >> base existing interop program. But it was hardly a definitive discussion.
> > > > 
> > > > Which will create 2 classes of testing for interop programs.
> > > 
> > > FWIW I would rather have a single way of doing "tests used in trademark
> > > programs" without differentiating between old and new trademark programs.
> > > 
> > > I fear that we are discussing solutions before defining the problem. We
> > > want:
> > > 
> > > 1- Decentralize test maintenance, through more tempest plugins, to
> > > account for limited QA resources
> > > 2- Additional codereview constraints and approval rules for tests that
> > > happen to be used in trademark programs
> > > 3- Discoverability/ease-of-install of the set of tests that happen to be
> > > used in trademark programs
> > > 4- A git repo layout that can be simply explained, for new teams to
> > > understand
> > > 
> > > It feels like the current git repo layout (result of that 2016-05-04
> > > resolution) optimizes for 2 and 3, which kind of works until you add
> > > more trademark programs, at which point it breaks 1 and 4.
> > > 
> > > I feel like you could get 2 and 3 without necessarily using git repo
> > > boundaries (using Gerrit approval rules and some tooling to install/run
> > > subset of tests across multiple git repos), which would allow you to
> > > optimize git repo layout to get 1 and 4...
> > > 
> > > Or am I missing something ?
> > > 
> > 
> > Right. The point of having the trademark tests "in tempest" was not
> > to have them "in the tempest repo", that was just an implementation
> > detail of the policy of "put them in a repository managed by people
> > who understand the expanded review rules".
> 
> There was more to it than this, a big part was duplication of effort as
> well.
> Tempest itself is almost a perfect fit for the scope of the testing
> defcore is
> doing. While tempest does additional testing that defcore doesn't use, a
> large
> subset is exactly what they want.
> 
> > 
> > There were a lot of unexpected issues when we started treating the
> > test suite as a production tool for validating a cloud.  We have
> > to be careful about how we change the behavior of tests, for example,
> > even if the API responses are expected to be the same.  It's not
> > fair to vendors or operators who get trademark approval with one
> > release to have significant changes in behavior in the exact same
> > tests for the next release.
> 
> I actually find this to be kinda misleading. Tempest has always had
> running on any cloud as part of it's mission. I think you're referring
> to the monster defcore thread from last summer about proprietary nova
> extensions
> adding on to API responses. This is honestly a completely separate
> problem
> which is not something I want to dive into again, because that was a much
> more
> nuanced problem that involved much more than just code review.
> 
> > 
> > At the early stage, when the DefCore team was still figuring out
> > these issues, it made sense to put all of the tests in one place
> > with a review team that was actively participating in establishing
> > the process. If we better understand the "rules" for these tests
> > now, we can document them and distribute the work of maintaining the
> > test suites.
> 
> I think you're overestimating how much work is actually being done
> bidirectionally here. The interaction with defcore is more straight
> consumption
> then you might think. They tend to just pick and choose from what tempest
> has
> and don't actually identify gaps or contribute back into tempest much.
> 
> This actually is the crux of my entire concern, that assuming we do
> widely
> expand the number of trademark programs there is an assumption that a
> bunch of
> people are going to show up, write tests, and maintain them. However, all
> past
> evidence shows that this just doesn't ever happen. I linked to that graph
> from
> one of my talks to try and help illustrate this. That was back in the
> days when
> tempest supported all official openstack projects and was a
> **requirement** for
> projects to graduate. Even then the contribution in this space was
> minimal or
> non-existent for newer projects.
> 
> But that being said it's just a concern, we have yet to actually reach
> this
> point because of defcore requirements, and we might never actually get
> there.
> 
> > 
> > And yes, I agree with the argument that we should be fair and treat
> > all projects the same way. If we're going to move tests out of the
> > tempest repository, we should move all of them. The QA team can
> > still help maintain the test suites for whatever projects they want,
> > even if those tests are in plugins.
> 
> Again, where has this been proposed? I've yet to see anything where
> tests being removed from tempest has being proposed anywhere. Also, moving
> tests from tempest to some other place doesn't magically solve any of the issues.
> The fundamental problem I'm concerned with is a large expansion in the number
> of trademark programs overloading a small team. Just saying the QA team can
> still help maintain them doesn't change the scaling problem. It just shifts
> that from the tempest repo to another repo. (or multiples)

Yeah, we should define and clear the problems before discussing the
solution. I
think Thierry already mentioned about that like below.
-----------
> > > I fear that we are discussing solutions before defining the problem. We
> > > want:
> > > 
> > > 1- Decentralize test maintenance, through more tempest plugins, to
> > > account for limited QA resources
> > > 2- Additional codereview constraints and approval rules for tests that
> > > happen to be used in trademark programs
> > > 3- Discoverability/ease-of-install of the set of tests that happen to be
> > > used in trademark programs
> > > 4- A git repo layout that can be simply explained, for new teams to
> > > understand
-----------

> 
> But as I've said so many times, where is there a formal proposal for the
> program expansion? Where has a proposed test for defcore been rejected from
> tempest? This all is still very in loose terms and seems to be a completely
> hypothetical discussion. I feel like we're searching for a problem to solve
> before there is actually one.

I agree here, too. Of course, we can have casual conversation. But I
feel this topic
it too big for a *casual*.

And if we should change something, I think it's better to have concrete
patches,
specs and/or *formal* proposal like Matthew said.
Otherwise, discussion may not be productive.

-- Masayuki Igawa


> 
> -Matt Treinish
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> Email had 1 attachment:
> + signature.asc
>   1k (application/pgp-signature)