[openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
sean at dague.net
Thu Mar 20 20:17:24 UTC 2014
I will agree that the TC language is not as strong as it should be (and
really should be clarified, but I don't think that's going to happen
until the release is looking solid). Honestly, though, I think Sahara is
a good example here of the level of that we expect. They have actively
engaged with upstream infra and used it to the full extent possible.
Then wrote additional tooling to do even more and report 3rd party in on
It's also worth noting they got there by actively participating in the
upstream community and conversations, and clearly making upstream test
integration a top priority for the cycle.
On 03/20/2014 03:13 PM, Malini Kamalambal wrote:
> Thanks Matt for your response !! It has clarified some of the 'cloudy
> areas' ;)
> "So having only looked at the Marconi ML thread and not the actual TC
> minutes I might be missing the whole picture. But, from what I saw when I
> at both a marconi commit and a tempest commit is that there is no gating
> devstack-gate job on marconi commits. It's only non-voting in the check
> Additionally, there isn't a non-voting job on tempest or devstack-gate. For
> example, look at how savanna has it's tempest jobs setup and this is what
> needs to have."
> I am not dismissing the fact that Marconi does not have a voting gate job.
> All we have currently is a non-voting check job in Marconi, and
> experimental jobs in Tempest & devstack-gate.
> So we definitely have more work to do there, starting with making the
> devstack-tempest job voting in Marconi.
> But what caught me by surprise is 'the extend of gate testing -
> st_queues.py' was brought up as a concern.
> We have always had an extensive functional test suite in Marconi , which
> is run at gate against a marconi-server that the test spins up (This was
> implemented before we had devstack integration in place. It is in our
> plans to run the functional test suite in Marconi against devstack
> But my point is not do a postmortem on everything that happened, rather
> come up with an objective list of items that I (or anybody who wants to
> meet the Tempest criteria for graduation- No way meant to imply that
> Tempest work stops after graduation ;) ) need to complete. I started a
> wiki page to document what will be a good enough graduation criteria to
> graduate from the QA perspective.
> It will help if the QA team can add their thoughts to this.
> This can maybe become a recommendation to the TC on what the QA graduation
> requirements are.
> "What do you mean by project specific functional testing? What makes
> a marconi failure in a tempest gate job any more involved than debugging a
> nova or neutron failure? Part of the point of having an integrated gate is
> saying that the project works well with all the others in OpenStack. IMO
> not just in project functionality but also in community. When there is an
> with a gate job everyone comes together to work on it. For example if you
> a keystone patch that breaks a marconi test in check there is open
> about what happened and how to fix it."
> 'project specific functional testing' in the Marconi context is treating
> Marconi as a complete system, making Marconi API calls & verifying the
> response - just like an end user would, but without keystone. If one of
> these tests fail, it is because there is a bug in the Marconi code , and
> not because its interaction with Keystone caused it to fail.
> "That being said there are certain cases where having a project specific
> functional test makes sense. For example swift has a functional test job
> starts swift in devstack. But, those things are normally handled on a per
> basis. In general if the project is meant to be part of the larger
> ecosystem then Tempest is the place to put functional testing. That way
> you know
> it works with all of the other components. The thing is in openstack what
> like a project isolated functional test almost always involves another
> in real use cases. (for example keystone auth with api requests)
> One of the concerns we heard in the review was 'having the functional
> tests elsewhere (I.e within the project itself) does not count and they
> have to be in Tempest'.
> This has made us as a team wonder if we should migrate all our functional
> tests to Tempest.
> But from Matt's response, I think it is reasonable to continue in our
> current path & have the functional tests in Marconi coexist along with
> the tests in Tempest.
> On 3/20/14 1:59 PM, "Matthew Treinish" <mtreinish at kortar.org> wrote:
>> On Thu, Mar 20, 2014 at 11:35:15AM +0000, Malini Kamalambal wrote:
>>> Hello all,
>>> I have been working on adding tests in Tempest for Marconi, for the
>>> last few months.
>>> While there are many amazing people to work with, the process has been
>>> more difficult than I expected.
>>> Couple of pain-points and suggestions to make the process easier for
>>> myself & future contributors.
>>> 1. The QA requirements for a project to graduate needs details beyond
>>> the "Project must have a *basic* devstack-gate job set up"
>>> 2. The scope of Tempest needs clarification - what tests should be in
>>> Tempest vs. in the individual projects? Or should they be in both
>>> tempest and the project?
>>> See details below.
>>> 1. There is little documentation on graduation requirement from a QA
>>> perspective beyond 'Project must have a basic devstack-gate job set up'.
>>> As a result, I hear different interpretations on what a basic devstack
>>> gate job is.
>>> This topic was discussed in one of the QA meetings a few weeks back .
>>> Based on the discussion there, having a basic job - such as one that
>>> will let us know 'if a keystone change broke marconi' was good enough.
>>> My efforts in getting Marconi meet graduation requirements w.r.t
>>> Tempest was based on the above discussion.
>>> However, my conversations with the TC during Marconi's graduation
>>> review lead me to believe that these requirements aren't yet formalized.
>>> We were told that we needed to have more test coverage in tempest, &
>>> having them elsewhere (i.e. functional tests in the Marconi project
>>> itself) was not good enough.
>> So having only looked at the Marconi ML thread and not the actual TC
>> minutes I might be missing the whole picture. But, from what I saw when I
>> at both a marconi commit and a tempest commit is that there is no gating
>> devstack-gate job on marconi commits. It's only non-voting in the check
>> Additionally, there isn't a non-voting job on tempest or devstack-gate.
>> example, look at how savanna has it's tempest jobs setup and this is what
>> needs to have.
>>> I will never debate the value of having good test coverage - after all
>>> I define myself professionally as a QA ;)
>>> I am proud of the unit and functional test suites & the test coverage
>>> we have in Marconi .
>>> Marconi team is continuing its efforts in this direction.
>>> We are looking forward to adding more tests in Tempest and making sure
>>> Marconi is in par with the community standards.
>>> But what frustrates me is that the test requirements seem to evolve,
>>> catching new contributors by surprise.
>>> It will really help to have these requirements documented in detail -
>>> answering at least the following questions,
>>> a. What tests are needed to graduate - API, Scenario, CLI?
>>> b. How much coverage is good enough to graduate?
>>> That will make sure that contributors focus their time & energy in the
>>> right direction.
>>> I am willing to lead the effort to document the QA-level graduation
>>> requirements for a project and help solidify them.
>> Testing contributions will always be an iterative process. The actual test
>> coverage doesn't matter as much up front. The graduation requirement as I
>> understood it was just to have the glue in place and to verify that
>> runs. As long as there is steady contribution and interaction from the
>> community with tempest IMO that matters far more then actually having
>> coverage upfront.
>>> 2. Clarify the scope of Tempest - what tests should be in Tempest vs
>>> in the individual projects ?
>>> It sounds like the scope of tempest is to make sure that,
>>> a. Projects are functionally tested (AND)
>>> b. Openstack components (a.k.a projects) do not have integration issues.
>>> Assuming my understanding is correct, does it make sense to have the
>>> project specific functional tests in Tempest?
>>> Troubleshooting failures related to project specific functionality
>>> requires deep understanding of the individual projects.
>>> Isn't it better to leave it to the individual projects to make sure
>>> that they are functional?
>>> That will help the contributors to Tempest spend their time on what
>>> only Tempest can do -i.e. identify integration issues.
>> What do you mean by project specific functional testing? What makes
>> a marconi failure in a tempest gate job any more involved than debugging a
>> nova or neutron failure? Part of the point of having an integrated gate is
>> saying that the project works well with all the others in OpenStack. IMO
>> not just in project functionality but also in community. When there is an
>> with a gate job everyone comes together to work on it. For example if you
>> a keystone patch that breaks a marconi test in check there is open
>> about what happened and how to fix it.
>> That being said there are certain cases where having a project specific
>> functional test makes sense. For example swift has a functional test job
>> starts swift in devstack. But, those things are normally handled on a per
>> basis. In general if the project is meant to be part of the larger
>> ecosystem then Tempest is the place to put functional testing. That way
>> you know
>> it works with all of the other components. The thing is in openstack what
>> like a project isolated functional test almost always involves another
>> in real use cases. (for example keystone auth with api requests)
>> Now for the boundary between what kind of test belongs where isn't always
>> and every project has a different rule of thumb on that. There really
>> isn't a
>> hard and fast rule for tempest on what's allowed as long as it fits
>> within the
>> criteria for the different test categories. It's handled on a per case
>> Honestly, though I normally recommend putting as much in tempest as you
>> because a big advantage of having an external test suite is that it keeps
>> honest because the tests are in a separate repo.
>> -Matt Treinish
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
Samsung Research America
sean at dague.net / sean.dague at samsung.com
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 482 bytes
Desc: OpenPGP digital signature
More information about the OpenStack-dev