Open Stack

Hello all,

I have been working on adding tests in Tempest for Marconi, for the last few months.
While there are many amazing people to work with, the process has been more difficult than I expected.

Couple of pain-points and suggestions to make the process easier for myself & future contributors.

1. The QA requirements for a project to graduate needs details beyond the "Project must have a *basic* devstack-gate job set up"
2. The scope of Tempest needs clarification  - what tests should be in Tempest vs. in the individual projects? Or should they be in both tempest and the project?

See details below.

1. There is little documentation on graduation requirement from a QA perspective beyond 'Project must have a basic devstack-gate job set up'.

As a result, I hear different interpretations on what a basic devstack gate job is.
This topic was discussed in one of the QA meetings a few weeks back [1].
Based on the discussion there, having a basic job - such as one that will let us know 'if a keystone change broke marconi' was  good enough.
My efforts in getting Marconi meet graduation requirements w.r.t Tempest was based on the above discussion.

However, my conversations with the TC during Marconi's graduation review  lead me to believe that these requirements aren't yet formalized.
We were told that we needed to have more test coverage in tempest, & having them elsewhere (i.e. functional tests in the Marconi project itself) was not good enough.

I will never debate the value of having good test coverage - after all I define myself professionally as a QA ;)
I am proud of the unit and functional test suites & the test coverage we have in Marconi [2].
Marconi team is continuing its efforts in this direction.
We are looking forward to adding more tests in Tempest and making sure Marconi is in par with the community standards.

But what frustrates me is that the test requirements seem to evolve, catching  new contributors by surprise.

It will really help to have these requirements documented in detail - answering at least the following questions,
a. What tests are needed to graduate - API, Scenario, CLI?
b. How much coverage is good enough to graduate?

That will make sure that contributors focus their time & energy in the right direction.
I am willing to lead the effort to document the QA-level graduation requirements for a project and help solidify them.

2. Clarify the scope of Tempest  - what tests should be in Tempest vs in the individual projects ?

It sounds like the scope of tempest is to make sure that,
a. Projects are functionally tested (AND)
b. Openstack components (a.k.a projects) do not have integration issues.

Assuming my understanding is correct, does it make sense to have the project specific functional tests in Tempest?
Troubleshooting failures related to project specific functionality requires deep understanding of the individual projects.
Isn't it better to leave it to the individual projects to make sure that they are functional?
That will help the contributors to Tempest spend their time on what only Tempest can do -i.e. identify integration issues.


My hope is that we can clearly outline these requirements as soon as possible.
Looking forward to hearing your thoughts!!

Best Regards,
Malini

[1] http://eavesdrop.openstack.org/meetings/qa/2014/qa.2014-02-13-17.00.log.html   (See discussion starting with timestamp 17:51:57)
[2] https://github.com/openstack/marconi/tree/master/tests

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140320/c8422d7a/attachment.html>