<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>
<div>Hello all,</div>
<div><br>
</div>
<div>I have been working on adding tests in Tempest for Marconi, for the last few months.</div>
<div>While there are many amazing people to work with, the process has been more difficult than I expected.</div>
<div><br>
</div>
<div>Couple of pain-points and suggestions to make the process easier for myself & future contributors.</div>
<div><br>
</div>
<div>1. The QA requirements for a project to graduate needs details beyond the "Project must have a *basic* devstack-gate job set up"</div>
<div>2. The scope of Tempest needs clarification - what tests should be in Tempest vs. in the individual projects? Or should they be in both tempest and the project?</div>
<div><br>
</div>
<div>See details below.</div>
<div><br>
</div>
<div>1. There is little documentation on graduation requirement from a QA perspective beyond 'Project must have a basic devstack-gate job set up'.</div>
<div><br>
</div>
<div>As a result, I hear different interpretations on what a basic devstack gate job is.</div>
<div>This topic was discussed in one of the QA meetings a few weeks back [1].</div>
<div>Based on the discussion there, having a basic job - such as one that will let us know 'if a keystone change broke marconi' was good enough.</div>
<div>My efforts in getting Marconi meet graduation requirements w.r.t Tempest was based on the above discussion.</div>
<div><br>
</div>
<div>However, my conversations with the TC during Marconi's graduation review lead me to believe that these requirements aren't yet formalized.</div>
<div>We were told that we needed to have more test coverage in tempest, & having them elsewhere (i.e. functional tests in the Marconi project itself) was not good enough.</div>
<div><br>
</div>
<div>I will never debate the value of having good test coverage - after all I define myself professionally as a QA ;)</div>
<div>I am proud of the unit and functional test suites & the test coverage we have in Marconi [2]. </div>
<div>Marconi team is continuing its efforts in this direction. </div>
<div>We are looking forward to adding more tests in Tempest and making sure Marconi is in par with the community standards. </div>
<div><br>
</div>
<div>But what frustrates me is that the test requirements seem to evolve, catching new contributors by surprise.</div>
<div><br>
</div>
<div>It will really help to have these requirements documented in detail - answering at least the following questions,</div>
<div>a. What tests are needed to graduate - API, Scenario, CLI?</div>
<div>b. How much coverage is good enough to graduate?</div>
<div><br>
</div>
<div>That will make sure that contributors focus their time & energy in the right direction.</div>
<div>I am willing to lead the effort to document the QA-level graduation requirements for a project and help solidify them.</div>
<div><br>
</div>
<div>2. Clarify the scope of Tempest - what tests should be in Tempest vs in the individual projects ?</div>
<div><br>
</div>
<div>It sounds like the scope of tempest is to make sure that,</div>
<div>a. Projects are functionally tested (AND)</div>
<div>b. Openstack components (a.k.a projects) do not have integration issues.</div>
<div><br>
</div>
<div>Assuming my understanding is correct, does it make sense to have the project specific functional tests in Tempest?</div>
<div>Troubleshooting failures related to project specific functionality requires deep understanding of the individual projects.</div>
<div>Isn't it better to leave it to the individual projects to make sure that they are functional?</div>
<div>That will help the contributors to Tempest spend their time on what only Tempest can do -i.e. identify integration issues. </div>
<div><br>
</div>
<div><br>
</div>
<div>My hope is that we can clearly outline these requirements as soon as possible.</div>
<div>Looking forward to hearing your thoughts!!</div>
<div><br>
</div>
<div>Best Regards,</div>
<div>Malini</div>
<div><br>
</div>
<div>[1] http://eavesdrop.openstack.org/meetings/qa/2014/qa.2014-02-13-17.00.log.html (See discussion starting with timestamp 17:51:57)</div>
<div>[2] https://github.com/openstack/marconi/tree/master/tests</div>
</div>
<div><br>
</div>
</body>
</html>