<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div><div>On Aug 11, 2014, at 12:00 PM, David Kranz <<a href="mailto:dkranz@redhat.com">dkranz@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
<div bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 08/06/2014 05:48 PM, John Griffith
wrote:<br>
</div>
<blockquote cite="mid:CA+qL3LWK5vHzc_HiFZXnDu+q4W-B1+pNVsxPsjNW+fv+dehViA@mail.gmail.com" type="cite">
<div dir="ltr">
<div class="gmail_default" style="font-family:courier
new,monospace">I have to agree with Duncan here. I also don't
know if I fully understand the limit in options. Stress test
seems like it could/should be different (again overlap isn't a
horrible thing) and I don't see it as siphoning off resources
so not sure of the issue. We've become quite wrapped up in
projects, programs and the like lately and it seems to hinder
forward progress more than anything else.</div>
<div class="gmail_default" style="font-family:courier
new,monospace"><br>
</div>
<div class="gmail_default" style="font-family:courier
new,monospace">I'm also not convinced that Tempest is where
all things belong, in fact I've been thinking more and more
that a good bit of what Tempest does today should fall more on
the responsibility of the projects themselves. For example
functional testing of features etc, ideally I'd love to have
more of that fall on the projects and their respective teams.
That might even be something as simple to start as saying "if
you contribute a new feature, you have to also provide a link
to a contribution to the Tempest test-suite that checks it".
Sort of like we do for unit tests, cross-project tracking is
difficult of course, but it's a start. The other idea is
maybe functional test harnesses live in their respective
projects.<br>
</div>
<div class="gmail_default" style="font-family:courier
new,monospace"><br>
</div>
<div class="gmail_default" style="font-family:courier
new,monospace">Honestly I think who better to write tests for
a project than the folks building and contributing to the
project. At some point IMO the QA team isn't going to scale.
I wonder if maybe we should be thinking about proposals for
delineating responsibility and goals in terms of functional
testing?</div>
<div class="gmail_default" style="font-family:courier
new,monospace"><br>
</div>
<div class="gmail_default" style="font-family:courier
new,monospace"><br>
</div>
</div>
<div class="gmail_extra"><br>
</div>
</blockquote>
All good points. Your last paragraph was discussed by the QA team
leading up to and at the Atlanta summit. The conclusion was that the
api/functional tests focused on a single project should be part of
that project. As Sean said, we can envision there being half (or
some other much smaller number) as many such tests in tempest going
forward.<br>
<br>
Details are under discussion, but the way this is likely to play out
is that individual projects will start by creating their own
functional tests outside of tempest. Swift already does this and
neutron seems to be moving in that direction. There is a spec to
break out parts of tempest
(<a class="moz-txt-link-freetext" href="https://github.com/openstack/qa-specs/blob/master/specs/tempest-library.rst">https://github.com/openstack/qa-specs/blob/master/specs/tempest-library.rst</a>)
into a library that might be used by projects implementing
functional tests. <br>
<br>
Once a project has "sufficient" functional testing, we can consider
removing its api tests from tempest. This is a bit tricky because
tempest needs to cover *all* cross-project interactions. In this
respect, there is no clear line in tempest between scenario tests
which have this goal explicitly, and api tests which may also
involve interactions that might not be covered in a scenario. So we
will need a principled way to make sure there is complete
cross-project coverage in tempest with a smaller number of api
tests. </div></blockquote><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000">
<br>
-David<br></div></blockquote><div><br></div><div>We need to be careful about dumping the tests from tempest now that the DefCore group is relying on them as well. Tempest is no longer just a developer/QA/operations tool. It’s also being used as the basis of a trademark enforcement tool. That’s not to say we can’t change the test suite, but we have to consider a new angle when doing so.</div><div><br></div><div>Doug</div><br><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000">
</div>
_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br></blockquote></div><br></body></html>