<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Changing subject line to continue
thread about new $subj<br>
<br>
On 08/12/2014 08:56 AM, Doug Hellmann wrote:<br>
</div>
<blockquote
cite="mid:497AB843-2A4E-45D0-AC93-73926FD6CBE2@doughellmann.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<br>
<div>
<div>On Aug 11, 2014, at 12:00 PM, David Kranz <<a
moz-do-not-send="true" href="mailto:dkranz@redhat.com">dkranz@redhat.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 08/06/2014 05:48 PM, John
Griffith wrote:<br>
</div>
<blockquote
cite="mid:CA+qL3LWK5vHzc_HiFZXnDu+q4W-B1+pNVsxPsjNW+fv+dehViA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_default" style="font-family:courier
new,monospace">I have to agree with Duncan here. I
also don't know if I fully understand the limit in
options. Stress test seems like it could/should be
different (again overlap isn't a horrible thing) and I
don't see it as siphoning off resources so not sure of
the issue. We've become quite wrapped up in projects,
programs and the like lately and it seems to hinder
forward progress more than anything else.</div>
<div class="gmail_default" style="font-family:courier
new,monospace"><br>
</div>
<div class="gmail_default" style="font-family:courier
new,monospace">I'm also not convinced that Tempest is
where all things belong, in fact I've been thinking
more and more that a good bit of what Tempest does
today should fall more on the responsibility of the
projects themselves. For example functional testing
of features etc, ideally I'd love to have more of that
fall on the projects and their respective teams. That
might even be something as simple to start as saying
"if you contribute a new feature, you have to also
provide a link to a contribution to the Tempest
test-suite that checks it". Sort of like we do for
unit tests, cross-project tracking is difficult of
course, but it's a start. The other idea is maybe
functional test harnesses live in their respective
projects.<br>
</div>
<div class="gmail_default" style="font-family:courier
new,monospace"><br>
</div>
<div class="gmail_default" style="font-family:courier
new,monospace">Honestly I think who better to write
tests for a project than the folks building and
contributing to the project. At some point IMO the QA
team isn't going to scale. I wonder if maybe we
should be thinking about proposals for delineating
responsibility and goals in terms of functional
testing?</div>
<div class="gmail_default" style="font-family:courier
new,monospace"><br>
</div>
<div class="gmail_default" style="font-family:courier
new,monospace"><br>
</div>
</div>
<div class="gmail_extra"><br>
</div>
</blockquote>
All good points. Your last paragraph was discussed by the QA
team leading up to and at the Atlanta summit. The conclusion
was that the api/functional tests focused on a single
project should be part of that project. As Sean said, we can
envision there being half (or some other much smaller
number) as many such tests in tempest going forward.<br>
<br>
Details are under discussion, but the way this is likely to
play out is that individual projects will start by creating
their own functional tests outside of tempest. Swift already
does this and neutron seems to be moving in that direction.
There is a spec to break out parts of tempest (<a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://github.com/openstack/qa-specs/blob/master/specs/tempest-library.rst">https://github.com/openstack/qa-specs/blob/master/specs/tempest-library.rst</a>)
into a library that might be used by projects implementing
functional tests. <br>
<br>
Once a project has "sufficient" functional testing, we can
consider removing its api tests from tempest. This is a bit
tricky because tempest needs to cover *all* cross-project
interactions. In this respect, there is no clear line in
tempest between scenario tests which have this goal
explicitly, and api tests which may also involve
interactions that might not be covered in a scenario. So we
will need a principled way to make sure there is complete
cross-project coverage in tempest with a smaller number of
api tests. </div>
</blockquote>
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000"> <br>
-David<br>
</div>
</blockquote>
<div><br>
</div>
<div>We need to be careful about dumping the tests from tempest
now that the DefCore group is relying on them as well. Tempest
is no longer just a developer/QA/operations tool. It’s also
being used as the basis of a trademark enforcement tool.
That’s not to say we can’t change the test suite, but we have
to consider a new angle when doing so.</div>
<div><br>
</div>
<div>Doug</div>
</div>
</blockquote>
Thanks, Doug. We need to get away from "acceptance test == tempest"
while retaining the ability to define and run an acceptance test as
easily as tempest can be run now. My view is that functional tests
in projects should have the capability to be run against real
clouds, and that in-project functional tests should "look like", and
be interchangeable with, the api tests in tempest. The in-project
tests would be focused on completeness of api testing and the
tempest tests focused on cross-project interaction, but they could
be run in similar ways. Then a trademark enforcement tool, or any
other kind of acceptance test, could select which tests to run. I
think this view may be a bit controversial but your point obviously
needs to be addressed. <br>
<br>
<br>
-David<br>
<blockquote
cite="mid:497AB843-2A4E-45D0-AC93-73926FD6CBE2@doughellmann.com"
type="cite">
<div><br>
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000"> </div>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
</div>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>