<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 10/30/2014 10:02 AM, David Kranz
wrote:<br>
</div>
<blockquote cite="mid:54524505.6050800@redhat.com" type="cite">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<div class="moz-cite-prefix">On 10/30/2014 09:52 AM, Sean Dague
wrote:<br>
</div>
<blockquote cite="mid:54524280.80504@dague.net" type="cite">
<meta http-equiv="Context-Type" content="text/html; ">
<div class="moz-cite-prefix">On 10/30/2014 09:33 AM, David Kranz
wrote:<br>
</div>
<blockquote cite="mid:54523E20.7070902@redhat.com" type="cite">
<div class="moz-cite-prefix">On 10/30/2014 07:49 AM, Sean
Dague wrote:<br>
</div>
<blockquote cite="mid:545225BA.2060601@dague.net" type="cite">
<div class="moz-cite-prefix">On 10/29/2014 12:30 PM, Matthew
Treinish wrote:<br>
</div>
<blockquote cite="mid:20141029163018.GA2156@Sazabi.treinish"
type="cite">
<pre wrap="">Hi everyone,
Before we start the larger discussion at summit next week about the future of
testing in OpenStack - specifically about spinning up functional testing and how
it relates to tempest - I would like to share some of my thoughts on how we can
get things started and how I think they'll eventually come together.
Currently in tempest we have a large number of tests (mostly api-focused)
which are probably a better fit for a project's functional test suite. The best
example I can think of is the nova flavors tests. Validation of flavor
manipulation doesn't need to run in the integrated test suite on every commit to
every project because it only requires Nova. A simple win for initiating in-tree
functional testing would be to move these kinds of tests into the projects and
run the tests from the project repos instead of from tempest.</pre>
</blockquote>
I think a lot of the negative API testing is also a great
thing to be done back at the project level. All of that
testing should be able to work without a full OpenStack, as
it should be caught and managed by the API service and never
get any further than that.<br>
<br>
<blockquote cite="mid:20141029163018.GA2156@Sazabi.treinish"
type="cite">
<pre wrap="">This would have the advantage of making tempest slimmer for every project
and begin the process of getting projects to take responsibility for their
functional testing rather than relying on tempest. As tests are moved tempest
can start to become the integration test suite it was meant to be. It would
retain only tests that involve multiple projects and stop being the OpenStack
black box testing suite. I think that this is the right direction for tempest
moving forward, especially as we move to having project-specific functional
testing.
Doing this migration is dependent on some refactors in tempest and moving
the required bits to tempest-lib so they can be easily consumed by the
other projects. This will be discussed at summit, is being planned
for implementation this cycle, and is similar to what is currently in progress
for the cli tests.
The only reason this testing existed in tempest in the first place was as
mechanism to block and then add friction against breaking api changes. Tempest's
api testing has been been pretty successful at achieving these goals. We'll want
to ensure that migrated tests retain these characteristics. If we are using
clients from tempest-lib we should get this automatically since to break
the api you'd have to change the api client. Another option proposed was to
introduce a hacking rule that would block changes to api tests at the same time
other code was being changed.
There is also a concern for external consumers of tempest if we move the tests
out of the tempest tree (I'm thinking refstack). I think the solution is
to maintain a load_tests discovery method inside of tempest or elsewhere that
will run the appropriate tests from the other repos for something like refstack.
Assuming that things are built in a compatible way using the same framework then
running the tests from separate repos should be a simple matter of pointing the
test runner in the right direction.</pre>
</blockquote>
I think we can see where this takes us. I'm still skeptical
of cross project loading of tests because it's often quite
fragile. However, if you look at what refstack did they had
a giant evaluation of all of tempest and pruned a bunch of
stuff out. I would imagine maybe there is a conversation
there about tests that refstack feels are important to stay
in Tempest for their validation reasons. I think having a
few paths that are tested both in Tempest and in project
functional tests is not a bad thing.<br>
</blockquote>
Refstack is not the only thing that cares about validation of
real clouds. As we move forward with this, it would be good to
separate the issues of "in which repo does a functional test
live" and "can a functional test be run against a real cloud".
IMO, over use of mocking (broadly defined) in functional tests
should be avoided unless it is configurable to also work in an
unmocked fashion. Whether the way to combine all of the
functional tests is by cross project loading of tests or by
some other means is more of an implementation detail.<br>
</blockquote>
Part of the perspective I'm bringing in is actually knowing what
to do when your tests fail. Using Tempest against real clouds is
great, people should keep doing that. But if you are rolling out
a real cloud yourself, in the future you should be running the
functional tests in staging to ensure you are functioning. Those
will also provide you, hopefully, with a better path to
understand what's wrong.<br>
</blockquote>
Sean, sorry if I was unclear. By "real clouds", I just meant the
tests should be able to use OpenStack apis with no mocking.<br>
</blockquote>
Ok, so part of this remains to be seen about what the biggest bang
for the buck is. The class of bugs I feel like we need to nail in
Nova right now are going to require tests that bring up pieces of
the wsgi stack, but are probably not runable on a real deploy.
Again, this is about debugability.<br>
<br>
But let's give it some time to breath and figure out what's useful.
I think there is a lot of theory and philosophy in this discussion,
which while good, might need to take a bit of a back seat to the
practicalities of nailing bugs. I expect what we're going to see in
K summit is a primer to get some ideas moving forward, and in L
we'll begin to be able to collect best practices.<br>
<br>
<blockquote cite="mid:54524505.6050800@redhat.com" type="cite"> <br>
-David<br>
<br>
<br>
<br>
<blockquote cite="mid:54524280.80504@dague.net" type="cite"> <br>
This will mean that as an arbitrary 3rd party accessing a public
cloud, you don't have a test suite that pushes every button of
the cloud. But I think that's an acceptable trade off. Because
pushing on odd edge cases might produce a fail, but if there is
no clear path from the fail to a fix, it doesn't help anyone. <br>
<br>
Finding failures in OpenStack is not hard. Finding failures in
OpenStack in a way that points towards resolution is what we
need to be striving for.<br>
<br>
<blockquote cite="mid:54523E20.7070902@redhat.com" type="cite">
<blockquote cite="mid:545225BA.2060601@dague.net" type="cite">
<br>
But I think that's an end of cycle at best discussion.<br>
<br>
Also, there probably need to be a few discussions anyway of
refstack/tempest/defcore. The fact that Keystone was dropped
from defcore because there were no non admin Keystone tests
explicitly in Tempest (even though we make over 5000
keystone non admin API calls over a tempest run) was very
odd. That is something that could have been fixed in a day.<br>
<br>
<blockquote cite="mid:20141029163018.GA2156@Sazabi.treinish"
type="cite">
<pre wrap="">I also want to comment on the role of functional testing. What I've proposed
here is only one piece of what project specific functional testing should be
and just what I feel is a good/easy start. I don't feel that this should be
the only testing done in the projects. I'm suggesting this as a first
step because the tests already exist and it should be a relatively simple task.
I also feel that using tempest-lib like this shouldn't be a hard requirement.
Ideally the client definitions shouldn't have to live externally, or if they did
they would be the official clients, but I am suggesting this as a first step to
start a migration out of tempest.
I don't want anyone to feel that they need block their functional testing
efforts until tempest-lib becomes more useable. The larger value from functional
testing is actually in enabling testing more tightly coupled to the projects
(e.g. whitebox testing). I feel that any work necessary to enable functional
testing should be prioritized.</pre>
</blockquote>
Agreed. From a Nova perspective I'm going to be layering up
some of this functionality with existing in tree tools. We
also have a bunch of potentially needed functional tests
based on what I found in bug triage this fall. So the
priority is going to be getting something working right to
nail the bugs in Nova first, then align where it makes
sense.<br>
</blockquote>
I'm not sure enabling white-box testing is the whole story.
Going forward, with project gates spending more time on
relevant tests and more tests running post-merge, there are
plenty of valuable black-box tests that we could have.
Whitebox testing would be a valuable addition of course.<br>
<blockquote cite="mid:545225BA.2060601@dague.net" type="cite">
<br>
I actually think the negative test generator code in tempest
would be a good candidate for tempest lib, and would
actually work better with those negative definitions in the
project as functional tests.<br>
</blockquote>
++<br>
<br>
I think a main reason more negative tests have not been
created using the generator is that it requires deep knowledge
of the schemas. Ideally, these schemas would be the output of
some generator in each project, or provided by the project
manually.<br>
<br>
-David<br>
<blockquote cite="mid:545225BA.2060601@dague.net" type="cite">
<br>
-Sean<br>
<br>
<blockquote cite="mid:20141029163018.GA2156@Sazabi.treinish"
type="cite">
<pre wrap="">-Matt Treinish
</pre>
</blockquote>
<blockquote cite="mid:20141029163018.GA2156@Sazabi.treinish"
type="cite"><br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Sean Dague
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://dague.net">http://dague.net</a>
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Sean Dague
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://dague.net">http://dague.net</a>
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Sean Dague
<a class="moz-txt-link-freetext" href="http://dague.net">http://dague.net</a>
</pre>
</body>
</html>