<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi,<br>
<br>
Let me express my concerns on this topic:<br>
<blockquote type="cite">
<pre wrap="">With some recent changes made to Tempest compatibility with
nosetests is going away.</pre>
</blockquote>
I think that we should not drop nosetests support from tempest or
any other project. The problem with testrepository is that it's
not providing any debugger support at all (and will never
provide). It also has some issues with proving error traces in
human readable form and it can be quite hard to find out what is
actually broken.<br>
<br>
Because of this I think we should try to avoid any kind of test
libraries that break compatibility with conventional test runners.
<br>
<br>
Our tests should be able to run correctly with nosetests,
teststools or plain old unittest runner. If for some reason test
libraries (like testscenarios) doesn't provide support for this we
should fix this libraries or avoid their usage.<br>
<br>
Regards,<br>
Alexei Kornienko<br>
<br>
On 02/27/2014 06:36 PM, Frittoli, Andrea (HP Cloud) wrote:<br>
</div>
<blockquote
cite="mid:A36B1DF8CF6E574F9E151E2766C8C361489A1FB5@G4W3298.americas.hpqcorp.net"
type="cite">
<pre wrap="">This is another example of achieving the same result (exclusion from a
list):
<a class="moz-txt-link-freetext" href="https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element">https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element</a>
s/tempest/tests2skip.py
<a class="moz-txt-link-freetext" href="https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element">https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element</a>
s/tempest/tests2skip.txt
andrea
-----Original Message-----
From: Matthew Treinish [<a class="moz-txt-link-freetext" href="mailto:mtreinish@kortar.org">mailto:mtreinish@kortar.org</a>]
Sent: 27 February 2014 15:49
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [QA] The future of nosetests with Tempest
On Tue, Feb 25, 2014 at 07:46:23PM -0600, Matt Riedemann wrote:
</pre>
<blockquote type="cite">
<pre wrap="">
On 2/12/2014 1:57 PM, Matthew Treinish wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:
</pre>
<blockquote type="cite">
<pre wrap="">
On 1/17/2014 8:34 AM, Matthew Treinish wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On 01/16/2014 10:56 PM, Matthew Treinish wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi everyone,
With some recent changes made to Tempest compatibility with
nosetests is going away. We've started using newer features that
nose just doesn't support. One example of this is that we've
started using testscenarios and we're planning to do this in more
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">places moving forward.
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
So at Icehouse-3 I'm planning to push the patch out to remove
nosetests from the requirements list and all the workarounds and
references to nose will be pulled out of the tree. Tempest will
also start raising an unsupported exception when you try to run
it with nose so that there isn't any confusion on this moving
forward. We talked about doing this at summit briefly and I've
brought it up a couple of times before, but I believe it is time
to do this now. I feel for tempest to move forward we need to do this
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">now so that there isn't any ambiguity as we add even more features and new
types of testing.
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">I'm with you up to here.
</pre>
<blockquote type="cite">
<pre wrap="">
Now, this will have implications for people running tempest with
python 2.6 since up until now we've set nosetests. There is a
workaround for getting tempest to run with python 2.6 and testr see:
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/59007/1/README.rst">https://review.openstack.org/#/c/59007/1/README.rst</a>
but essentially this means that when nose is marked as
unsupported on tempest python 2.6 will also be unsupported by
Tempest. (which honestly it basically has been for while now just
we've gone without making it official)
</pre>
</blockquote>
<pre wrap="">The way we handle different runners/os can be categorized as
"tested in gate", "unsupported" (should work, possibly some hacks
needed), and "hostile". At present, both nose and py2.6 I would
say are in the unsupported category. The title of this message and
the content up to here says we are moving nose to the hostile
category. With only 2 months to feature freeze I see no
justification in moving
py2.6 to the hostile category. I don't see what new testing
features scheduled for the next two months will be enabled by
saying that tempest cannot and will not run on 2.6. It has been
agreed I think by all projects that py2.6 will be dropped in J. It
is OK that py2.6 will require some hacks to work and if in the
next few months it needs a few more then that is ok. If I am
missing another connection between the py2.6 and nose issues, please
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">explain.
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
So honestly we're already at this point in tempest. Nose really
just doesn't work with tempest, and we're adding more features to
tempest, your negative test generator being one of them, that
interfere further with nose. I've seen several
</pre>
</blockquote>
<pre wrap="">
I disagree here, my team is running Tempest API, CLI and scenario
tests every day with nose on RHEL 6 with minimal issues. I had to
workaround the negative test discovery by simply sed'ing that out of
the tests before running it, but that's acceptable to me until we
can start testing on RHEL 7. Otherwise I'm completely OK with
saying py26 isn't really supported and isn't used in the gate, and
it's a buyer beware situation to make it work, which includes
pushing up trivial patches to make it work (which I did a few of
last week, and they were small syntax changes or usages of
testtools).
I don't understand how the core projects can be running unit tests
in the gate on py26 but our functional integration project is going
to actively go out and make it harder to run Tempest with py26, that
sucks.
If we really want to move the test project away from py26, let's
make the concerted effort to get the core projects to move with it.
</pre>
</blockquote>
<pre wrap="">
So as I said before the python 2.6 story for tempest remains the same
after this change. The only thing that we'll be doing is actively
preventing nose from working with tempest.
</pre>
<blockquote type="cite">
<pre wrap="">
And FWIW, I tried the discover.py patch with unittest2 and
testscenarios last week and either I botched it, it's not documented
properly on how to apply it, or I screwed something up, but it
didn't work for me, so I'm not convinced that's the workaround.
What's the other option for running Tempest on py26 (keeping RHEL 6
in mind)? Using tox with testr and pip? I'm doing this all
single-node.
</pre>
</blockquote>
<pre wrap="">
Yes, that is what the discover patch is used to enable. By disabling
nose the only path to run tempest with py2.6 is to use testr. (which
is what it always should have been)
Attila confirmed it was working here:
<a class="moz-txt-link-freetext" href="http://fpaste.org/76651/32143139/">http://fpaste.org/76651/32143139/</a>
in that example he applies 2 patches the second one is currently in
the gate for tempest. (<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/72388/">https://review.openstack.org/#/c/72388/</a> ) So
all that needs to be done is to apply that discover patch:
<a class="moz-txt-link-freetext" href="https://code.google.com/p/unittest-ext/issues/detail?id=79">https://code.google.com/p/unittest-ext/issues/detail?id=79</a>
(which I linked to before)
Then tempest should run more or less the same between 2.7 and 2.6.
(The only difference I've seen is in how skips are handled)
</pre>
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">patches this cycle that attempted to introduce incorrect behavior
while trying to fix compatibility with nose. That's why I think we
need a clear message on this sooner than later. Which is why I'm
proposing actively raising an error when things are run with nose
upfront so there isn't any illusion that things are expected to work.
This doesn't necessarily mean we're moving python 2.6 to the hostile
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">category.
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">Nose support is independent of python 2.6 support. Py26 I would
still consider to be unsupported, the issue is that the hack to
make py26 work is outside of tempest. This is why we've recommended
that people using python 2.6 run with nose, which really is no
longer an option. Attila's abandoned patch that I linked above
documents points to this bug with a patch to discover which is need to
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">get python 2.6 working with tempest and testr:
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
<a class="moz-txt-link-freetext" href="https://code.google.com/p/unittest-ext/issues/detail?id=79">https://code.google.com/p/unittest-ext/issues/detail?id=79</a>
</pre>
</blockquote>
</blockquote>
<pre wrap="">
_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<pre wrap="">
One question I had was is there an easy way to setup a config file to
specify the test bucket and what can be excluded, like you can with
nose.cfg and nose? We used that for filtering out API tests that
didn't work with the PowerVM driver in Nova but I'm not aware of
something similar with testr.
</pre>
</blockquote>
<pre wrap="">
So I'm not sure if it's exactly what you're looking for but testr has the
--load-list option which you can use to specify a file which lists the tests
you want to run. I don't think there is a method to exclude tests besides
using a regex filter right now. There is a bug open about this:
<a class="moz-txt-link-freetext" href="https://bugs.launchpad.net/testrepository/+bug/1208610">https://bugs.launchpad.net/testrepository/+bug/1208610</a>
So I can see you doing this 2 ways either writing a little script that will
generate a list file by doing something like:
1. testr list-tests > file
2. remove excludes from file
3. testr run --load-list file
or making a long unwieldy regex that excludes the tests you need to.
Something like what I did here:
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/51275/4/tox.ini">https://review.openstack.org/#/c/51275/4/tox.ini</a>
-Matt Treinish
_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>