Open Stack

This is another example of achieving the same result (exclusion from a
list):
https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element
s/tempest/tests2skip.py
https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element
s/tempest/tests2skip.txt

andrea

-----Original Message-----
From: Matthew Treinish [mailto:mtreinish at kortar.org] 
Sent: 27 February 2014 15:49
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [QA] The future of nosetests with Tempest

On Tue, Feb 25, 2014 at 07:46:23PM -0600, Matt Riedemann wrote:
> 
> 
> On 2/12/2014 1:57 PM, Matthew Treinish wrote:
> >On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:
> >>
> >>
> >>On 1/17/2014 8:34 AM, Matthew Treinish wrote:
> >>>On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote:
> >>>>On 01/16/2014 10:56 PM, Matthew Treinish wrote:
> >>>>>Hi everyone,
> >>>>>
> >>>>>With some recent changes made to Tempest compatibility with 
> >>>>>nosetests is going away. We've started using newer features that 
> >>>>>nose just doesn't support. One example of this is that we've 
> >>>>>started using testscenarios and we're planning to do this in more
places moving forward.
> >>>>>
> >>>>>So at Icehouse-3 I'm planning to push the patch out to remove 
> >>>>>nosetests from the requirements list and all the workarounds and 
> >>>>>references to nose will be pulled out of the tree. Tempest will 
> >>>>>also start raising an unsupported exception when you try to run 
> >>>>>it with nose so that there isn't any confusion on this moving 
> >>>>>forward. We talked about doing this at summit briefly and I've 
> >>>>>brought it up a couple of times before, but I believe it is time 
> >>>>>to do this now. I feel for tempest to move forward we need to do this
now so that there isn't any ambiguity as we add even more features and new
types of testing.
> >>>>I'm with you up to here.
> >>>>>
> >>>>>Now, this will have implications for people running tempest with 
> >>>>>python 2.6 since up until now we've set nosetests. There is a 
> >>>>>workaround for getting tempest to run with python 2.6 and testr see:
> >>>>>
> >>>>>https://review.openstack.org/#/c/59007/1/README.rst
> >>>>>
> >>>>>but essentially this means that when nose is marked as 
> >>>>>unsupported on tempest python 2.6 will also be unsupported by 
> >>>>>Tempest. (which honestly it basically has been for while now just 
> >>>>>we've gone without making it official)
> >>>>The way we handle different runners/os can be categorized as 
> >>>>"tested in gate", "unsupported" (should work, possibly some hacks 
> >>>>needed), and "hostile". At present, both nose and py2.6 I would 
> >>>>say are in the unsupported category. The title of this message and 
> >>>>the content up to here says we are moving nose to the hostile 
> >>>>category. With only 2 months to feature freeze I see no 
> >>>>justification in moving
> >>>>py2.6 to the hostile category. I don't see what new testing 
> >>>>features scheduled for the next two months will be enabled by 
> >>>>saying that tempest cannot and will not run on 2.6. It has been 
> >>>>agreed I think by all projects that py2.6 will be dropped in J. It 
> >>>>is OK that py2.6 will require some hacks to work and if in the 
> >>>>next few months it needs a few more then that is ok. If I am 
> >>>>missing another connection between the py2.6 and nose issues, please
explain.
> >>>>
> >>>
> >>>So honestly we're already at this point in tempest. Nose really 
> >>>just doesn't work with tempest, and we're adding more features to 
> >>>tempest, your negative test generator being one of them, that 
> >>>interfere further with nose. I've seen several
> >>
> >>I disagree here, my team is running Tempest API, CLI and scenario 
> >>tests every day with nose on RHEL 6 with minimal issues.  I had to 
> >>workaround the negative test discovery by simply sed'ing that out of 
> >>the tests before running it, but that's acceptable to me until we 
> >>can start testing on RHEL 7.  Otherwise I'm completely OK with 
> >>saying py26 isn't really supported and isn't used in the gate, and 
> >>it's a buyer beware situation to make it work, which includes 
> >>pushing up trivial patches to make it work (which I did a few of 
> >>last week, and they were small syntax changes or usages of 
> >>testtools).
> >>
> >>I don't understand how the core projects can be running unit tests 
> >>in the gate on py26 but our functional integration project is going 
> >>to actively go out and make it harder to run Tempest with py26, that 
> >>sucks.
> >>
> >>If we really want to move the test project away from py26, let's 
> >>make the concerted effort to get the core projects to move with it.
> >
> >So as I said before the python 2.6 story for tempest remains the same 
> >after this change. The only thing that we'll be doing is actively 
> >preventing nose from working with tempest.
> >
> >>
> >>And FWIW, I tried the discover.py patch with unittest2 and 
> >>testscenarios last week and either I botched it, it's not documented 
> >>properly on how to apply it, or I screwed something up, but it 
> >>didn't work for me, so I'm not convinced that's the workaround.
> >>
> >>What's the other option for running Tempest on py26 (keeping RHEL 6 
> >>in mind)?  Using tox with testr and pip?  I'm doing this all 
> >>single-node.
> >
> >Yes, that is what the discover patch is used to enable. By disabling 
> >nose the only path to run tempest with py2.6 is to use testr. (which 
> >is what it always should have been)
> >
> >Attila confirmed it was working here:
> >http://fpaste.org/76651/32143139/
> >in that example he applies 2 patches the second one is currently in 
> >the gate for tempest. (https://review.openstack.org/#/c/72388/ ) So 
> >all that needs to be done is to apply that discover patch:
> >
> >https://code.google.com/p/unittest-ext/issues/detail?id=79
> >
> >(which I linked to before)
> >
> >Then tempest should run more or less the same between 2.7 and 2.6. 
> >(The only difference I've seen is in how skips are handled)
> >
> >>
> >>>patches this cycle that attempted to introduce incorrect behavior 
> >>>while trying to fix compatibility with nose. That's why I think we 
> >>>need a clear message on this sooner than later. Which is why I'm 
> >>>proposing actively raising an error when things are run with nose 
> >>>upfront so there isn't any illusion that things are expected to work.
> >>>
> >>>This doesn't necessarily mean we're moving python 2.6 to the hostile
category.
> >>>Nose support is independent of python 2.6 support. Py26 I would 
> >>>still consider to be unsupported, the issue is that the hack to 
> >>>make py26 work is outside of tempest. This is why we've recommended 
> >>>that people using python 2.6 run with nose, which really is no 
> >>>longer an option. Attila's abandoned patch that I linked above 
> >>>documents points to this bug with a patch to discover which is need to
get python 2.6 working with tempest and testr:
> >>>
> >>>https://code.google.com/p/unittest-ext/issues/detail?id=79
> >
> >_______________________________________________
> >OpenStack-dev mailing list
> >OpenStack-dev at lists.openstack.org
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> One question I had was is there an easy way to setup a config file to 
> specify the test bucket and what can be excluded, like you can with 
> nose.cfg and nose?  We used that for filtering out API tests that 
> didn't work with the PowerVM driver in Nova but I'm not aware of 
> something similar with testr.

So I'm not sure if it's exactly what you're looking for but testr has the
--load-list option which you can use to specify a file which lists the tests
you want to run. I don't think there is a method to exclude tests besides
using a regex filter right now. There is a bug open about this:

https://bugs.launchpad.net/testrepository/+bug/1208610

So I can see you doing this 2 ways either writing a little script that will
generate a list file by doing something like:

1. testr list-tests > file
2. remove excludes from file
3. testr run --load-list file

or making a long unwieldy regex that excludes the tests you need to.
Something like what I did here:

https://review.openstack.org/#/c/51275/4/tox.ini

-Matt Treinish

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6199 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140227/e4a44549/attachment.bin>