[openstack-dev] [QA] The future of nosetests with Tempest
Yair Fried
yfried at redhat.com
Thu Mar 6 08:05:51 UTC 2014
----- Original Message -----
> From: "Alexei Kornienko" <alexei.kornienko at gmail.com>
> To: openstack-dev at lists.openstack.org
> Sent: Friday, February 28, 2014 12:43:13 PM
> Subject: Re: [openstack-dev] [QA] The future of nosetests with
> Tempest
> Hi,
> Let me express my concerns on this topic:
> > With some recent changes made to Tempest compatibility with
>
> > nosetests is going away.
>
> I think that we should not drop nosetests support from tempest or any
> other project. The problem with testrepository is that it's not
> providing any debugger support at all (and will never provide). It
> also has some issues with proving error traces in human readable
> form and it can be quite hard to find out what is actually broken.
> Because of this I think we should try to avoid any kind of test
> libraries that break compatibility with conventional test runners.
> Our tests should be able to run correctly with nosetests, teststools
> or plain old unittest runner. If for some reason test libraries
> (like testscenarios) doesn't provide support for this we should fix
> this libraries or avoid their usage.
+1
I have the same concern about debugging. It's an essential tool (for me, at least) in creating scenario tests. The more complex the test, the harder it is to rely on simple log-prints.
> Regards,
> Alexei Kornienko
> On 02/27/2014 06:36 PM, Frittoli, Andrea (HP Cloud) wrote:
> > This is another example of achieving the same result (exclusion
> > from
> > a
>
> > list):
> > https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element
> > s/tempest/tests2skip.py
> > https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element
> > s/tempest/tests2skip.txt
>
> > andrea
>
> > -----Original Message-----
>
> > From: Matthew Treinish [ mailto:mtreinish at kortar.org ]
>
> > Sent: 27 February 2014 15:49
>
> > To: OpenStack Development Mailing List (not for usage questions)
>
> > Subject: Re: [openstack-dev] [QA] The future of nosetests with
> > Tempest
>
> > On Tue, Feb 25, 2014 at 07:46:23PM -0600, Matt Riedemann wrote:
>
> > > On 2/12/2014 1:57 PM, Matthew Treinish wrote:
> >
>
> > > > On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:
> > >
> >
>
> > > > > On 1/17/2014 8:34 AM, Matthew Treinish wrote:
> > > >
> > >
> >
>
> > > > > > On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz
> > > > > > wrote:
> > > > >
> > > >
> > >
> >
>
> > > > > > > On 01/16/2014 10:56 PM, Matthew Treinish wrote:
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > Hi everyone,
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > With some recent changes made to Tempest compatibility
> > > > > > > > with
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > nosetests is going away. We've started using newer
> > > > > > > > features
> > > > > > > > that
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > nose just doesn't support. One example of this is that
> > > > > > > > we've
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > started using testscenarios and we're planning to do
> > > > > > > > this
> > > > > > > > in
> > > > > > > > more
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > places moving forward.
>
> > > > > > > > So at Icehouse-3 I'm planning to push the patch out to
> > > > > > > > remove
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > nosetests from the requirements list and all the
> > > > > > > > workarounds
> > > > > > > > and
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > references to nose will be pulled out of the tree.
> > > > > > > > Tempest
> > > > > > > > will
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > also start raising an unsupported exception when you
> > > > > > > > try
> > > > > > > > to
> > > > > > > > run
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > it with nose so that there isn't any confusion on this
> > > > > > > > moving
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > forward. We talked about doing this at summit briefly
> > > > > > > > and
> > > > > > > > I've
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > brought it up a couple of times before, but I believe
> > > > > > > > it
> > > > > > > > is
> > > > > > > > time
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > to do this now. I feel for tempest to move forward we
> > > > > > > > need
> > > > > > > > to
> > > > > > > > do
> > > > > > > > this
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > now so that there isn't any ambiguity as we add even more features
> > and new
>
> > types of testing.
>
> > > > > > > I'm with you up to here.
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > Now, this will have implications for people running
> > > > > > > > tempest
> > > > > > > > with
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > python 2.6 since up until now we've set nosetests.
> > > > > > > > There
> > > > > > > > is
> > > > > > > > a
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > workaround for getting tempest to run with python 2.6
> > > > > > > > and
> > > > > > > > testr
> > > > > > > > see:
> > > > > > > > https://review.openstack.org/#/c/59007/1/README.rst but
> > > > > > > > essentially
> > > > > > > > this means that when nose is marked as
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > unsupported on tempest python 2.6 will also be
> > > > > > > > unsupported
> > > > > > > > by
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > Tempest. (which honestly it basically has been for
> > > > > > > > while
> > > > > > > > now
> > > > > > > > just
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > > we've gone without making it official)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > The way we handle different runners/os can be categorized
> > > > > > > as
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > "tested in gate", "unsupported" (should work, possibly
> > > > > > > some
> > > > > > > hacks
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > needed), and "hostile". At present, both nose and py2.6 I
> > > > > > > would
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > say are in the unsupported category. The title of this
> > > > > > > message
> > > > > > > and
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > the content up to here says we are moving nose to the
> > > > > > > hostile
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > category. With only 2 months to feature freeze I see no
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > justification in moving
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > py2.6 to the hostile category. I don't see what new
> > > > > > > testing
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > features scheduled for the next two months will be
> > > > > > > enabled
> > > > > > > by
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > saying that tempest cannot and will not run on 2.6. It
> > > > > > > has
> > > > > > > been
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > agreed I think by all projects that py2.6 will be dropped
> > > > > > > in
> > > > > > > J.
> > > > > > > It
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > is OK that py2.6 will require some hacks to work and if
> > > > > > > in
> > > > > > > the
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > next few months it needs a few more then that is ok. If I
> > > > > > > am
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > > > > > > missing another connection between the py2.6 and nose
> > > > > > > issues,
> > > > > > > please
> > > > > >
> > > > >
> > > >
> > >
> >
>
> > explain.
>
> > > > > > So honestly we're already at this point in tempest. Nose
> > > > > > really
> > > > >
> > > >
> > >
> >
>
> > > > > > just doesn't work with tempest, and we're adding more
> > > > > > features
> > > > > > to
> > > > >
> > > >
> > >
> >
>
> > > > > > tempest, your negative test generator being one of them,
> > > > > > that
> > > > >
> > > >
> > >
> >
>
> > > > > > interfere further with nose. I've seen several
> > > > >
> > > >
> > >
> >
>
> > > > > I disagree here, my team is running Tempest API, CLI and
> > > > > scenario
> > > >
> > >
> >
>
> > > > > tests every day with nose on RHEL 6 with minimal issues. I
> > > > > had
> > > > > to
> > > >
> > >
> >
>
> > > > > workaround the negative test discovery by simply sed'ing that
> > > > > out
> > > > > of
> > > >
> > >
> >
>
> > > > > the tests before running it, but that's acceptable to me
> > > > > until
> > > > > we
> > > >
> > >
> >
>
> > > > > can start testing on RHEL 7. Otherwise I'm completely OK
> > > > > with
> > > >
> > >
> >
>
> > > > > saying py26 isn't really supported and isn't used in the
> > > > > gate,
> > > > > and
> > > >
> > >
> >
>
> > > > > it's a buyer beware situation to make it work, which includes
> > > >
> > >
> >
>
> > > > > pushing up trivial patches to make it work (which I did a few
> > > > > of
> > > >
> > >
> >
>
> > > > > last week, and they were small syntax changes or usages of
> > > >
> > >
> >
>
> > > > > testtools).
> > > >
> > >
> >
>
> > > > > I don't understand how the core projects can be running unit
> > > > > tests
> > > >
> > >
> >
>
> > > > > in the gate on py26 but our functional integration project is
> > > > > going
> > > >
> > >
> >
>
> > > > > to actively go out and make it harder to run Tempest with
> > > > > py26,
> > > > > that
> > > >
> > >
> >
>
> > > > > sucks.
> > > >
> > >
> >
>
> > > > > If we really want to move the test project away from py26,
> > > > > let's
> > > >
> > >
> >
>
> > > > > make the concerted effort to get the core projects to move
> > > > > with
> > > > > it.
> > > >
> > >
> >
>
> > > > So as I said before the python 2.6 story for tempest remains
> > > > the
> > > > same
> > >
> >
>
> > > > after this change. The only thing that we'll be doing is
> > > > actively
> > >
> >
>
> > > > preventing nose from working with tempest.
> > >
> >
>
> > > > > And FWIW, I tried the discover.py patch with unittest2 and
> > > >
> > >
> >
>
> > > > > testscenarios last week and either I botched it, it's not
> > > > > documented
> > > >
> > >
> >
>
> > > > > properly on how to apply it, or I screwed something up, but
> > > > > it
> > > >
> > >
> >
>
> > > > > didn't work for me, so I'm not convinced that's the
> > > > > workaround.
> > > >
> > >
> >
>
> > > > > What's the other option for running Tempest on py26 (keeping
> > > > > RHEL
> > > > > 6
> > > >
> > >
> >
>
> > > > > in mind)? Using tox with testr and pip? I'm doing this all
> > > >
> > >
> >
>
> > > > > single-node.
> > > >
> > >
> >
>
> > > > Yes, that is what the discover patch is used to enable. By
> > > > disabling
> > >
> >
>
> > > > nose the only path to run tempest with py2.6 is to use testr.
> > > > (which
> > >
> >
>
> > > > is what it always should have been)
> > >
> >
>
> > > > Attila confirmed it was working here:
> > > > http://fpaste.org/76651/32143139/ in that example he applies 2
> > > > patches the second one is currently in
> > >
> >
>
> > > > the gate for tempest. ( https://review.openstack.org/#/c/72388/
> > > > )
> > > > So
> > >
> >
>
> > > > all that needs to be done is to apply that discover patch:
> > > > https://code.google.com/p/unittest-ext/issues/detail?id=79
> > > > (which
> > > > I
> > > > linked to before)
> > >
> >
>
> > > > Then tempest should run more or less the same between 2.7 and
> > > > 2.6.
> > >
> >
>
> > > > (The only difference I've seen is in how skips are handled)
> > >
> >
>
> > > > > > patches this cycle that attempted to introduce incorrect
> > > > > > behavior
> > > > >
> > > >
> > >
> >
>
> > > > > > while trying to fix compatibility with nose. That's why I
> > > > > > think
> > > > > > we
> > > > >
> > > >
> > >
> >
>
> > > > > > need a clear message on this sooner than later. Which is
> > > > > > why
> > > > > > I'm
> > > > >
> > > >
> > >
> >
>
> > > > > > proposing actively raising an error when things are run
> > > > > > with
> > > > > > nose
> > > > >
> > > >
> > >
> >
>
> > > > > > upfront so there isn't any illusion that things are
> > > > > > expected
> > > > > > to
> > > > > > work.
> > > > >
> > > >
> > >
> >
>
> > > > > > This doesn't necessarily mean we're moving python 2.6 to
> > > > > > the
> > > > > > hostile
> > > > >
> > > >
> > >
> >
>
> > category.
>
> > > > > > Nose support is independent of python 2.6 support. Py26 I
> > > > > > would
> > > > >
> > > >
> > >
> >
>
> > > > > > still consider to be unsupported, the issue is that the
> > > > > > hack
> > > > > > to
> > > > >
> > > >
> > >
> >
>
> > > > > > make py26 work is outside of tempest. This is why we've
> > > > > > recommended
> > > > >
> > > >
> > >
> >
>
> > > > > > that people using python 2.6 run with nose, which really is
> > > > > > no
> > > > >
> > > >
> > >
> >
>
> > > > > > longer an option. Attila's abandoned patch that I linked
> > > > > > above
> > > > >
> > > >
> > >
> >
>
> > > > > > documents points to this bug with a patch to discover which
> > > > > > is
> > > > > > need
> > > > > > to
> > > > >
> > > >
> > >
> >
>
> > get python 2.6 working with tempest and testr:
>
> > > > > > https://code.google.com/p/unittest-ext/issues/detail?id=79
> > > > >
> > > >
> > >
> >
>
> > > > _______________________________________________
> > >
> >
>
> > > > OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
>
> > > One question I had was is there an easy way to setup a config
> > > file
> > > to
> >
>
> > > specify the test bucket and what can be excluded, like you can
> > > with
> >
>
> > > nose.cfg and nose? We used that for filtering out API tests that
> >
>
> > > didn't work with the PowerVM driver in Nova but I'm not aware of
> >
>
> > > something similar with testr.
> >
>
> > So I'm not sure if it's exactly what you're looking for but testr
> > has
> > the
>
> > --load-list option which you can use to specify a file which lists
> > the tests
>
> > you want to run. I don't think there is a method to exclude tests
> > besides
>
> > using a regex filter right now. There is a bug open about this:
> > https://bugs.launchpad.net/testrepository/+bug/1208610 So I can see
> > you doing this 2 ways either writing a little script that will
>
> > generate a list file by doing something like:
>
> > 1. testr list-tests > file
>
> > 2. remove excludes from file
>
> > 3. testr run --load-list file
>
> > or making a long unwieldy regex that excludes the tests you need
> > to.
>
> > Something like what I did here:
> > https://review.openstack.org/#/c/51275/4/tox.ini -Matt Treinish
>
> > _______________________________________________
>
> > OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> > _______________________________________________
>
> > OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
More information about the OpenStack-dev
mailing list