[openstack-dev] [QA] The future of nosetests with Tempest

Matt Riedemann mriedem at linux.vnet.ibm.com
Thu Feb 27 03:33:13 UTC 2014



On 2/25/2014 7:46 PM, Matt Riedemann wrote:
>
>
> On 2/12/2014 1:57 PM, Matthew Treinish wrote:
>> On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:
>>>
>>>
>>> On 1/17/2014 8:34 AM, Matthew Treinish wrote:
>>>> On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote:
>>>>> On 01/16/2014 10:56 PM, Matthew Treinish wrote:
>>>>>> Hi everyone,
>>>>>>
>>>>>> With some recent changes made to Tempest compatibility with
>>>>>> nosetests is going
>>>>>> away. We've started using newer features that nose just doesn't
>>>>>> support. One
>>>>>> example of this is that we've started using testscenarios and
>>>>>> we're planning to
>>>>>> do this in more places moving forward.
>>>>>>
>>>>>> So at Icehouse-3 I'm planning to push the patch out to remove
>>>>>> nosetests from the
>>>>>> requirements list and all the workarounds and references to nose
>>>>>> will be pulled
>>>>>> out of the tree. Tempest will also start raising an unsupported
>>>>>> exception when
>>>>>> you try to run it with nose so that there isn't any confusion on
>>>>>> this moving
>>>>>> forward. We talked about doing this at summit briefly and I've
>>>>>> brought it up a
>>>>>> couple of times before, but I believe it is time to do this now. I
>>>>>> feel for
>>>>>> tempest to move forward we need to do this now so that there isn't
>>>>>> any ambiguity
>>>>>> as we add even more features and new types of testing.
>>>>> I'm with you up to here.
>>>>>>
>>>>>> Now, this will have implications for people running tempest with
>>>>>> python 2.6
>>>>>> since up until now we've set nosetests. There is a workaround for
>>>>>> getting
>>>>>> tempest to run with python 2.6 and testr see:
>>>>>>
>>>>>> https://review.openstack.org/#/c/59007/1/README.rst
>>>>>>
>>>>>> but essentially this means that when nose is marked as unsupported
>>>>>> on tempest
>>>>>> python 2.6 will also be unsupported by Tempest. (which honestly it
>>>>>> basically has
>>>>>> been for while now just we've gone without making it official)
>>>>> The way we handle different runners/os can be categorized as "tested
>>>>> in gate", "unsupported" (should work, possibly some hacks needed),
>>>>> and "hostile". At present, both nose and py2.6 I would say are in
>>>>> the unsupported category. The title of this message and the content
>>>>> up to here says we are moving nose to the hostile category. With
>>>>> only 2 months to feature freeze I see no justification in moving
>>>>> py2.6 to the hostile category. I don't see what new testing features
>>>>> scheduled for the next two months will be enabled by saying that
>>>>> tempest cannot and will not run on 2.6. It has been agreed I think
>>>>> by all projects that py2.6 will be dropped in J. It is OK that py2.6
>>>>> will require some hacks to work and if in the next few months it
>>>>> needs a few more then that is ok. If I am missing another connection
>>>>> between the py2.6 and nose issues, please explain.
>>>>>
>>>>
>>>> So honestly we're already at this point in tempest. Nose really just
>>>> doesn't
>>>> work with tempest, and we're adding more features to tempest, your
>>>> negative test
>>>> generator being one of them, that interfere further with nose. I've
>>>> seen several
>>>
>>> I disagree here, my team is running Tempest API, CLI and scenario
>>> tests every day with nose on RHEL 6 with minimal issues.  I had to
>>> workaround the negative test discovery by simply sed'ing that out of
>>> the tests before running it, but that's acceptable to me until we
>>> can start testing on RHEL 7.  Otherwise I'm completely OK with
>>> saying py26 isn't really supported and isn't used in the gate, and
>>> it's a buyer beware situation to make it work, which includes
>>> pushing up trivial patches to make it work (which I did a few of
>>> last week, and they were small syntax changes or usages of
>>> testtools).
>>>
>>> I don't understand how the core projects can be running unit tests
>>> in the gate on py26 but our functional integration project is going
>>> to actively go out and make it harder to run Tempest with py26, that
>>> sucks.
>>>
>>> If we really want to move the test project away from py26, let's
>>> make the concerted effort to get the core projects to move with it.
>>
>> So as I said before the python 2.6 story for tempest remains the same
>> after this
>> change. The only thing that we'll be doing is actively preventing nose
>> from
>> working with tempest.
>>
>>>
>>> And FWIW, I tried the discover.py patch with unittest2 and
>>> testscenarios last week and either I botched it, it's not documented
>>> properly on how to apply it, or I screwed something up, but it
>>> didn't work for me, so I'm not convinced that's the workaround.
>>>
>>> What's the other option for running Tempest on py26 (keeping RHEL 6
>>> in mind)?  Using tox with testr and pip?  I'm doing this all
>>> single-node.
>>
>> Yes, that is what the discover patch is used to enable. By disabling
>> nose the
>> only path to run tempest with py2.6 is to use testr. (which is what it
>> always
>> should have been)
>>
>> Attila confirmed it was working here:
>> http://fpaste.org/76651/32143139/
>> in that example he applies 2 patches the second one is currently in
>> the gate for
>> tempest. (https://review.openstack.org/#/c/72388/ ) So all that needs
>> to be done
>> is to apply that discover patch:
>>
>> https://code.google.com/p/unittest-ext/issues/detail?id=79
>>
>> (which I linked to before)
>>
>> Then tempest should run more or less the same between 2.7 and 2.6.
>> (The only
>> difference I've seen is in how skips are handled)
>>
>>>
>>>> patches this cycle that attempted to introduce incorrect behavior
>>>> while trying
>>>> to fix compatibility with nose. That's why I think we need a clear
>>>> message on
>>>> this sooner than later. Which is why I'm proposing actively raising
>>>> an error
>>>> when things are run with nose upfront so there isn't any illusion
>>>> that things
>>>> are expected to work.
>>>>
>>>> This doesn't necessarily mean we're moving python 2.6 to the hostile
>>>> category.
>>>> Nose support is independent of python 2.6 support. Py26 I would
>>>> still consider
>>>> to be unsupported, the issue is that the hack to make py26 work is
>>>> outside of
>>>> tempest. This is why we've recommended that people using python 2.6
>>>> run with
>>>> nose, which really is no longer an option. Attila's abandoned patch
>>>> that I
>>>> linked above documents points to this bug with a patch to discover
>>>> which is
>>>> need to get python 2.6 working with tempest and testr:
>>>>
>>>> https://code.google.com/p/unittest-ext/issues/detail?id=79
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> One question I had was is there an easy way to setup a config file to
> specify the test bucket and what can be excluded, like you can with
> nose.cfg and nose?  We used that for filtering out API tests that didn't
> work with the PowerVM driver in Nova but I'm not aware of something
> similar with testr.
>

Another question I guess, and I know this is too late now, but the 
nosexcover package allowed us to get xunit xml results for the test runs 
and then jenkins has a nice way to process that xml and keep a history 
of failures and dig into the stack traces, how long the tests take, 
trends, etc.  I'm assuming there is nothing like that available with testr?

-- 

Thanks,

Matt Riedemann




More information about the OpenStack-dev mailing list