Open Stack

On Feb 26, 2014 7:37 PM, "Matt Riedemann" <mriedem at linux.vnet.ibm.com>
wrote:
>
>
>
> On 2/25/2014 7:46 PM, Matt Riedemann wrote:
>>
>>
>>
>> On 2/12/2014 1:57 PM, Matthew Treinish wrote:
>>>
>>> On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote:
>>>>
>>>>
>>>>
>>>> On 1/17/2014 8:34 AM, Matthew Treinish wrote:
>>>>>
>>>>> On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote:
>>>>>>
>>>>>> On 01/16/2014 10:56 PM, Matthew Treinish wrote:
>>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> With some recent changes made to Tempest compatibility with
>>>>>>> nosetests is going
>>>>>>> away. We've started using newer features that nose just doesn't
>>>>>>> support. One
>>>>>>> example of this is that we've started using testscenarios and
>>>>>>> we're planning to
>>>>>>> do this in more places moving forward.
>>>>>>>
>>>>>>> So at Icehouse-3 I'm planning to push the patch out to remove
>>>>>>> nosetests from the
>>>>>>> requirements list and all the workarounds and references to nose
>>>>>>> will be pulled
>>>>>>> out of the tree. Tempest will also start raising an unsupported
>>>>>>> exception when
>>>>>>> you try to run it with nose so that there isn't any confusion on
>>>>>>> this moving
>>>>>>> forward. We talked about doing this at summit briefly and I've
>>>>>>> brought it up a
>>>>>>> couple of times before, but I believe it is time to do this now. I
>>>>>>> feel for
>>>>>>> tempest to move forward we need to do this now so that there isn't
>>>>>>> any ambiguity
>>>>>>> as we add even more features and new types of testing.
>>>>>>
>>>>>> I'm with you up to here.
>>>>>>>
>>>>>>>
>>>>>>> Now, this will have implications for people running tempest with
>>>>>>> python 2.6
>>>>>>> since up until now we've set nosetests. There is a workaround for
>>>>>>> getting
>>>>>>> tempest to run with python 2.6 and testr see:
>>>>>>>
>>>>>>> https://review.openstack.org/#/c/59007/1/README.rst
>>>>>>>
>>>>>>> but essentially this means that when nose is marked as unsupported
>>>>>>> on tempest
>>>>>>> python 2.6 will also be unsupported by Tempest. (which honestly it
>>>>>>> basically has
>>>>>>> been for while now just we've gone without making it official)
>>>>>>
>>>>>> The way we handle different runners/os can be categorized as "tested
>>>>>> in gate", "unsupported" (should work, possibly some hacks needed),
>>>>>> and "hostile". At present, both nose and py2.6 I would say are in
>>>>>> the unsupported category. The title of this message and the content
>>>>>> up to here says we are moving nose to the hostile category. With
>>>>>> only 2 months to feature freeze I see no justification in moving
>>>>>> py2.6 to the hostile category. I don't see what new testing features
>>>>>> scheduled for the next two months will be enabled by saying that
>>>>>> tempest cannot and will not run on 2.6. It has been agreed I think
>>>>>> by all projects that py2.6 will be dropped in J. It is OK that py2.6
>>>>>> will require some hacks to work and if in the next few months it
>>>>>> needs a few more then that is ok. If I am missing another connection
>>>>>> between the py2.6 and nose issues, please explain.
>>>>>>
>>>>>
>>>>> So honestly we're already at this point in tempest. Nose really just
>>>>> doesn't
>>>>> work with tempest, and we're adding more features to tempest, your
>>>>> negative test
>>>>> generator being one of them, that interfere further with nose. I've
>>>>> seen several
>>>>
>>>>
>>>> I disagree here, my team is running Tempest API, CLI and scenario
>>>> tests every day with nose on RHEL 6 with minimal issues.  I had to
>>>> workaround the negative test discovery by simply sed'ing that out of
>>>> the tests before running it, but that's acceptable to me until we
>>>> can start testing on RHEL 7.  Otherwise I'm completely OK with
>>>> saying py26 isn't really supported and isn't used in the gate, and
>>>> it's a buyer beware situation to make it work, which includes
>>>> pushing up trivial patches to make it work (which I did a few of
>>>> last week, and they were small syntax changes or usages of
>>>> testtools).
>>>>
>>>> I don't understand how the core projects can be running unit tests
>>>> in the gate on py26 but our functional integration project is going
>>>> to actively go out and make it harder to run Tempest with py26, that
>>>> sucks.
>>>>
>>>> If we really want to move the test project away from py26, let's
>>>> make the concerted effort to get the core projects to move with it.
>>>
>>>
>>> So as I said before the python 2.6 story for tempest remains the same
>>> after this
>>> change. The only thing that we'll be doing is actively preventing nose
>>> from
>>> working with tempest.
>>>
>>>>
>>>> And FWIW, I tried the discover.py patch with unittest2 and
>>>> testscenarios last week and either I botched it, it's not documented
>>>> properly on how to apply it, or I screwed something up, but it
>>>> didn't work for me, so I'm not convinced that's the workaround.
>>>>
>>>> What's the other option for running Tempest on py26 (keeping RHEL 6
>>>> in mind)?  Using tox with testr and pip?  I'm doing this all
>>>> single-node.
>>>
>>>
>>> Yes, that is what the discover patch is used to enable. By disabling
>>> nose the
>>> only path to run tempest with py2.6 is to use testr. (which is what it
>>> always
>>> should have been)
>>>
>>> Attila confirmed it was working here:
>>> http://fpaste.org/76651/32143139/
>>> in that example he applies 2 patches the second one is currently in
>>> the gate for
>>> tempest. (https://review.openstack.org/#/c/72388/ ) So all that needs
>>> to be done
>>> is to apply that discover patch:
>>>
>>> https://code.google.com/p/unittest-ext/issues/detail?id=79
>>>
>>> (which I linked to before)
>>>
>>> Then tempest should run more or less the same between 2.7 and 2.6.
>>> (The only
>>> difference I've seen is in how skips are handled)
>>>
>>>>
>>>>> patches this cycle that attempted to introduce incorrect behavior
>>>>> while trying
>>>>> to fix compatibility with nose. That's why I think we need a clear
>>>>> message on
>>>>> this sooner than later. Which is why I'm proposing actively raising
>>>>> an error
>>>>> when things are run with nose upfront so there isn't any illusion
>>>>> that things
>>>>> are expected to work.
>>>>>
>>>>> This doesn't necessarily mean we're moving python 2.6 to the hostile
>>>>> category.
>>>>> Nose support is independent of python 2.6 support. Py26 I would
>>>>> still consider
>>>>> to be unsupported, the issue is that the hack to make py26 work is
>>>>> outside of
>>>>> tempest. This is why we've recommended that people using python 2.6
>>>>> run with
>>>>> nose, which really is no longer an option. Attila's abandoned patch
>>>>> that I
>>>>> linked above documents points to this bug with a patch to discover
>>>>> which is
>>>>> need to get python 2.6 working with tempest and testr:
>>>>>
>>>>> https://code.google.com/p/unittest-ext/issues/detail?id=79
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> One question I had was is there an easy way to setup a config file to
>> specify the test bucket and what can be excluded, like you can with
>> nose.cfg and nose?  We used that for filtering out API tests that didn't
>> work with the PowerVM driver in Nova but I'm not aware of something
>> similar with testr.
>>
>
> Another question I guess, and I know this is too late now, but the
nosexcover package allowed us to get xunit xml results for the test runs
and then jenkins has a nice way to process that xml and keep a history of
failures and dig into the stack traces, how long the tests take, trends,
etc.  I'm assuming there is nothing like that available with testr?
>
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

There is a subunit to xunit converter bundled with python-subunit.
subunit2xunit maybe? On a phone so hard to check.

Also fwiw using Jenkins to mangle that data doesn't scale well which is why
we don't do it. This may or may not be a problem. In particular the xunit
plugin has a global lock and jobs that use it can't finish until the
previous job has had its xunit processed. This is terrible.

Clark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140226/7fcb0a62/attachment.html>