Open Stack

On Wed, Mar 11, 2020 at 10:32:32PM +0200, Oleksii Petrenko wrote:
> > > > Starting with stestr, could you explain why it was not good enough for
> > > > your use case?
> > >
> Stestr will not provide us with fixtures for django (for future use),
> also with the help of pytest, we probably would be able to unify html
> creation all across our projects. Also, xml exporting in different
> formats can help users with aggregating test statistics.

The aggregated data view already exists:

http://status.openstack.org/openstack-health/#/

We also have 2 different html views of a test run depending on the level of
detail you want:

https://7dd927a4891851ac968e-517bfbb0b76f5445108257ba8a306671.ssl.cf5.rackcdn.com/712315/2/check/tempest-full-py3/c20f9f1/testr_results.html
and
https://7dd927a4891851ac968e-517bfbb0b76f5445108257ba8a306671.ssl.cf5.rackcdn.com/712315/2/check/tempest-full-py3/c20f9f1/controller/logs/stackviz/index.html#/stdin/timeline

As for "xml exporting" I assume you're talking about xunitxml. There are
several limitations around it, especially for parallel test execution
which is why stestr is built around and uses subunit. But, if you want to
generate xunitxml from subunit for any reason this is straightforward to
do, it's built into subunit:

https://github.com/testing-cabal/subunit/blob/master/filters/subunit2junitxml

> 
> > > more of a general question if they are test only deps that wont be used at runtime which i think is the case in all of
> > > the above do they enven need to be in Global-requirements? ignoring the fact that devstack installes all test-
> > > requireemtes when it in stall packages which is a different topic if this is only used for generating html report for
> > > tests then it seams liek we would not need to corrdiate the software version.
> pytest is needed to generate coverage reports.

I don't understand this either, we have coverage jobs already running on most
projects. The reports get published as part of the job artifacts:

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fd1/706509/2/check/openstack-tox-cover/fd1df05/cover/index.html


Also as I pointed out on the review, this is not the first time we've
discussed this. Since I started working on OpenStack why not runner or
framework X (at one point it was nose, then it switched to pytest) has
been brought up by someone. We tried to write it down in the project
testing interface:

https://governance.openstack.org/tc/reference/pti/python.html#python-test-running

Basically, by using a unittest based runner anybody can use their preferred
test runner locally. stestr is used for CI because of the parallel execution
and subunit integration to leverage all the infra tooling built around it.

That being said horizon has always been an exception because django
has special requirements for testing (mainly they publish their testing
framework as an extension for a test frameworks other than stdlib unittest). In
the past it was needed a nose extension and now it looks like that has been
updated to be a pytest exception. I don't see a problem to just morph the old
exception that horizon uses nose to horizon uses pytest if it's really
necessary to test django.

If you do end up using pytest because there is no other choice for django
testing, you can convert the xunitxml to subunit to integrate it into all
those existing tools I mentioned before with either:

https://github.com/mtreinish/health-helm/blob/master/junitxml2subunit.py
or
https://github.com/mtreinish/junitxml2subunit

(do note stackviz and subunit2sql/openstack-health won't be really useful
with xunitxml to subunit conversion because xunitxml doesn't track
execution timestamps)

-Matt Treinish
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200311/2c0f73a6/attachment.sig>