Open Stack

I am commenting only on horizon specific cases.
I believe it helps understanding the situation and discussing the direction.

> That being said horizon has always been an exception because django
> has special requirements for testing (mainly they publish their testing
> framework as an extension for a test frameworks other than stdlib unittest). In
> the past it was needed a nose extension and now it looks like that has been
> updated to be a pytest exception. I don't see a problem to just morph the old
> exception that horizon uses nose to horizon uses pytest if it's really
> necessary to test django.

Previously we used nose for the test runner, but the Django (default)
test runner
is used now. It happens because Django support in nose looks unmaintained and
the migration to the Django test runner is simplest and we are
confident that it is
maintained as long as the Django project is live. Only reason we
did/could not choose
stestr is that there is no integration support for Django applications
in stestr.

IIUC, roughly speaking, what Django testing framework provides are:
(1) convenient methods for writing tests (for example, custom
assertion, sending requests to Django framework and so on)
(2) setup Django for testing mainly including loading Django settings
(3) Fixtures for Django Database integration (which is not used in horizon)
(4) provide Django test runner

(1) is useful for test writers and (2) is required as Django settings
module is common in Django projects.
(1) and (2) are things re-implemented by individual projects like horizon, and
(2) is what pytest-django and nose Django plugin do.

(2) is missing in stestr.
We already have test runners for Django: the Django default test
runner and pytest (with pytest-django)
and they are maintained well, so from the horizon team perspective it
is better to use an existing one.

The downside of the Django default test runner is there seems no good
way to handle test results.
It has no subunit support. It looks like that only stdout is a place
to see test results.
I think this is the reason Oleksii proposed pytest usage.

The above is my understanding around Django testing.

I have no preference on a test runner in horizon.
If someone can work on stestr Django integration, it would be great.
It provides more consistency with other OpenStack projects.
If pytest(-django) is adopted I would like to see zuul role(s) along
with the proposal.

Akihiro Motoki (irc: amotoki)

On Thu, Mar 12, 2020 at 6:10 AM Matthew Treinish <mtreinish at kortar.org> wrote:
>
> On Wed, Mar 11, 2020 at 10:32:32PM +0200, Oleksii Petrenko wrote:
> > > > > Starting with stestr, could you explain why it was not good enough for
> > > > > your use case?
> > > >
> > Stestr will not provide us with fixtures for django (for future use),
> > also with the help of pytest, we probably would be able to unify html
> > creation all across our projects. Also, xml exporting in different
> > formats can help users with aggregating test statistics.
>
> The aggregated data view already exists:
>
> http://status.openstack.org/openstack-health/#/
>
> We also have 2 different html views of a test run depending on the level of
> detail you want:
>
> https://7dd927a4891851ac968e-517bfbb0b76f5445108257ba8a306671.ssl.cf5.rackcdn.com/712315/2/check/tempest-full-py3/c20f9f1/testr_results.html
> and
> https://7dd927a4891851ac968e-517bfbb0b76f5445108257ba8a306671.ssl.cf5.rackcdn.com/712315/2/check/tempest-full-py3/c20f9f1/controller/logs/stackviz/index.html#/stdin/timeline
>
> As for "xml exporting" I assume you're talking about xunitxml. There are
> several limitations around it, especially for parallel test execution
> which is why stestr is built around and uses subunit. But, if you want to
> generate xunitxml from subunit for any reason this is straightforward to
> do, it's built into subunit:
>
> https://github.com/testing-cabal/subunit/blob/master/filters/subunit2junitxml
>
> >
> > > > more of a general question if they are test only deps that wont be used at runtime which i think is the case in all of
> > > > the above do they enven need to be in Global-requirements? ignoring the fact that devstack installes all test-
> > > > requireemtes when it in stall packages which is a different topic if this is only used for generating html report for
> > > > tests then it seams liek we would not need to corrdiate the software version.
> > pytest is needed to generate coverage reports.
>
> I don't understand this either, we have coverage jobs already running on most
> projects. The reports get published as part of the job artifacts:
>
> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fd1/706509/2/check/openstack-tox-cover/fd1df05/cover/index.html
>
>
> Also as I pointed out on the review, this is not the first time we've
> discussed this. Since I started working on OpenStack why not runner or
> framework X (at one point it was nose, then it switched to pytest) has
> been brought up by someone. We tried to write it down in the project
> testing interface:
>
> https://governance.openstack.org/tc/reference/pti/python.html#python-test-running
>
> Basically, by using a unittest based runner anybody can use their preferred
> test runner locally. stestr is used for CI because of the parallel execution
> and subunit integration to leverage all the infra tooling built around it.
>
> That being said horizon has always been an exception because django
> has special requirements for testing (mainly they publish their testing
> framework as an extension for a test frameworks other than stdlib unittest). In
> the past it was needed a nose extension and now it looks like that has been
> updated to be a pytest exception. I don't see a problem to just morph the old
> exception that horizon uses nose to horizon uses pytest if it's really
> necessary to test django.
>
> If you do end up using pytest because there is no other choice for django
> testing, you can convert the xunitxml to subunit to integrate it into all
> those existing tools I mentioned before with either:
>
> https://github.com/mtreinish/health-helm/blob/master/junitxml2subunit.py
> or
> https://github.com/mtreinish/junitxml2subunit
>
> (do note stackviz and subunit2sql/openstack-health won't be really useful
> with xunitxml to subunit conversion because xunitxml doesn't track
> execution timestamps)
>
> -Matt Treinish