Add pytest, pytest-django and pytest-html to global requirements
Adding pytest, pytest-django and pytest-html allows import of tests results in xml, html formats for openstack-horizon. What do you think about this?
On 20-03-11 21:01:42, Oleksii Petrenko wrote:
Adding pytest, pytest-django and pytest-html allows import of tests results in xml, html formats for openstack-horizon. What do you think about this?
I think the question is in refrence to using something that is already included in global-requirements (like stestr or something else). Also, here's the review https://review.opendev.org/712315 Starting with stestr, could you explain why it was not good enough for your use case? -- Matthew Thode
On 20-03-11 21:01:42, Oleksii Petrenko wrote:
Adding pytest, pytest-django and pytest-html allows import of tests results in xml, html formats for openstack-horizon. What do you think about this?
I think the question is in refrence to using something that is already included in global-requirements (like stestr or something else).
Also, here's the review https://review.opendev.org/712315
Starting with stestr, could you explain why it was not good enough for your use case? more of a general question if they are test only deps that wont be used at runtime which i think is the case in all of
On Wed, 2020-03-11 at 14:06 -0500, Matthew Thode wrote: the above do they enven need to be in Global-requirements? ignoring the fact that devstack installes all test- requireemtes when it in stall packages which is a different topic if this is only used for generating html report for tests then it seams liek we would not need to corrdiate the software version. that said i am also curious why the normal html report that gets generated form our tox runs in the ci is not sufficent. what does pytest-html add? is it just the ablity to do a local html report when you run tox manually? or do you plan to use pytest-django and pytest-html to do some other testing that you cant currently do?
On Wed, 2020-03-11 at 14:06 -0500, Matthew Thode wrote:
On 20-03-11 21:01:42, Oleksii Petrenko wrote:
Adding pytest, pytest-django and pytest-html allows import of tests results in xml, html formats for openstack-horizon. What do you think about this?
I think the question is in refrence to using something that is already included in global-requirements (like stestr or something else).
Also, here's the review https://review.opendev.org/712315
Starting with stestr, could you explain why it was not good enough for your use case?
more of a general question if they are test only deps that wont be used at runtime which i think is the case in all of the above do they enven need to be in Global-requirements? ignoring the fact that devstack installes all test- requireemtes when it in stall packages which is a different topic if this is only used for generating html report for tests then it seams liek we would not need to corrdiate the software version.
that said i am also curious why the normal html report that gets generated form our tox runs in the ci is not sufficent. what does pytest-html add? is it just the ablity to do a local html report when you run tox manually? or do you plan to use pytest-django and pytest-html to do some other testing that you cant currently do? ah i see that pytest-django will allow the removal of django test which is presuable to adress https://bugs.launchpad.net/horizon/+bug/1866666
On Wed, 2020-03-11 at 19:42 +0000, Sean Mooney wrote: based on https://review.opendev.org/#/c/711195/ if hoizong is thinking of moveing away form its current custome test runner https://github.com/openstack/horizon/blob/stable/pike/manage.py then stestr should at least be considerd but if there is a valid technical reason to go with pytest instead. that said as someone that does not work on horizon im supriced its not already using stestr. give it still has a testr.conf meanign at somepoint it moved a way form testrepostry and to its current custom runner when the other project moved to os-testr and then to stestr. that said i kind of like pytest as a test runner and it would solve some other issues with subunit in nova so im not against adding it.
Starting with stestr, could you explain why it was not good enough for your use case?
Stestr will not provide us with fixtures for django (for future use), also with the help of pytest, we probably would be able to unify html creation all across our projects. Also, xml exporting in different formats can help users with aggregating test statistics.
more of a general question if they are test only deps that wont be used at runtime which i think is the case in all of the above do they enven need to be in Global-requirements? ignoring the fact that devstack installes all test- requireemtes when it in stall packages which is a different topic if this is only used for generating html report for tests then it seams liek we would not need to corrdiate the software version. pytest is needed to generate coverage reports.
On Wed, Mar 11, 2020 at 10:32:32PM +0200, Oleksii Petrenko wrote:
Starting with stestr, could you explain why it was not good enough for your use case?
Stestr will not provide us with fixtures for django (for future use), also with the help of pytest, we probably would be able to unify html creation all across our projects. Also, xml exporting in different formats can help users with aggregating test statistics.
The aggregated data view already exists: http://status.openstack.org/openstack-health/#/ We also have 2 different html views of a test run depending on the level of detail you want: https://7dd927a4891851ac968e-517bfbb0b76f5445108257ba8a306671.ssl.cf5.rackcd... and https://7dd927a4891851ac968e-517bfbb0b76f5445108257ba8a306671.ssl.cf5.rackcd... As for "xml exporting" I assume you're talking about xunitxml. There are several limitations around it, especially for parallel test execution which is why stestr is built around and uses subunit. But, if you want to generate xunitxml from subunit for any reason this is straightforward to do, it's built into subunit: https://github.com/testing-cabal/subunit/blob/master/filters/subunit2junitxm...
more of a general question if they are test only deps that wont be used at runtime which i think is the case in all of the above do they enven need to be in Global-requirements? ignoring the fact that devstack installes all test- requireemtes when it in stall packages which is a different topic if this is only used for generating html report for tests then it seams liek we would not need to corrdiate the software version. pytest is needed to generate coverage reports.
I don't understand this either, we have coverage jobs already running on most projects. The reports get published as part of the job artifacts: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/z... Also as I pointed out on the review, this is not the first time we've discussed this. Since I started working on OpenStack why not runner or framework X (at one point it was nose, then it switched to pytest) has been brought up by someone. We tried to write it down in the project testing interface: https://governance.openstack.org/tc/reference/pti/python.html#python-test-ru... Basically, by using a unittest based runner anybody can use their preferred test runner locally. stestr is used for CI because of the parallel execution and subunit integration to leverage all the infra tooling built around it. That being said horizon has always been an exception because django has special requirements for testing (mainly they publish their testing framework as an extension for a test frameworks other than stdlib unittest). In the past it was needed a nose extension and now it looks like that has been updated to be a pytest exception. I don't see a problem to just morph the old exception that horizon uses nose to horizon uses pytest if it's really necessary to test django. If you do end up using pytest because there is no other choice for django testing, you can convert the xunitxml to subunit to integrate it into all those existing tools I mentioned before with either: https://github.com/mtreinish/health-helm/blob/master/junitxml2subunit.py or https://github.com/mtreinish/junitxml2subunit (do note stackviz and subunit2sql/openstack-health won't be really useful with xunitxml to subunit conversion because xunitxml doesn't track execution timestamps) -Matt Treinish
I am commenting only on horizon specific cases. I believe it helps understanding the situation and discussing the direction.
That being said horizon has always been an exception because django has special requirements for testing (mainly they publish their testing framework as an extension for a test frameworks other than stdlib unittest). In the past it was needed a nose extension and now it looks like that has been updated to be a pytest exception. I don't see a problem to just morph the old exception that horizon uses nose to horizon uses pytest if it's really necessary to test django.
Previously we used nose for the test runner, but the Django (default) test runner is used now. It happens because Django support in nose looks unmaintained and the migration to the Django test runner is simplest and we are confident that it is maintained as long as the Django project is live. Only reason we did/could not choose stestr is that there is no integration support for Django applications in stestr. IIUC, roughly speaking, what Django testing framework provides are: (1) convenient methods for writing tests (for example, custom assertion, sending requests to Django framework and so on) (2) setup Django for testing mainly including loading Django settings (3) Fixtures for Django Database integration (which is not used in horizon) (4) provide Django test runner (1) is useful for test writers and (2) is required as Django settings module is common in Django projects. (1) and (2) are things re-implemented by individual projects like horizon, and (2) is what pytest-django and nose Django plugin do. (2) is missing in stestr. We already have test runners for Django: the Django default test runner and pytest (with pytest-django) and they are maintained well, so from the horizon team perspective it is better to use an existing one. The downside of the Django default test runner is there seems no good way to handle test results. It has no subunit support. It looks like that only stdout is a place to see test results. I think this is the reason Oleksii proposed pytest usage. The above is my understanding around Django testing. I have no preference on a test runner in horizon. If someone can work on stestr Django integration, it would be great. It provides more consistency with other OpenStack projects. If pytest(-django) is adopted I would like to see zuul role(s) along with the proposal. Akihiro Motoki (irc: amotoki) On Thu, Mar 12, 2020 at 6:10 AM Matthew Treinish <mtreinish@kortar.org> wrote:
On Wed, Mar 11, 2020 at 10:32:32PM +0200, Oleksii Petrenko wrote:
Starting with stestr, could you explain why it was not good enough for your use case?
Stestr will not provide us with fixtures for django (for future use), also with the help of pytest, we probably would be able to unify html creation all across our projects. Also, xml exporting in different formats can help users with aggregating test statistics.
The aggregated data view already exists:
http://status.openstack.org/openstack-health/#/
We also have 2 different html views of a test run depending on the level of detail you want:
https://7dd927a4891851ac968e-517bfbb0b76f5445108257ba8a306671.ssl.cf5.rackcd... and https://7dd927a4891851ac968e-517bfbb0b76f5445108257ba8a306671.ssl.cf5.rackcd...
As for "xml exporting" I assume you're talking about xunitxml. There are several limitations around it, especially for parallel test execution which is why stestr is built around and uses subunit. But, if you want to generate xunitxml from subunit for any reason this is straightforward to do, it's built into subunit:
https://github.com/testing-cabal/subunit/blob/master/filters/subunit2junitxm...
more of a general question if they are test only deps that wont be used at runtime which i think is the case in all of the above do they enven need to be in Global-requirements? ignoring the fact that devstack installes all test- requireemtes when it in stall packages which is a different topic if this is only used for generating html report for tests then it seams liek we would not need to corrdiate the software version. pytest is needed to generate coverage reports.
I don't understand this either, we have coverage jobs already running on most projects. The reports get published as part of the job artifacts:
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/z...
Also as I pointed out on the review, this is not the first time we've discussed this. Since I started working on OpenStack why not runner or framework X (at one point it was nose, then it switched to pytest) has been brought up by someone. We tried to write it down in the project testing interface:
https://governance.openstack.org/tc/reference/pti/python.html#python-test-ru...
Basically, by using a unittest based runner anybody can use their preferred test runner locally. stestr is used for CI because of the parallel execution and subunit integration to leverage all the infra tooling built around it.
That being said horizon has always been an exception because django has special requirements for testing (mainly they publish their testing framework as an extension for a test frameworks other than stdlib unittest). In the past it was needed a nose extension and now it looks like that has been updated to be a pytest exception. I don't see a problem to just morph the old exception that horizon uses nose to horizon uses pytest if it's really necessary to test django.
If you do end up using pytest because there is no other choice for django testing, you can convert the xunitxml to subunit to integrate it into all those existing tools I mentioned before with either:
https://github.com/mtreinish/health-helm/blob/master/junitxml2subunit.py or https://github.com/mtreinish/junitxml2subunit
(do note stackviz and subunit2sql/openstack-health won't be really useful with xunitxml to subunit conversion because xunitxml doesn't track execution timestamps)
-Matt Treinish
participants (5)
-
Akihiro Motoki
-
Matthew Thode
-
Matthew Treinish
-
Oleksii Petrenko
-
Sean Mooney