On Wed, Mar 11, 2020 at 10:32:32PM +0200, Oleksii Petrenko wrote:
Starting with stestr, could you explain why it was not good enough for your use case?
Stestr will not provide us with fixtures for django (for future use), also with the help of pytest, we probably would be able to unify html creation all across our projects. Also, xml exporting in different formats can help users with aggregating test statistics.
The aggregated data view already exists: http://status.openstack.org/openstack-health/#/ We also have 2 different html views of a test run depending on the level of detail you want: https://7dd927a4891851ac968e-517bfbb0b76f5445108257ba8a306671.ssl.cf5.rackcd... and https://7dd927a4891851ac968e-517bfbb0b76f5445108257ba8a306671.ssl.cf5.rackcd... As for "xml exporting" I assume you're talking about xunitxml. There are several limitations around it, especially for parallel test execution which is why stestr is built around and uses subunit. But, if you want to generate xunitxml from subunit for any reason this is straightforward to do, it's built into subunit: https://github.com/testing-cabal/subunit/blob/master/filters/subunit2junitxm...
more of a general question if they are test only deps that wont be used at runtime which i think is the case in all of the above do they enven need to be in Global-requirements? ignoring the fact that devstack installes all test- requireemtes when it in stall packages which is a different topic if this is only used for generating html report for tests then it seams liek we would not need to corrdiate the software version. pytest is needed to generate coverage reports.
I don't understand this either, we have coverage jobs already running on most projects. The reports get published as part of the job artifacts: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/z... Also as I pointed out on the review, this is not the first time we've discussed this. Since I started working on OpenStack why not runner or framework X (at one point it was nose, then it switched to pytest) has been brought up by someone. We tried to write it down in the project testing interface: https://governance.openstack.org/tc/reference/pti/python.html#python-test-ru... Basically, by using a unittest based runner anybody can use their preferred test runner locally. stestr is used for CI because of the parallel execution and subunit integration to leverage all the infra tooling built around it. That being said horizon has always been an exception because django has special requirements for testing (mainly they publish their testing framework as an extension for a test frameworks other than stdlib unittest). In the past it was needed a nose extension and now it looks like that has been updated to be a pytest exception. I don't see a problem to just morph the old exception that horizon uses nose to horizon uses pytest if it's really necessary to test django. If you do end up using pytest because there is no other choice for django testing, you can convert the xunitxml to subunit to integrate it into all those existing tools I mentioned before with either: https://github.com/mtreinish/health-helm/blob/master/junitxml2subunit.py or https://github.com/mtreinish/junitxml2subunit (do note stackviz and subunit2sql/openstack-health won't be really useful with xunitxml to subunit conversion because xunitxml doesn't track execution timestamps) -Matt Treinish