Hi Clark, I've been reading this discussion, and I don't really have anything to add to this, except that I've seen the term "standard compliant test cases" used quite a lot, and I'm a little bit confused by it. Would you be so nice and please link to the relevant specification, so that everyone can see what standard exactly you are talking about? Thank you! On Fri, Nov 7, 2025 at 4:57 PM Clark Boylan <cboylan@sapwetik.org> wrote:
On 11/6/25 4:03 PM, Jan Jasek wrote:
Hello everyone, the previous Horizon tests were unstable, hard to maintain, slow in execution. Before completely rewriting the tests we were experimenting and discussing what way to go with the goal to try to find the best way to make the tests stable, easy to maintain and fast. And Using PyTest with all the features it provides - Fixtures, scopings, parameterizations, etc. Rich plugin ecosystem like pytest-html for generating test reports. Basically cover the Horizon well, using modern ways and not reinvent the wheel again.
To be completely honest - I implemented the majority of the new tests for Horizon and I did not know that PyTest is not allowed to be used. If I knew about it I would definitely have discussed it with someone (TC) before.
I see PyTest as a very popular, widely used, and feature-rich open- source framework. In my point of view PyTest is the modern, de facto standard/industry-adopted for testing in the Python ecosystem - That is the reason why I am so surprised that after reimplementation of all Horizon tests (that are now, after years, super stable, easy to
On Thu, Nov 6, 2025, at 2:02 PM, Thomas Goirand wrote: maintain
and running well for multiple cycles already) and started with coverage for Manila-UI plugin, it came out on PTG from watcher team that it is probably not Allowed to use PyTest in OpenStack project. Hi Jan,
What I'm surprised is why a test runner is so important? Can't we run your tests with stestr, even if they use pytest decorator / libs inside?
Pytest the test runner is capable of running standard compliant test cases. Pytest the test framework library introduces features that are not standards compliant that require pytest to execute (or at least this was the case the last time pytest came up). This is one of the motivations for using a standard compliant test runner in the gate. It ensures that you can opt into using pytest or any other runner of your choice. But if you start using pytest it is easy to unknowingly stop working with any other test runner.
I think we're all operating under the assumption that the horizon test cases cannot execute under stestr and require pytest (otherwise I'm not sure what the concern is). That said I don't know if anyone has actually tested if this is the case.
Also, from a package maintainer perspective, it's ok for me if you use pytest, as long as you don't use weird extension not packaged in distro (I'd have to package them just for running Horizon tests, which is a lot of extra work I prefer to avoid). Apart from that, sticking with the most common pytest stuff is very much ok.
What I've been frustrated with Horizon though, is the impossibility to run tests in parallel. When one does, everything falls apart. Is this now fixed? Can I use pytest-xdist, and run pytest with "-n auto", so I can fully use all the cores of my laptop to run Horizon tests?
Thanks for your work, Cheers,
Thomas Goirand (zigo)