Hi Thomas,

any chance that worker 6 happened to execute tests from a single class? If I remember correctly (it's been a couple of years since I was experimenting with this) the scheduler in stestr wasn't able to schedule tests from the same class to different workers. As a result, in a few tempest plugins, we had to break down several classes that contained too many tests into several classes to avoid this limitation.

Depending on what your situation is, there are 2 things that could help, however, neither of those helps if there are many tests in a single test class as all this scheduling happens on class level:

1. Timing data - stestr saves time data per tests and takes that time into account when running the tests again - try to run your tests twice and if the second run has more favourable results, you can leverage the timing data (reuse them in future runs)

2. Worker file - there is a way how to force the scheduler to schedule the tests per your custom regex using the "--worker-file" - https://stestr.readthedocs.io/en/latest/MANUAL.html#test-scheduling

Hopefully some of this helps,

On Tue, 1 Oct 2024 at 01:37, Thomas Goirand <zigo@debian.org> wrote:

Hi,

This is the 2nd time I'm opening a thread about it. Sorry for this, but
it really bothers me.

Today, running tempest, I got these stats from stestr:

- Worker 0 (132 tests) => 0:24:49.209885
- Worker 1 (100 tests) => 0:17:14.791845
- Worker 2 (124 tests) => 0:42:52.690906
- Worker 3 (189 tests) => 0:41:21.307241
- Worker 4 (159 tests) => 0:45:49.503031
- Worker 5 (143 tests) => 0:28:13.282371
- Worker 6 (156 tests) => 3:16:52.364976
- Worker 7 (103 tests) => 0:46:05.366089

So, thread #1 ran 17 minutes, sitting idle for the rest of the 3 hours
run of thread #6. While I thought stestr was deviding the number of
tests by the number of thread, I don't get why worker #1 only had 100
tests assigned.

All together, all tests could have been ran within maybe less than an
hour of time (rather than 3h16 above), if not-yet-ran-tests were
reassigned to idle treads.

I'm currently spending a lot of time on running tempest. Indeed, I'm
running tempest on each OpenStack upgrade, from Victoria up to
Dalmatian. With the current way stestr run, it may take 2 full days to
do that (and that's not even counting when upgrade will fail and will
need fixing...), when it could be done in maybe 8 hours (if I count 1h
per OpenStack release).

So, knowing the above, it might be a good use of my time to dig into
stestr and see if I can fix this... or not!

Does anyone have a suggestion for another test runner, that's compatible
with stestr, at least for the tests selection with a regular expression?
As much as I know, pytest cannot take a regexp for test selection, can
it? Or is there maybe a plugin for it?

If there's no compatible test runner, where should I dig in the stestr
code to rewrite things in a smarter way?

Cheers,

Thomas Goirand (zigo)

P.S: On Caracal today, I just had this:

==============
Worker Balance
==============
- Worker 0 (117 tests) => 0:59:32.291385
- Worker 1 (147 tests) => 1:22:28.228980
- Worker 2 (125 tests) => 0:45:20.969397
- Worker 3 (114 tests) => 1:46:21.667579
- Worker 4 (170 tests) => 0:45:27.577738
- Worker 5 (190 tests) => 2:28:39.744920
- Worker 6 (182 tests) => 2:29:01.402255
- Worker 7 (152 tests) => 2:29:10.183359

this looks better, but that's probably 1/ random 2/ still not perfect,
with worker #0, #2 and #4 doing nothing 2/3rd of the time.

Martin Kopec (kopecmartin)