<div dir="ltr"><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Mar 10, 2014 at 7:20 PM, Zane Bitter <span dir="ltr"><<a href="mailto:zbitter@redhat.com" target="_blank">zbitter@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="">On 10/03/14 16:04, Clark Boylan wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
On Mon, Mar 10, 2014 at 11:31 AM, Zane Bitter <<a href="mailto:zbitter@redhat.com" target="_blank">zbitter@redhat.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Thanks Clark for this great write-up. However, I think the solution to the<br>
problem in question is richer commands and better output formatting, not<br>
discarding information.<br>
<br>
<br>
On 07/03/14 16:30, Clark Boylan wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
But running tests in parallel introduces some fun problems. Like where<br>
do you send logging and stdout output. If you send it to the console<br>
it will be interleaved and essentially useless. The solution to this<br>
problem (for which I am probably to blame) is to have each test<br>
collect the logging, stdout, and stderr associated to that test and<br>
attach it to that tests subunit reporting. This way you get all of the<br>
output associated with a single test attached to that test and don't<br>
have crazy interleaving that needs to be demuxed. The capturing of<br>
</blockquote>
<br>
<br>
This is not really a problem unique to parallel test runners. Printing to<br>
the console is just not a great way to handle stdout/stderr in general<br>
because it messes up the output of the test runner, and nose does exactly<br>
the same thing as testr in collecting them - except that nose combines<br>
messages from the 3 sources and prints the output for human consumption,<br>
rather than in separate groups surrounded by lots of {{{random braces}}}.<br>
<br>
</blockquote>
Except nose can make them all the same file descriptor and let<br>
everything multiplex together. Nose isn't demuxing arbitrary numbers<br>
of file descriptors from arbitrary numbers of processes.<br>
</blockquote>
<br></div>
Can't each subunit process do the same thing?<br>
<br>
As a user, here's how I want it to work:<br>
- Each subunit process works like nose - multiplexing the various streams of output together and associating it with a particular test - except that nothing is written to the console but instead returned to testr in subunit format.<br>
- testr reads the subunit data and saves it to the test repository.<br>
- testr prints a report to the console based on the data it just received/saved.<br>
<br>
How it actually seems to work:<br>
- A magic pixie creates a TestCase class with a magic incantation to capture your stdout/stderr/logging without breaking other test runners.<br>
- Or they don't! You're hosed. The magic incantation is undocumented.<br>
- You change all of your TestCases to inherit from the class with the magic pixie dust.<br>
- Each subunit process associates the various streams of output (if you set it up to) with a particular test, but keeps them separate so that if you want to figure out the order of events you have to direct them all to the same channel - which, in practice, means you can only use logging (since some of the events you are interested in probably already exist in the code as logs).<br>
- when you want to debug a test, you have to all the tedious loigging setup if it doesn't already exist in the file. It probably won't, because flake8 would have made you delete it unless it's being used already.<br>
- testr reads the subunit data and saves it to the test repository.<br>
- testr prints a report to the console based on the data it just received/saved, though parts of it look like a raw data dump.<br>
<br>
While there may be practical reasons why it currently works like the latter, I would submit that there is no technical reason it could not work like the former. In particular, there is nothing about the concept of running the tests in parallel that would prevent it, just as there is nothing about what nose does that would prevent two copies of nose from running at the same time on different sets of tests.<div class="">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
this data is toggleable in the test suite using environment variables<br>
and is off by default so that when you are not using testr you don't<br>
get this behavior [0]. However we seem to have neglected log capture<br>
toggles.<br>
</blockquote>
<br>
<br>
Oh wow, there is actually a way to get the stdout and stderr? Fantastic! Why<br>
on earth are these disabled?<br>
<br>
</blockquote>
See above, testr has to deal with multiple writers to stdout and<br>
stderr, you really don't want them all going to the same place when<br>
using testr (which is why stdout and stderr are captured when running<br>
testr but not otherwise).<br>
</blockquote>
<br></div>
Ah, OK, I think I understand now. testr passes the environment variables automatically, so you only have to know the magic incantation at the time you're writing the test, not when you're running it.<div class="">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Please, please, please don't turn off the logging too. That's the only tool<br>
left for debugging now that stdout goes into a black hole.<br>
<br>
</blockquote>
Logging goes into the same "black hole" today, I am suggesting that we<br>
make this toggleable like we have made stdout and stderr capturing<br>
toggleable. FWIW this isn't a black hole it is all captured on disk<br>
and you can refer back to it at any time (the UI around doing this<br>
could definitely be better though).<br>
</blockquote>
<br></div>
Hmm, now that you mention it, I remember Clint did the setup work in Heat to get the logging working. So maybe we have to do the same for stdout and stderr. At the moment they really do go into a black hole - we can't see them in the testr output at all and nor are they stored in the repository on disk.<br>
</blockquote><div><br></div><div><div class="gmail_default" style="font-size:small">If you're going to do this in your project, would you contribute the patch to oslo.test as well (or instead)?</div><div class="gmail_default" style="font-size:small">
<br></div><div class="gmail_default"><a href="http://git.openstack.org/cgit/openstack/oslo.test">http://git.openstack.org/cgit/openstack/oslo.test</a><br></div><div class="gmail_default" style="font-size:small"><br></div>
<div class="gmail_default" style="font-size:small">Doug</div><div class="gmail_default" style="font-size:small"> <br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
It just seems bizarre to me that the _tests_ have to figure out what test runner they are being run by and redirect their output to the correct location to oblige it. Surely the point of a test runner is to do the Right Thing(TM) for any test, regardless of what it knows about the test runner.<div class="">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
BTW, since I'm on the subject, testr would be a lot more<br>
confidence-inspiring if running `testr failing` immediately after running<br>
`testr` reported the same number of failures, or indeed if running `testr`<br>
twice in a row was guaranteed to report the same number of failures<br>
(assuming that the tests themselves are deterministic). I can't imagine why<br>
one of the subunit processes reporting a test failure is considered a<br>
failure in itself (that's what it's supposed to do!), particularly when<br>
`testr failing` manages to filter them out OK.<br>
<br>
</blockquote>
IIRC the test runner exiting non zero is a failure in itself to handle<br>
cases where test runners die a horrible death without outputting any<br>
failed subunit data. It could probably be cleaner but this way it is<br>
preserved in the subunit log.<br>
</blockquote>
<br></div>
It makes sense for the case where the test runner has died without reporting data, but why should it be reported as a separate failure when it has reported data that testr has regarded as valid enough to use and display?<br>
<br>
My machine has 4 cores with hyperthreads (a common configuration, I would imagine), so I have to do this mental translation every time:<br>
<br>
1 failure -> everything failed<br>
2 failures -> 1 failure<br>
3 failures -> 2 failures<br>
4 failures -> 2-3 failures<br>
5 failures -> 3-4 failures<br>
6 failures -> 3-5 failures<br>
7 failures -> 4-6 failures<br>
8 failures -> 4-7 failures<br>
9 failures -> 5-8 failures<br>
10 failures -> 5-9 failures<br>
11 failures -> 6-10 failures<br>
12 failures -> 6-11 failures<br>
13 failures -> 7-12 failures<br>
14 failures -> 7-13 failures<br>
15 failures -> 8-14 failures<br>
16 failures -> 8-15 failures<br>
17 failures -> 9-16 failures<br>
18 failures -> 10-17 failures<br>
n>18 failures -> (n-8)-(n-1) failures<br>
<br>
This means that the change statistic that testr prints is completely useless for determining what I really needed to know, which is whether the thing I just modified fixed any tests or not (or, alternatively, whether I have an unstable test). For low-ish numbers of test failures (say <15), it's dominated by random noise. Call me old-fashioned, but random noise is not the distribution I'm looking for in reports from my test runner :p<div class="">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
(I understand that some of the things mentioned above may already have been<br>
improved since the latest 0.0.18 release. I can't actually find the repo at<br>
the moment to check, because it's not linked from PyPI or Launchpad and<br>
'testrepository' turns out to be an extremely common name for things on the<br>
Internet.)<br>
<br>
</blockquote>
<a href="https://launchpad.net/testrepository" target="_blank">https://launchpad.net/<u></u>testrepository</a> is linked from pypi....<br>
</blockquote>
<br></div>
Ah, thanks. I looked at that page, but in my head I thought I had once seen a Git repo for it and I couldn't see a link from there... I never guessed that the upstream was in Launchpad's bzr.<br>
<br>
cheers,<br>
Zane.<div class=""><div class="h5"><br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>