<div dir="ltr"><div>Just to summarize for those in the future who might run into this issue.</div><div><br></div>Discussed with Darragh over IRC and he came up with a fix for this issue [1]. Once the patch is merged it should be available in the next release.<div><br></div><div>In the meantime if you encounter this issue the workaround is to pass the "-o /path/to/dir" option to the test command to redirect the output to a directory.</div><div><br></div><div>Thanh</div><div><div><br></div><div>[1] <a href="https://review.openstack.org/265959">https://review.openstack.org/265959</a></div><div class="gmail_extra"><br><div class="gmail_quote">On 11 January 2016 at 12:51, Darragh Bailey <span dir="ltr"><<a href="mailto:daragh.bailey@gmail.com" target="_blank">daragh.bailey@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi Thanh<br>
<span class=""><br>
On 11 January 2016 at 16:03, Thanh Ha <<a href="mailto:thanh.ha@linuxfoundation.org">thanh.ha@linuxfoundation.org</a>> wrote:<br>
> Hi Darragh,<br>
><br>
> The extremely strange thing about this one is I've only been able to<br>
> reproduce it in Jenkins. So imagine having JJB installed on the same server<br>
> as your Jenkins instance or slave. If you ssh to the machine and run JJB on<br>
> the commandline it works. Create a job running on the same system and<br>
> Jenkins will see the failure in console logs.<br>
><br>
> In case it helps with reproducing. The git repo I've been using to reproduce<br>
> this error is this one:<br>
><br>
> <a href="https://git.opendaylight.org/gerrit/#/admin/projects/releng/builder" rel="noreferrer" target="_blank">https://git.opendaylight.org/gerrit/#/admin/projects/releng/builder</a><br>
><br>
> After cloning this repo run "jenkins-jobs test -r jjb/".<br>
><br>
> Regards,<br>
><br>
> Thanh<br>
><br>
><br>
<br>
</span>This triggered a suspicion with me that it wasn't anything to do with<br>
the project, and it's something to do with how jenkins is reading the<br>
console output from processes which causes python to use a different<br>
object for stdout.<br>
<br>
I managed to reproduce by creating a local pipe, tailed from that in<br>
one terminal and run the above command in another and redirecting the<br>
output into the pipe.<br>
<br>
term 1:<br>
mkfifo /tmp/test_jjb_output_to_pipe<br>
tail -f /tmp/test_jjb_output_to_pipe<br>
<br>
term 2:<br>
jenkins-jobs test -r builder/jjb > /tmp/test_jjb_output_to_pipe<br>
<br>
<br>
However on closer examination of the code, I suspect that the<br>
following piece is a little stupid and is causing the problem. Quite<br>
why it works some of the time and doesn't fail consistently has me<br>
stumped.<br>
<br>
if output:<br>
for job in self.parser.xml_jobs:<br>
if hasattr(output, 'write'):<br>
# `output` is a file-like object<br>
<a href="http://logger.info" rel="noreferrer" target="_blank">logger.info</a>("Job name: %s", <a href="http://job.name" rel="noreferrer" target="_blank">job.name</a>)<br>
logger.debug("Writing XML to '{0}'".format(output))<br>
output = utils.wrap_stream(output)<br>
try:<br>
output.write(job.output())<br>
except IOError as exc:<br>
if exc.errno == errno.EPIPE:<br>
# EPIPE could happen if piping output to something<br>
# that doesn't read the whole input (e.g.: the UNIX<br>
# `head` command)<br>
return<br>
raise<br>
continue<br>
<br>
This ends up wrapping the object on each an every loop iteration, so<br>
if it's writing to stdout it wraps the output many, many times. I'm<br>
going to fix that.<br>
<br>
The reason why it may only occur when using a named pipe possibly has<br>
something to do with the object writing to it not being able to set<br>
the encoding, so therefore it keeps trying to wrap it with<br>
"codecs.EncodedFile(stream, encoding, stream_enc)", where as for<br>
normal pipes and stdout, the encoding is set and read correctly and<br>
therefore it just returns the stream on subsequent iterations.<br>
<br>
Wrapping a StreamReader from codecs with a subsequent StreamReader<br>
object, would eventually result in the getattr call hitting the<br>
recursion limit as it tries to find the base object attribute to read.<br>
<br>
<br>
Should be able to knock together a sensible solution relatively easily.<br>
<div class=""><div class="h5"><br>
<br>
--<br>
Darragh Bailey<br>
"Nothing is foolproof to a sufficiently talented fool"<br>
</div></div></blockquote></div><br></div></div></div>