<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 4, 2019 at 1:02 PM Adam Kimball <<a href="mailto:baha@linux.vnet.ibm.com">baha@linux.vnet.ibm.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello all,<br>
<br>
Mike Turek and I have been working to get a ppc64le job to build tripleo <br>
containers for quite some time now. A few weeks back, we stumbled into a <br>
strange issue where the containers build job was issuing a return code <br>
1, despite all containers successfully building. This has largely <br>
blocked our ability to push containers up to docker hub or report <br>
success through the delorean API.<br>
<br>
We've been pushing a number of patches to expand the logging of both our <br>
job, and the tripleo playbooks themselves, to get a better idea of <br>
what's going on. The most recent was a patch to show which containers <br>
have python futures that have been throwing exceptions, and why. [1]<br>
<br>
The end result of this seems to be that a number of jobs are reporting <br>
as incomplete. An example of this can be seen at timestamp 20:16:07 of <br>
an example build log. [2]<br>
<br>
However, upon checking the list of successfully built containers [3], or <br>
the RDO registry itself [4], one can see that the containers producing <br>
job not complete errors have actually built, and are being uploaded. The <br>
error log generated by the tripleo playbooks is also empty. [5]<br>
<br>
At this point, we're wondering what the path forward is. It seems like <br>
the issue stems from some unintended behavior in the tripleo playbooks <br>
themselves, not anything from our job. We're trying to figure out if <br>
this behavior is something that should be preventing us from reporting <br>
successful builds, and if so, how it can be fixed.<br>
<br>
Thanks,<br>
Adam Kimball<br>
<br>
<br>
[1] - <a href="https://review.opendev.org/#/c/695723/" rel="noreferrer" target="_blank">https://review.opendev.org/#/c/695723/</a><br>
[2] - <br>
<a href="https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1803/logs/logs/build.log" rel="noreferrer" target="_blank">https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1803/logs/logs/build.log</a></blockquote><div><br></div><div>In reviewing this, it looks like it's failing on the push to the rdo registry. Would it be possible to keep a node and manually see what a push command is doing? For example,</div><div><br></div><div><pre style="color:rgb(0,0,0);white-space:pre-wrap">sudo buildah push --tls-verify=False <a href="http://trunk.registry.rdoproject.org/tripleomaster/centos-binary-mistral-executor:36a84820e51bad57c6bbb92429f3afb3d9da29c2_6e3b098e_ppc64le">trunk.registry.rdoproject.org/tripleomaster/centos-binary-mistral-executor:36a84820e51bad57c6bbb92429f3afb3d9da29c2_6e3b098e_ppc64le</a> docker://<a href="http://trunk.registry.rdoproject.org/tripleomaster/centos-binary-mistral-executor:36a84820e51bad57c6bbb92429f3afb3d9da29c2_6e3b098e_ppc64le">trunk.registry.rdoproject.org/tripleomaster/centos-binary-mistral-executor:36a84820e51bad57c6bbb92429f3afb3d9da29c2_6e3b098e_ppc64le</a></pre></div><div> We output the commands that are being run so I'm wondering if there's some form of output that's getting eaten.</div><div><br></div><div>Thanks,</div><div>-Alex</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
[3] - <br>
<a href="https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1803/logs/logs/containers-successfully-built.log" rel="noreferrer" target="_blank">https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1803/logs/logs/containers-successfully-built.log</a><br>
[4] - <a href="https://console.registry.rdoproject.org/registry" rel="noreferrer" target="_blank">https://console.registry.rdoproject.org/registry</a><br>
[5] - <br>
<a href="https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1803/logs/logs/containers-build-errors.log" rel="noreferrer" target="_blank">https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1803/logs/logs/containers-build-errors.log</a><br>
<br>
</blockquote></div></div>