<div dir="ltr">Dear stackers,<br><br>FYI. Eventually I report this problem to libguestfs. A workaround has been included into libguestfs code to fix it. Thanks for your supporting!<br><a rel="nofollow" href="https://bugzilla.redhat.com/show_bug.cgi?id=1123007">https://bugzilla.redhat.com/show_bug.cgi?id=1123007</a><br>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Jun 7, 2014 at 3:27 AM, Qin Zhao <span dir="ltr"><<a href="mailto:chaochin@gmail.com" target="_blank">chaochin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Yuriy,<br><br>And I think if we use proxy object of multiprocessing, the green thread will not switch during we call libguestfs. Is that correct?<br>
</div><div class="gmail_extra"><div><div class="h5"><br><br><div class="gmail_quote">
On Fri, Jun 6, 2014 at 2:44 AM, Qin Zhao <span dir="ltr"><<a href="mailto:chaochin@gmail.com" target="_blank">chaochin@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><div><div>Hi Yuriy,<br><br></div>I read multiprocessing source code just now. Now I feel it may not solve this problem very easily. For example, let us assume that we will use the proxy object in Manager's process to call libguestfs. In manager.py, I see it needs to create a pipe, before fork the child process. The write end of this pipe is required by child process.<br>
<br><a href="http://sourcecodebrowser.com/python-multiprocessing/2.6.2.1/classmultiprocessing_1_1managers_1_1_base_manager.html#a57fe9abe7a3d281286556c4bf3fbf4d5" target="_blank">http://sourcecodebrowser.com/python-multiprocessing/2.6.2.1/classmultiprocessing_1_1managers_1_1_base_manager.html#a57fe9abe7a3d281286556c4bf3fbf4d5</a><br>
<br></div>And in Process._bootstrp(), I think we will need to register a function to be called by _run_after_forkers(), in order to closed the fds inherited from Nova process.<br><br><a href="http://sourcecodebrowser.com/python-multiprocessing/2.6.2.1/classmultiprocessing_1_1process_1_1_process.html#ae594800e7bdef288d9bfbf8b79019d2e" target="_blank">http://sourcecodebrowser.com/python-multiprocessing/2.6.2.1/classmultiprocessing_1_1process_1_1_process.html#ae594800e7bdef288d9bfbf8b79019d2e</a><br>
<br>And we also can not close the write end fd created by Manager in _run_after_forkers(). One feasible way may be getting that fd from the 5th element of _args attribute of Process object, then skip to close this fd.... I have not investigate if or not Manager need to use other fds, besides this pipe. Personally, I feel such an implementation will be a little tricky and risky, because it tightly depends on Manager code. If Manager opens other files, or change the argument order, our code will fail to run. Am I wrong? Is there any other safer way?<br>
</div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 5, 2014 at 11:40 PM, Yuriy Taraday <span dir="ltr"><<a href="mailto:yorik.sar@gmail.com" target="_blank">yorik.sar@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Please take a look at <a href="https://docs.python.org/2.7/library/multiprocessing.html#managers" target="_blank">https://docs.python.org/2.7/library/multiprocessing.html#managers</a> - everything is already implemented there.<div>
All you need is to start one manager that would serve all your requests to libguestfs. The implementation in stdlib will provide you with all exceptions and return values with minimum code changes on Nova side.</div><div>
Create a new Manager, register an libguestfs "endpoint" in it and call start(). It will spawn a separate process that will speak with calling process over very simple RPC.</div><div>From the looks of it all you need to do is replace tpool.Proxy calls in VFSGuestFS.setup method to calls to this new Manager.</div>
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 5, 2014 at 7:21 PM, Qin Zhao <span dir="ltr"><<a href="mailto:chaochin@gmail.com" target="_blank">chaochin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Yuriy,<br><br></div>Thanks for reading my bug! You are right. Python 3.3 or 3.4 should not have this issue, since they have can secure the file descriptor. Before OpenStack move to Python 3, we may still need a solution. Calling libguestfs in a separate process seems to be a way. This way, Nova code can close those fd by itself, not depending upon CLOEXEC. However, that will be an expensive solution, since it requires a lot of code change. At least we need to write code to pass the return value and exception between these two processes. That will make this solution very complex. Do you agree?<br>
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 5, 2014 at 9:39 PM, Yuriy Taraday <span dir="ltr"><<a href="mailto:yorik.sar@gmail.com" target="_blank">yorik.sar@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">This behavior of os.pipe() has changed in Python 3.x so it won't be an issue on newer Python (if only it was accessible for us).<div>
<br><div>From the looks of it you can mitigate the problem by running libguestfs requests in a separate process (multiprocessing.managers comes to mind). This way the only descriptors child process could theoretically inherit would be long-lived pipes to main process although they won't leak because they should be marked with CLOEXEC before any libguestfs request is run. The other benefit is that this separate process won't be busy opening and closing tons of fds so the problem with inheriting will be avoided.</div>
</div></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">On Thu, Jun 5, 2014 at 2:17 PM, laserjetyang <span dir="ltr"><<a href="mailto:laserjetyang@gmail.com" target="_blank">laserjetyang@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<table width="550"><tbody><tr><td>Will this patch of Python fix your problem? <b><a href="http://bugs.python.org/issue7213" target="_blank">http://bugs.python.org/issue7213</a></b>
</td></tr></tbody><tbody></tbody></table><div><div><br><br>
<div class="gmail_quote">On Wed, Jun 4, 2014 at 10:41 PM, Qin Zhao <span dir="ltr"><<a href="mailto:chaochin@gmail.com" target="_blank">chaochin@gmail.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div dir="ltr">
<div>
<div>Hi Zhu Zhu,<br><br></div>Thank you for reading my diagram! I need to clarify that this problem does not occur during data injection. Before creating the ISO, the driver code will extend the disk. Libguestfs is invoked in that time frame.<br>
<br></div>
<div>And now I think this problem may occur at any time, if the code use tpool to invoke libguestfs, and one external commend is executed in another green thread simultaneously. Please correct me if I am wrong.<br><br></div>
<div>I think one simple solution for this issue is to call libguestfs routine in greenthread, rather than another native thread. But it will impact the performance very much. So I do not think that is an acceptable solution.<br>
<br></div></div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">
<div>
<div>On Wed, Jun 4, 2014 at 12:00 PM, Zhu Zhu <span dir="ltr"><<a href="mailto:bjzzu.zz@gmail.com" target="_blank">bjzzu.zz@gmail.com</a>></span> wrote:<br></div></div>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div>
<div>
<div>
<div><span></span>Hi Qin Zhao,</div>
<div><br></div>
<div><span style="LINE-HEIGHT:1.5;BACKGROUND-COLOR:window;FONT-SIZE:10.5pt">Thanks for raising this issue and analysis. </span>According to the issue description and happen scenario(<a style="LINE-HEIGHT:1.5;BACKGROUND-COLOR:window;FONT-SIZE:10.5pt" href="https://docs.google.com/drawings/d/1pItX9urLd6fmjws3BVovXQvRg_qMdTHS-0JhYfSkkVc/pub?w=960&h=720" target="_blank">https://docs.google.com/drawings/d/1pItX9urLd6fmjws3BVovXQvRg_qMdTHS-0JhYfSkkVc/pub?w=960&h=720</a>)<span style="LINE-HEIGHT:1.5;BACKGROUND-COLOR:window;FONT-SIZE:10.5pt">, </span><span style="LINE-HEIGHT:1.5;BACKGROUND-COLOR:window;FONT-SIZE:10.5pt">if that's the case, concurrent mutiple KVM spawn instances(</span><b style="LINE-HEIGHT:1.5;FONT-SIZE:10.5pt">with both config drive and data injection enabled</b><span style="LINE-HEIGHT:1.5;BACKGROUND-COLOR:window;FONT-SIZE:10.5pt">) are triggered, the issue can be very likely to happen. </span></div>
<div>As in libvirt/driver.py <span style="LINE-HEIGHT:1.5;FONT-SIZE:10.5pt">_create_image method</span><span style="LINE-HEIGHT:1.5;BACKGROUND-COLOR:window;FONT-SIZE:10.5pt">, right after iso making "</span><span style="LINE-HEIGHT:1.5;FONT-SIZE:10.5pt">cdb.make_drive", the driver will attempt "data injection" which will call the libguestfs launch in another thread. </span></div>
<div><span style="LINE-HEIGHT:1.5;BACKGROUND-COLOR:window;FONT-SIZE:10.5pt"><br></span></div>
<div><span style="LINE-HEIGHT:1.5;BACKGROUND-COLOR:window;FONT-SIZE:10.5pt">Looks there were also a couple of libguestfs hang issues from Launch pad as below. . I am not sure if libguestfs itself can have certain mechanism to free/close the fds that inherited from parent process instead of require explicitly calling the tear down. Maybe open a defect to libguestfs to see what their thoughts? </span></div>
<div><span style="LINE-HEIGHT:1.5;BACKGROUND-COLOR:window;FONT-SIZE:10.5pt"><br></span></div>
<div>
<div><span><a href="https://bugs.launchpad.net/nova/+bug/1286256" target="_blank">https://bugs.launchpad.net/nova/+bug/1286256</a></span></div><span></span></div>
<div><a style="LINE-HEIGHT:1.5;BACKGROUND-COLOR:window;FONT-SIZE:10.5pt" href="https://bugs.launchpad.net/nova/+bug/1270304" target="_blank">https://bugs.launchpad.net/nova/+bug/1270304</a><span style="LINE-HEIGHT:1.5;FONT-SIZE:10.5pt"> </span></div>
<div><br></div>
<hr style="MIN-HEIGHT:1px;WIDTH:210px" color="#b5c4df" align="left" size="1">
<div><span>
<div style="MARGIN:10px;FONT-FAMILY:verdana;FONT-SIZE:10pt">
<div>Zhu Zhu</div>
<div>Best Regards</div></div></span></div>
<blockquote style="MARGIN-TOP:0px;MARGIN-BOTTOM:0px;MARGIN-LEFT:2em">
<div> </div>
<div style="BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0cm;PADDING-LEFT:0cm;PADDING-RIGHT:0cm;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<div style="PADDING-BOTTOM:8px;PADDING-LEFT:8px;PADDING-RIGHT:8px;FONT-FAMILY:tahoma;BACKGROUND:#efefef;COLOR:#000000;FONT-SIZE:12px;PADDING-TOP:8px">
<div><b>From:</b> <a href="mailto:chaochin@gmail.com" target="_blank">Qin Zhao</a></div>
<div><b>Date:</b> <a href="tel:2014-05-31%C2%A001" value="+12014053101" target="_blank">2014-05-31 01</a>:25</div>
<div>
<div><b>To:</b> <a href="mailto:openstack-dev@lists.openstack.org" target="_blank">OpenStack Development Mailing List (not for usage questions)</a></div>
<div><b>Subject:</b> [openstack-dev] [Nova] nova-compute deadlock</div></div></div></div>
<div>
<div>
<div>
<div>
<div dir="ltr">
<div>
<div>Hi all,<br><br></div>When I run Icehouse code, I encountered a strange problem. The nova-compute service becomes stuck, when I boot instances. I report this bug in <a href="https://bugs.launchpad.net/nova/+bug/1313477" target="_blank">https://bugs.launchpad.net/nova/+bug/1313477</a>.<br>
<br></div>After thinking several days, I feel I know its root cause. This bug should be a deadlock problem cause by pipe fd leaking. I draw a diagram to illustrate this problem. <a href="https://docs.google.com/drawings/d/1pItX9urLd6fmjws3BVovXQvRg_qMdTHS-0JhYfSkkVc/pub?w=960&h=720" target="_blank">https://docs.google.com/drawings/d/1pItX9urLd6fmjws3BVovXQvRg_qMdTHS-0JhYfSkkVc/pub?w=960&h=720</a><br clear="all">
<div>
<div>
<div>
<div>
<div><br></div>
<div>However, I have not find a very good solution to prevent this deadlock. This problem is related with Python runtime, libguestfs, and eventlet. The situation is a little complicated. Is there any expert who can help me to look for a solution? I will appreciate for your help!<br>
</div>
<div><br>-- <br>
<div dir="ltr">Qin Zhao<br></div></div></div></div></div></div></div></div></div></div></div></blockquote></div><br></div></div>_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br><br></blockquote></div><span><font color="#888888"><br>
<br clear="all"><br>-- <br>
<div dir="ltr">Qin Zhao<br></div></font></span></div><br>_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br><br></blockquote></div><br>
</div></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><br></div></div><div>Kind regards, Yuriy.</div>
</div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div dir="ltr">Qin Zhao<br></div>
</div>
</div></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><br><div>Kind regards, Yuriy.</div>
</div>
</div></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div dir="ltr">Qin Zhao<br></div>
</div>
</div></div></blockquote></div><br><br clear="all"><br></div></div><span class="HOEnZb"><font color="#888888">-- <br><div dir="ltr">Qin Zhao<br></div>
</font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br><div dir="ltr">Qin Zhao<br></div>
</div>