<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace;font-size:small">​another issue we(kolla gate) meet is libvirtd got segment fault.</div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small">Here is the log from host messages file. But i do not get other issue in nova.conf expect can not connect libvirt.</div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small"><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small"><span style="color:rgb(0,0,0);white-space:pre-wrap;font-family:arial,sans-serif">Dec 25 13:40:54 localhost kernel: libvirtd[5720]: segfault at 8 ip 00007fa70379b823 sp 00007fa6faa061c0 error 4 in libvirt.so.0.2000.0[<wbr>7fa703648000+353000]</span>​</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 23, 2016 at 10:00 PM, Neil Jerram <span dir="ltr"><<a href="mailto:neil@tigera.io" target="_blank">neil@tigera.io</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><div class="h5"><div dir="ltr">On Fri, Dec 23, 2016 at 1:37 PM Steve Gordon <<a href="mailto:sgordon@redhat.com" target="_blank">sgordon@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">----- Original Message -----<br class="m_8127664939847850212gmail_msg">
> From: "Neil Jerram" <<a href="mailto:neil@tigera.io" class="m_8127664939847850212gmail_msg" target="_blank">neil@tigera.io</a>><br class="m_8127664939847850212gmail_msg">
> To: "OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" class="m_8127664939847850212gmail_msg" target="_blank">openstack-dev@lists.<wbr>openstack.org</a>><br class="m_8127664939847850212gmail_msg">
><br class="m_8127664939847850212gmail_msg">
> I appreciate that even libvirt 2.0.0 will be ancient history by now, to its<br class="m_8127664939847850212gmail_msg">
> developers, but I am seeing further issues that look associated with the<br class="m_8127664939847850212gmail_msg">
> recent CentOS 7 transition from libvirt 1.2.7 to libvirt 2.0.0, and would<br class="m_8127664939847850212gmail_msg">
> appreciate any comments on them that people may have.  I believe these<br class="m_8127664939847850212gmail_msg">
> issues are independent of those that have already been discussed on other<br class="m_8127664939847850212gmail_msg">
> threads.<br class="m_8127664939847850212gmail_msg">
><br class="m_8127664939847850212gmail_msg">
> First, this traceback in nova-compute.log:<br class="m_8127664939847850212gmail_msg">
><br class="m_8127664939847850212gmail_msg">
> Traceback (most recent call last):<br class="m_8127664939847850212gmail_msg">
>   File "/usr/lib/python2.7/site-<wbr>packages/nova/compute/manager.<wbr>py", line<br class="m_8127664939847850212gmail_msg">
> 2156, in _build_resources<br class="m_8127664939847850212gmail_msg">
>     yield resources<br class="m_8127664939847850212gmail_msg">
>   File "/usr/lib/python2.7/site-<wbr>packages/nova/compute/manager.<wbr>py", line<br class="m_8127664939847850212gmail_msg">
> 2009, in _build_and_run_instance<br class="m_8127664939847850212gmail_msg">
>     block_device_info=block_<wbr>device_info)<br class="m_8127664939847850212gmail_msg">
>   File "/usr/lib/python2.7/site-<wbr>packages/nova/virt/libvirt/<wbr>driver.py", line<br class="m_8127664939847850212gmail_msg">
> 2534, in spawn<br class="m_8127664939847850212gmail_msg">
>     block_device_info=block_<wbr>device_info)<br class="m_8127664939847850212gmail_msg">
>   File "/usr/lib/python2.7/site-<wbr>packages/nova/virt/libvirt/<wbr>driver.py", line<br class="m_8127664939847850212gmail_msg">
> 4620, in _create_domain_and_network<br class="m_8127664939847850212gmail_msg">
>     xml, pause=pause, power_on=power_on)<br class="m_8127664939847850212gmail_msg">
>   File "/usr/lib/python2.7/site-<wbr>packages/nova/virt/libvirt/<wbr>driver.py", line<br class="m_8127664939847850212gmail_msg">
> 4550, in _create_domain<br class="m_8127664939847850212gmail_msg">
>     guest.launch(pause=pause)<br class="m_8127664939847850212gmail_msg">
>   File "/usr/lib/python2.7/site-<wbr>packages/nova/virt/libvirt/<wbr>guest.py", line<br class="m_8127664939847850212gmail_msg">
> 142, in launch<br class="m_8127664939847850212gmail_msg">
>     self._encoded_xml, errors='ignore')<br class="m_8127664939847850212gmail_msg">
>   File "/usr/lib/python2.7/site-<wbr>packages/oslo_utils/excutils.<wbr>py", line 195,<br class="m_8127664939847850212gmail_msg">
> in __exit__<br class="m_8127664939847850212gmail_msg">
>     six.reraise(self.type_, self.value, self.tb)<br class="m_8127664939847850212gmail_msg">
>   File "/usr/lib/python2.7/site-<wbr>packages/nova/virt/libvirt/<wbr>guest.py", line<br class="m_8127664939847850212gmail_msg">
> 137, in launch<br class="m_8127664939847850212gmail_msg">
>     return self._domain.createWithFlags(<wbr>flags)<br class="m_8127664939847850212gmail_msg">
>   File "/usr/lib/python2.7/site-<wbr>packages/eventlet/tpool.py", line 186, in<br class="m_8127664939847850212gmail_msg">
> doit<br class="m_8127664939847850212gmail_msg">
>     result = proxy_call(self._autowrap, f, *args, **kwargs)<br class="m_8127664939847850212gmail_msg">
>   File "/usr/lib/python2.7/site-<wbr>packages/eventlet/tpool.py", line 144, in<br class="m_8127664939847850212gmail_msg">
> proxy_call<br class="m_8127664939847850212gmail_msg">
>     rv = execute(f, *args, **kwargs)<br class="m_8127664939847850212gmail_msg">
>   File "/usr/lib/python2.7/site-<wbr>packages/eventlet/tpool.py", line 125, in<br class="m_8127664939847850212gmail_msg">
> execute<br class="m_8127664939847850212gmail_msg">
>     six.reraise(c, e, tb)<br class="m_8127664939847850212gmail_msg">
>   File "/usr/lib/python2.7/site-<wbr>packages/eventlet/tpool.py", line 83, in<br class="m_8127664939847850212gmail_msg">
> tworker<br class="m_8127664939847850212gmail_msg">
>     rv = meth(*args, **kwargs)<br class="m_8127664939847850212gmail_msg">
>   File "/usr/lib64/python2.7/site-<wbr>packages/libvirt.py", line 1065, in<br class="m_8127664939847850212gmail_msg">
> createWithFlags<br class="m_8127664939847850212gmail_msg">
>     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed',<br class="m_8127664939847850212gmail_msg">
> dom=self)<br class="m_8127664939847850212gmail_msg">
> libvirtError: Cannot find '' in path: No such file or directory<br class="m_8127664939847850212gmail_msg">
><br class="m_8127664939847850212gmail_msg">
> which I believe is caused by the empty path attribute in this part of the<br class="m_8127664939847850212gmail_msg">
> XML:<br class="m_8127664939847850212gmail_msg">
><br class="m_8127664939847850212gmail_msg">
>     <interface type='ethernet'><br class="m_8127664939847850212gmail_msg">
>       <mac address='fa:16:3e:3c:96:33'/><br class="m_8127664939847850212gmail_msg">
>       <script path=''/><br class="m_8127664939847850212gmail_msg">
>       <target dev='tap06992dfb-5d'/><br class="m_8127664939847850212gmail_msg">
>       <model type='virtio'/><br class="m_8127664939847850212gmail_msg">
>       <driver name='qemu'/><br class="m_8127664939847850212gmail_msg">
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'<br class="m_8127664939847850212gmail_msg">
> function='0x0'/><br class="m_8127664939847850212gmail_msg">
>     </interface><br class="m_8127664939847850212gmail_msg">
><br class="m_8127664939847850212gmail_msg">
> which is in turn caused, I think, by<br class="m_8127664939847850212gmail_msg">
> <a href="https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L61" rel="noreferrer" class="m_8127664939847850212gmail_msg" target="_blank">https://github.com/openstack/<wbr>nova/blob/master/nova/virt/<wbr>libvirt/designer.py#L61</a><br class="m_8127664939847850212gmail_msg">
><br class="m_8127664939847850212gmail_msg">
> Is it plausible that libvirt 1.2.7 would have avoided trying to invoke a<br class="m_8127664939847850212gmail_msg">
> script with an empty path, whereas libvirt 2.0.0 does not?<br class="m_8127664939847850212gmail_msg">
<br class="m_8127664939847850212gmail_msg">
It seems someone filed this as <a href="https://bugs.launchpad.net/nova/+bug/1649527" rel="noreferrer" class="m_8127664939847850212gmail_msg" target="_blank">https://bugs.launchpad.net/<wbr>nova/+bug/1649527</a> from a Nova POV. The Libvirt folks helped me narrow this down to this commit being the first one that would have exhibited this behavior:<br class="m_8127664939847850212gmail_msg">
<br class="m_8127664939847850212gmail_msg">
    <a href="https://libvirt.org/git/?p=libvirt.git;a=commit;h=9c17d665fdc5f0ab74500a14c30627014c11b2c0" rel="noreferrer" class="m_8127664939847850212gmail_msg" target="_blank">https://libvirt.org/git/?p=<wbr>libvirt.git;a=commit;h=<wbr>9c17d665fdc5f0ab74500a14c30627<wbr>014c11b2c0</a><br class="m_8127664939847850212gmail_msg">
<br class="m_8127664939847850212gmail_msg">
Michal Privoznik provided some additional context:<br class="m_8127664939847850212gmail_msg">
<br class="m_8127664939847850212gmail_msg">
"Previously, libvirt merely just appended 'script=' onto qemu cmd line<br class="m_8127664939847850212gmail_msg">
according to what <script path=''/> contained letting qemu execute the<br class="m_8127664939847850212gmail_msg">
script. That was flawed from security POV (we don't want qemu to be<br class="m_8127664939847850212gmail_msg">
allowed to execute anything) thus libvirt is executing the script now.<br class="m_8127664939847850212gmail_msg">
However, we obviously forgot about this corner case."<br class="m_8127664939847850212gmail_msg">
<br class="m_8127664939847850212gmail_msg">
This actually apparently first appeared in v1.3.3-rc1~71, so it's not new in Libvirt 2.0.0 *but*:<br class="m_8127664939847850212gmail_msg">
<br class="m_8127664939847850212gmail_msg">
* The Ubuntu-based gate is apparently running v1.3.1 at the moment.<br class="m_8127664939847850212gmail_msg">
* The RHEL 7.2 and aligned CentOS included v1.2.17.<br class="m_8127664939847850212gmail_msg">
<br class="m_8127664939847850212gmail_msg">
This is at least partially why this was not seen until recently,</blockquote><div><br></div></div></div><div>Thanks for pinning all of that down!<br> <br></div><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> but it also seems like it might be specific to certain guest networking setup.</blockquote><div><br></div></span><div>Yes, there are only a few networking backends that use <interface type=ethernet>, I believe; mine is networking-calico, which uses VIF_TYPE_TAP; I think (from code reading) that the others are 'ivs', 'iovisor', 'midonet' and 'vrouter'.  (Although another thing to keep in mind here is that os-vif should soon be taking over this area of code - so any fixes in nova/virt/libvirt may also be needed in os-vif.)<br><br><br></div><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> You mentioned you have multiple interfaces attached to the guest, which VIF types and associated Neutron plugins are you using?<br class="m_8127664939847850212gmail_msg"></blockquote><div><br></div></span><div>Note that the multiple interfaces issue is an additional one.  The empty path issue causes all VIF_TYPE_TAP tests to fail, whether with just a single VM interface, or multiple VM interfaces.  I fixed most of those by patching my Nova locally (as in <a href="https://review.openstack.org/#/c/411936" target="_blank">https://review.openstack.org/#<wbr>/c/411936</a>), but am still seeing failures in tests with multiple interfaces, which I haven't yet investigated further.<br><br></div><div>Regards,<br></div><div>      Neil<br><br></div><span class=""><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="m_8127664939847850212gmail_msg">
Thanks,<br class="m_8127664939847850212gmail_msg">
<br class="m_8127664939847850212gmail_msg">
Steve<br class="m_8127664939847850212gmail_msg">
<br class="m_8127664939847850212gmail_msg">
______________________________<wbr>______________________________<wbr>______________<br class="m_8127664939847850212gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="m_8127664939847850212gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="m_8127664939847850212gmail_msg" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br class="m_8127664939847850212gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="m_8127664939847850212gmail_msg" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br class="m_8127664939847850212gmail_msg">
</blockquote></span></div></div>
<br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><span style="font-size:13px;border-collapse:collapse"><font face="monospace, monospace">Regards,</font></span></div><div><span style="font-size:13px;border-collapse:collapse"><font face="monospace, monospace">Jeffrey Zhang</font></span></div><div><span style="font-family:monospace,monospace;font-size:12.8px">Blog: </span><a href="http://xcodest.me/" style="font-family:monospace,monospace;font-size:12.8px" target="_blank">http://xcodest.me</a><font face="monospace, monospace"><br></font></div></div></div></div></div></div></div></div></div>
</div>