Hello all!<br><br>I'm the original author of the hairpin patch, and things have changed a little bit in Essex and Folsom from the original Diablo target. I believe I can shed some light on what should be done here to solve the issue in either case.<br>
<br>---<br>For Essex (stable/essex), in nova/virt/libvirt/connection.py:<br>---<br><br>Currently _enable_hairpin() is only being called from spawn(). However, spawn() is not the only place that vifs (veth#) get added to a bridge (which is when we need to enable hairpin_mode on them). The more relevant function is _create_new_domain(), which is called from spawn() and other places. Without changing the information that gets passed to _create_new_domain() (which is just 'xml' from to_xml()), we can easily rewrite the first 2 lines in _enable_hairpin(), as follows:<br>
<br>def _enable_hairpin(self, xml):<br> interfaces = self.get_interfaces(xml['name'])<br><br>Then, we can move the self._enable_hairpin(instance) call from spawn() up into _create_new_domain(), and pass it xml as follows:<br>
<br>[...]<br>self._enable_hairpin(xml)<br>return domain<br><br>This will run the hairpin code every time a domain gets created, which is also when the domain's vif(s) gets inserted into the bridge with the default of hairpin_mode=0.<br>
<br>---<br>For Folsom (trunk), in nova/virt/libvirt/driver.py:<br>---<br><br>There've been a lot more changes made here, but the same strategy as above should work. Here, _create_new_domain() has been split into _create_domain() and _create_domain_and_network(), and _enable_hairpin() was moved from spawn() to _create_domain_and_network(), which seems like it'd be the right thing to do, but doesn't quite cover all of the cases of vif reinsertion, since _create_domain() is the only function which actually creates the domain (_create_domain_and_network() just calls it after doing some pre-work). The solution here is likewise fairly simple; make the same 2 changes to _enable_hairpin():<br>
<br>def _enable_hairpin(self, xml):<br> interfaces = self.get_interfaces(xml['name'])<br><br>And move it from _create_domain_and_network() to _create_domain(), like before:<br><br>[...]<br>self._enable_hairpin(xml)<br>
return domain<br><br>I haven't yet tested this on my Essex clusters and I don't have a Folsom cluster handy at present, but the change is simple and makes sense. Looking at to_xml() and _prepare_xml_info(), it appears that the 'xml' variable _create_[new_]domain() gets is just a python dictionary, and xml['name'] = instance['name'], exactly what _enable_hairpin() was using the 'instance' variable for previously.<br>
<br>Let me know if this works, or doesn't work, or doesn't make sense, or if you need an address to send gifts, etc. Hope it's solved!<br><br>-Evan<br><br><div class="gmail_quote">On Thu, Aug 23, 2012 at 11:20 AM, Sam Su <span dir="ltr"><<a href="mailto:susltd.su@gmail.com" target="_blank">susltd.su@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Hi Oleg,</div><div><br></div><div>Thank you for your investigation. Good lucky!</div><div><br></div><div>Can you let me know if find how to fix the bug?</div>
<div><br></div><div>Thanks,</div><div>Sam</div><div class="HOEnZb"><div class="h5"><br><div class="gmail_quote">
On Wed, Aug 22, 2012 at 12:50 PM, Oleg Gelbukh <span dir="ltr"><<a href="mailto:ogelbukh@mirantis.com" target="_blank">ogelbukh@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<div><br></div><div>Is it possible that, during snapshotting, libvirt just tears down virtual interface at some point, and then re-creates it, with hairpin_mode disabled again?</div><div>This bugfix [<a href="https://bugs.launchpad.net/nova/+bug/933640" target="_blank">https://bugs.launchpad.net/nova/+bug/933640</a>] implies that fix works on spawn of instance. This means that upon resume after snapshot, hairpin is not restored. May be if we insert the _enable_hairpin() call in snapshot procedure, it helps.</div>
<div>We're currently investigating this issue in one of our environments, hope to come up with answer by tomorrow.</div><div><br></div><div>--</div><div>Best regards,</div><div>Oleg</div><div><br><div class="gmail_quote">
<div><div>
On Wed, Aug 22, 2012 at 11:29 PM, Sam Su <span dir="ltr"><<a href="mailto:susltd.su@gmail.com" target="_blank">susltd.su@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div>
<div>My friend has found a way to enable ping itself, when this problem happened. But not found why this happen.</div><div>sudo echo "1" > /sys/class/net/br1000/brif/<virtual-interface-name>/hairpin_mode</div>
<div><br></div><div>I file a ticket to report this problem:</div><div><a href="https://bugs.launchpad.net/nova/+bug/1040255" target="_blank">https://bugs.launchpad.net/nova/+bug/1040255</a></div><div><br></div><div>hopefully someone can find why this happen and solve it.</div>
<div><br></div><div>Thanks,</div><div>Sam</div><div><div><div><br></div><br><div class="gmail_quote">On Fri, Jul 20, 2012 at 3:50 PM, Gabriel Hurley <span dir="ltr"><<a href="mailto:Gabriel.Hurley@nebula.com" target="_blank">Gabriel.Hurley@nebula.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-US">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I ran into some similar issues with the _<i>enable</i>_hairpin() call. The call is allowed to fail silently and (in my case) was failing. I couldn’t for the
life of me figure out why, though, and since I’m really not a networking person I didn’t trace it along too far.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Just thought I’d share my similar pain.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p style="margin-left:27.0pt">
<u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span>-<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Gabriel<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> openstack-bounces+gabriel.hurley=<a href="mailto:nebula.com@lists.launchpad.net" target="_blank">nebula.com@lists.launchpad.net</a> [mailto:<a href="mailto:openstack-bounces%2Bgabriel.hurley" target="_blank">openstack-bounces+gabriel.hurley</a>=<a href="mailto:nebula.com@lists.launchpad.net" target="_blank">nebula.com@lists.launchpad.net</a>]
<b>On Behalf Of </b>Sam Su<br>
<b>Sent:</b> Thursday, July 19, 2012 11:50 AM<br>
<b>To:</b> Brian Haley<br>
<b>Cc:</b> openstack<br>
<b>Subject:</b> Re: [Openstack] VM can't ping self floating IP after a snapshot is taken<u></u><u></u></span></p>
</div>
</div><div><div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Thank you for your support.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">I checked the file nova/virt/libvirt/connection.py, the sentence self._enable_hairpin(instance) is already added to the function _hard_reboot().<u></u><u></u></p>
<div>
<p class="MsoNormal">It looks like there are some difference between taking snapshot and reboot instance. I tried to figure out how to fix this bug but failed. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">It will be much appreciated if anyone can give some hints.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Sam<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Thu, Jul 19, 2012 at 8:37 AM, Brian Haley <<a href="mailto:brian.haley@hp.com" target="_blank">brian.haley@hp.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">On 07/17/2012 05:56 PM, Sam Su wrote:<br>
> Hi,<br>
><br>
> Just This always happens in Essex release. After I take a snapshot of my VM ( I<br>
> tried Ubuntu 12.04 or CentOS 5.8), VM can't ping its self floating IP; before I<br>
> take a snapshot though, VM can ping its self floating IP.<br>
><br>
> This looks closely related to <a href="https://bugs.launchpad.net/nova/+bug/933640" target="_blank">
https://bugs.launchpad.net/nova/+bug/933640</a>, but<br>
> still a little different. In 933640, it sounds like VM can't ping its self<br>
> floating IP regardless whether we take a snapshot or not.<br>
><br>
> Any suggestion to make an easy fix? And what is the root cause of the problem?<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">It might be because there's a missing _enable_hairpin() call in the reboot()<br>
function. Try something like this...<br>
<br>
nova/virt/libvirt/connection.py, _hard_reboot():<br>
<br>
self._create_new_domain(xml)<br>
+ self._enable_hairpin(instance)<br>
self.firewall_driver.apply_instance_filter(instance, network_info)<br>
<br>
At least that's what I remember doing myself recently when testing after a<br>
reboot, don't know about snapshot.<br>
<br>
Folsom has changed enough that something different would need to be done there.<br>
<span style="color:#888888"><br>
<span>-Brian</span></span><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
</div>
</blockquote></div><br>
</div></div><br></div></div>_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br>
</div></div><br>_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br></blockquote></div><br>