<div>Hi,</div><div><br></div>I also reported this bug:<div> <a href="https://bugs.launchpad.net/nova/+bug/1040255" target="_blank">https://bugs.launchpad.net/nova/+bug/1040255</a>
</div><div><br></div><div> If someone can combine you guys solution and get a perfect way to fix this bug, that will be great. </div><div><br></div><div>BRs,</div><div>Sam<br><br><div class="gmail_quote">On Thu, Aug 23, 2012 at 9:27 PM, heut2008 <span dir="ltr"><<a href="mailto:heut2008@gmail.com" target="_blank">heut2008@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">this bug has been filed here <a href="https://bugs.launchpad.net/nova/+bug/1040537" target="_blank">https://bugs.launchpad.net/nova/+bug/1040537</a><br>
<br>
2012/8/24 Vishvananda Ishaya <<a href="mailto:vishvananda@gmail.com">vishvananda@gmail.com</a>>:<br>
<div class="HOEnZb"><div class="h5">> +1 to this. Evan, can you report a bug (if one hasn't been reported yet) and<br>
> propose the fix? Or else I can find someone else to propose it.<br>
><br>
> Vish<br>
><br>
> On Aug 23, 2012, at 1:38 PM, Evan Callicoat <<a href="mailto:diopter@gmail.com">diopter@gmail.com</a>> wrote:<br>
><br>
> Hello all!<br>
><br>
> I'm the original author of the hairpin patch, and things have changed a<br>
> little bit in Essex and Folsom from the original Diablo target. I believe I<br>
> can shed some light on what should be done here to solve the issue in either<br>
> case.<br>
><br>
> ---<br>
> For Essex (stable/essex), in nova/virt/libvirt/connection.py:<br>
> ---<br>
><br>
> Currently _enable_hairpin() is only being called from spawn(). However,<br>
> spawn() is not the only place that vifs (veth#) get added to a bridge (which<br>
> is when we need to enable hairpin_mode on them). The more relevant function<br>
> is _create_new_domain(), which is called from spawn() and other places.<br>
> Without changing the information that gets passed to _create_new_domain()<br>
> (which is just 'xml' from to_xml()), we can easily rewrite the first 2 lines<br>
> in _enable_hairpin(), as follows:<br>
><br>
> def _enable_hairpin(self, xml):<br>
> interfaces = self.get_interfaces(xml['name'])<br>
><br>
> Then, we can move the self._enable_hairpin(instance) call from spawn() up<br>
> into _create_new_domain(), and pass it xml as follows:<br>
><br>
> [...]<br>
> self._enable_hairpin(xml)<br>
> return domain<br>
><br>
> This will run the hairpin code every time a domain gets created, which is<br>
> also when the domain's vif(s) gets inserted into the bridge with the default<br>
> of hairpin_mode=0.<br>
><br>
> ---<br>
> For Folsom (trunk), in nova/virt/libvirt/driver.py:<br>
> ---<br>
><br>
> There've been a lot more changes made here, but the same strategy as above<br>
> should work. Here, _create_new_domain() has been split into _create_domain()<br>
> and _create_domain_and_network(), and _enable_hairpin() was moved from<br>
> spawn() to _create_domain_and_network(), which seems like it'd be the right<br>
> thing to do, but doesn't quite cover all of the cases of vif reinsertion,<br>
> since _create_domain() is the only function which actually creates the<br>
> domain (_create_domain_and_network() just calls it after doing some<br>
> pre-work). The solution here is likewise fairly simple; make the same 2<br>
> changes to _enable_hairpin():<br>
><br>
> def _enable_hairpin(self, xml):<br>
> interfaces = self.get_interfaces(xml['name'])<br>
><br>
> And move it from _create_domain_and_network() to _create_domain(), like<br>
> before:<br>
><br>
> [...]<br>
> self._enable_hairpin(xml)<br>
> return domain<br>
><br>
> I haven't yet tested this on my Essex clusters and I don't have a Folsom<br>
> cluster handy at present, but the change is simple and makes sense. Looking<br>
> at to_xml() and _prepare_xml_info(), it appears that the 'xml' variable<br>
> _create_[new_]domain() gets is just a python dictionary, and xml['name'] =<br>
> instance['name'], exactly what _enable_hairpin() was using the 'instance'<br>
> variable for previously.<br>
><br>
> Let me know if this works, or doesn't work, or doesn't make sense, or if you<br>
> need an address to send gifts, etc. Hope it's solved!<br>
><br>
> -Evan<br>
><br>
> On Thu, Aug 23, 2012 at 11:20 AM, Sam Su <<a href="mailto:susltd.su@gmail.com">susltd.su@gmail.com</a>> wrote:<br>
>><br>
>> Hi Oleg,<br>
>><br>
>> Thank you for your investigation. Good lucky!<br>
>><br>
>> Can you let me know if find how to fix the bug?<br>
>><br>
>> Thanks,<br>
>> Sam<br>
>><br>
>> On Wed, Aug 22, 2012 at 12:50 PM, Oleg Gelbukh <<a href="mailto:ogelbukh@mirantis.com">ogelbukh@mirantis.com</a>><br>
>> wrote:<br>
>>><br>
>>> Hello,<br>
>>><br>
>>> Is it possible that, during snapshotting, libvirt just tears down virtual<br>
>>> interface at some point, and then re-creates it, with hairpin_mode disabled<br>
>>> again?<br>
>>> This bugfix [<a href="https://bugs.launchpad.net/nova/+bug/933640" target="_blank">https://bugs.launchpad.net/nova/+bug/933640</a>] implies that<br>
>>> fix works on spawn of instance. This means that upon resume after snapshot,<br>
>>> hairpin is not restored. May be if we insert the _enable_hairpin() call in<br>
>>> snapshot procedure, it helps.<br>
>>> We're currently investigating this issue in one of our environments, hope<br>
>>> to come up with answer by tomorrow.<br>
>>><br>
>>> --<br>
>>> Best regards,<br>
>>> Oleg<br>
>>><br>
>>> On Wed, Aug 22, 2012 at 11:29 PM, Sam Su <<a href="mailto:susltd.su@gmail.com">susltd.su@gmail.com</a>> wrote:<br>
>>>><br>
>>>> My friend has found a way to enable ping itself, when this problem<br>
>>>> happened. But not found why this happen.<br>
>>>> sudo echo "1" ><br>
>>>> /sys/class/net/br1000/brif/<virtual-interface-name>/hairpin_mode<br>
>>>><br>
>>>> I file a ticket to report this problem:<br>
>>>> <a href="https://bugs.launchpad.net/nova/+bug/1040255" target="_blank">https://bugs.launchpad.net/nova/+bug/1040255</a><br>
>>>><br>
>>>> hopefully someone can find why this happen and solve it.<br>
>>>><br>
>>>> Thanks,<br>
>>>> Sam<br>
>>>><br>
>>>><br>
>>>> On Fri, Jul 20, 2012 at 3:50 PM, Gabriel Hurley<br>
>>>> <<a href="mailto:Gabriel.Hurley@nebula.com">Gabriel.Hurley@nebula.com</a>> wrote:<br>
>>>>><br>
>>>>> I ran into some similar issues with the _enable_hairpin() call. The<br>
>>>>> call is allowed to fail silently and (in my case) was failing. I couldn’t<br>
>>>>> for the life of me figure out why, though, and since I’m really not a<br>
>>>>> networking person I didn’t trace it along too far.<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> Just thought I’d share my similar pain.<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> - Gabriel<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> From: openstack-bounces+gabriel.hurley=<a href="mailto:nebula.com@lists.launchpad.net">nebula.com@lists.launchpad.net</a><br>
>>>>> [mailto:<a href="mailto:openstack-bounces%2Bgabriel.hurley">openstack-bounces+gabriel.hurley</a>=<a href="mailto:nebula.com@lists.launchpad.net">nebula.com@lists.launchpad.net</a>] On<br>
>>>>> Behalf Of Sam Su<br>
>>>>> Sent: Thursday, July 19, 2012 11:50 AM<br>
>>>>> To: Brian Haley<br>
>>>>> Cc: openstack<br>
>>>>> Subject: Re: [Openstack] VM can't ping self floating IP after a<br>
>>>>> snapshot is taken<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> Thank you for your support.<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> I checked the file nova/virt/libvirt/connection.py, the sentence<br>
>>>>> self._enable_hairpin(instance) is already added to the function<br>
>>>>> _hard_reboot().<br>
>>>>><br>
>>>>> It looks like there are some difference between taking snapshot and<br>
>>>>> reboot instance. I tried to figure out how to fix this bug but failed.<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> It will be much appreciated if anyone can give some hints.<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> Thanks,<br>
>>>>><br>
>>>>> Sam<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> On Thu, Jul 19, 2012 at 8:37 AM, Brian Haley <<a href="mailto:brian.haley@hp.com">brian.haley@hp.com</a>><br>
>>>>> wrote:<br>
>>>>><br>
>>>>> On 07/17/2012 05:56 PM, Sam Su wrote:<br>
>>>>> > Hi,<br>
>>>>> ><br>
>>>>> > Just This always happens in Essex release. After I take a snapshot of<br>
>>>>> > my VM ( I<br>
>>>>> > tried Ubuntu 12.04 or CentOS 5.8), VM can't ping its self floating<br>
>>>>> > IP; before I<br>
>>>>> > take a snapshot though, VM can ping its self floating IP.<br>
>>>>> ><br>
>>>>> > This looks closely related to<br>
>>>>> > <a href="https://bugs.launchpad.net/nova/+bug/933640" target="_blank">https://bugs.launchpad.net/nova/+bug/933640</a>, but<br>
>>>>> > still a little different. In 933640, it sounds like VM can't ping its<br>
>>>>> > self<br>
>>>>> > floating IP regardless whether we take a snapshot or not.<br>
>>>>> ><br>
>>>>> > Any suggestion to make an easy fix? And what is the root cause of the<br>
>>>>> > problem?<br>
>>>>><br>
>>>>> It might be because there's a missing _enable_hairpin() call in the<br>
>>>>> reboot()<br>
>>>>> function. Try something like this...<br>
>>>>><br>
>>>>> nova/virt/libvirt/connection.py, _hard_reboot():<br>
>>>>><br>
>>>>> self._create_new_domain(xml)<br>
>>>>> + self._enable_hairpin(instance)<br>
>>>>> self.firewall_driver.apply_instance_filter(instance,<br>
>>>>> network_info)<br>
>>>>><br>
>>>>> At least that's what I remember doing myself recently when testing<br>
>>>>> after a<br>
>>>>> reboot, don't know about snapshot.<br>
>>>>><br>
>>>>> Folsom has changed enough that something different would need to be<br>
>>>>> done there.<br>
>>>>><br>
>>>>> -Brian<br>
>>>>><br>
>>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>> _______________________________________________<br>
>>>> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>>> Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
>>>> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>>> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
>>>><br>
>>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>> Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
>> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
>><br>
><br>
> _______________________________________________<br>
> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
> Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
> Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
><br>
<br>
_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
</div></div></blockquote></div><br></div>