<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"MS UI Gothic";
panose-1:2 11 6 0 7 2 5 8 2 4;}
@font-face
{font-family:"\@MS UI Gothic";
panose-1:2 11 6 0 7 2 5 8 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Courier New";
color:windowtext;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:"Courier New"">We ran into an issue (change of default behavior) with GRE/VxLAN tunnel<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">fragmentation when we moved to the 3.13 kernel. One workaround for this<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">issue is to set "options:df_default=false" on the ovs tunnels, see [1].<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Fragmenting those packets is not ideal, but it would be interesting to<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">see if this is the problem you’re running into.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">- Jack<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[1] https://review.openstack.org/#/c/75281/9/neutron/agent/linux/ovs_lib.py<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Martinx -
</span><span style="font-size:10.0pt;font-family:"MS UI Gothic","sans-serif"">ジェームズ</span><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> [mailto:thiagocmartinsc@gmail.com]
<br>
<b>Sent:</b> Tuesday, May 20, 2014 12:03 PM<br>
<b>To:</b> Jason Bishop<br>
<b>Cc:</b> openstack@lists.openstack.org<br>
<b>Subject:</b> Re: [Openstack] IceHouse Neutron L3 - Connectivity issue<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Apparently we're all facing the same issue. The fact is, Neutron is too much complex...<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Back with Havana, we faced this: <a href="https://bugs.launchpad.net/neutron/+bug/1252900" target="_blank">https://bugs.launchpad.net/neutron/+bug/1252900</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">But now, with Ubuntu 14.04 + IceHouse, seems to be a different problem.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">"Neutron + GRE or VXLAN", is very hard to stabilize and doesn't scale very well...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'm seeing that it works better with "VLAN / Flat Networks" (i.e. without using the Neutron Network Node as Instance's default gateway). I'll try it instead...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Maybe James Denton, from Rackspace, might help us again! :-P<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Best!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Thiago<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 20 May 2014 11:21, Jason Bishop <<a href="mailto:jason.bishop@gmail.com" target="_blank">jason.bishop@gmail.com</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">that happened to me too. maybe same root cause?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">see <a href="http://ubuntuforums.org/showthread.php?t=2223789" target="_blank">http://ubuntuforums.org/showthread.php?t=2223789</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">cheers<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">jason<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On Mon, May 19, 2014 at 10:59 PM, Martinx - <span style="font-family:"MS Gothic"">
ジェームズ</span> <<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal">I managed to login into a Ubuntu Instance via `SPICE Console` and there is more network outages from within it, take a look ("apt-get update" froze too):<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="http://i.imgur.com/IeEXIjR.png" target="_blank">http://i.imgur.com/IeEXIjR.png</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Am I missing something?!<o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 20 May 2014 02:47, Martinx - <span style="font-family:"MS Gothic"">
ジェームズ</span> <<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal">Hey Stackers!<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'm facing a weird network issue here, with <b>IceHouse</b> and
<i>Neutron L3</i>... Using "Per-Project Router with Private Networks"... <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I am unable to connect via SSH into an Instance, from its own "Project Namespace Router", look:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">root@net-controller-1:~# ip netns exec qrouter-c252886a-1fd0-45dd-9e3b-4233786e7960 ssh -i ~/pem
<a href="mailto:ubuntu@192.168.1.17" target="_blank">ubuntu@192.168.1.17</a> -v</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">debug1: Reading configuration data /etc/ssh/ssh_config</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">debug1: /etc/ssh/ssh_config line 19: Applying options for *</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">debug1: Connecting to 192.168.1.17 [192.168.1.17] port 22.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">debug1: Connection established.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">debug1: permanently_set_uid: 0/0</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">debug1: identity file /root/pem type -1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">debug1: identity file /root/pem-cert type -1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">debug1: Enabling compatibility mode for protocol 2.0</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6p1 Ubuntu-2ubuntu1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">debug1: match: OpenSSH_6.6p1 Ubuntu-2ubuntu1 pat OpenSSH_6.5*,OpenSSH_6.6* compat 0x14000000</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">debug1: SSH2_MSG_KEXINIT sent</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">...<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">...ssh stucked!!!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The SSH connection froze at "<span style="font-family:"Courier New"">debug1: SSH2_MSG_KEXINIT sent</span>" and it never establishes...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">From its attached `Floating IP`, the symptom is the same, I can't connect via SSH into a Instance, I tried it with 12.04 and 14.04, same result for both...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The Instance looks good from SPICE Consoles (but I need to login with "ubuntu user" via network first, using ssh key).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'm running tcpdump on both "qr-$INT" and at the Compute Node DATA NETWORK, to see the GREv0 traffic too, here it is:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">root@net-controller-1:~# ip netns exec qrouter-c252886a-1fd0-45dd-9e3b-4233786e7960 tcpdump -v -ni qr-eeb0d2f3-42</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="http://paste.openstack.org/show/80969/" target="_blank">http://paste.openstack.org/show/80969/</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">NOTE: From the line ~16 to 32, SSH was already frozen...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">root@compute-node-1:~# tcpdump -ni eth1 | grep -v IP6</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="http://paste.openstack.org/show/80970/" target="_blank">http://paste.openstack.org/show/80970/</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">What can I do to deep investigate this?!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I double checked everything, including OVS bridges, sysctl.conf, `<span style="font-family:"Courier New"">ethtool --offload ethX gro off</span>` and etc... Don't know what's is wrong...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I think that this is the last problem I'm facing with IceHouse, I would like to put it into prod but, I am unable to do it right now... :-/<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Tks in advance!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Thiago<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>