Open Stack

Cool!

 The problem seems to be:


 GRE (VXLAN too?) tunnel of Data Network (between Instance and its
namespace router) doesn't work with default mtu=1500.       ???


 It was fixed and there are two solutions for this problem (as I'm
seeing... I didn't patched ovs_lib.py), as follows:


 1- (Easier / + generic?) Lower the mtu to 1450 for Instances or;

 2- (Recommended?) Enable Jumbo Frames for Data Network (your IPv4 subnet
where OVS GRE/VXLAN tunnel flows).


 Both solutions have fixed the issue for me. I'm sticking with Jumbo
Frames, problem fixed (plus performance boost? I'll test it more)...


About 1- To lower the mtu for the Instances, add the following line to
dhcp_agent.ini:

---
dnsmasq_config_file = /etc/neutron/dnsmasq-neutron.conf
---

 And write dnsmasq-neutron.conf with:

---
dhcp-option-force=26,1450
---


 About 2- Enable Jumbo Frames on interfaces X / Y (of "Data Network" /
"local_ip var subnet")...


 I think that the IceHouse install document should add a note about this
issue, maybe here:
http://docs.openstack.org/icehouse/install-guide/install/apt/content/neutron-ml2-network-node.html

Should I fill a documentation bug?

Thanks!
Thiago


On 20 May 2014 17:56, McCann, Jack <jack.mccann at hp.com> wrote:

>  Yes.  Personally I’d lean toward jumbo frames if you have that
> flexibility.
>
>
>
> - Jack
>
>
>
> *From:* Martinx - ジェームズ [mailto:thiagocmartinsc at gmail.com]
> *Sent:* Tuesday, May 20, 2014 2:36 PM
> *To:* McCann, Jack
>
> *Subject:* Re: [Openstack] IceHouse Neutron L3 - Connectivity issue
>
>
>
> BTW Jack, do you think that, lowering the MTU for the Instances (via
> dhcp_agent) and / or enabling jumbo-frames for DATA NETWORK (GRE / VXLAN
> tunnels), might also help to solve this issue?!
>
>
>
> -
>
>  Thiago
>
>
>
> On 20 May 2014 14:36, Martinx - ジェームズ <thiagocmartinsc at gmail.com> wrote:
>
> Cool! I'll try that tonight... Tks!
>
>
>
> On 20 May 2014 14:00, McCann, Jack <jack.mccann at hp.com> wrote:
>
> We ran into an issue (change of default behavior) with GRE/VxLAN tunnel
>
> fragmentation when we moved to the 3.13 kernel.  One workaround for this
>
> issue is to set "options:df_default=false" on the ovs tunnels, see [1].
>
> Fragmenting those packets is not ideal, but it would be interesting to
>
> see if this is the problem you’re running into.
>
>
>
> - Jack
>
>
>
> [1]
> https://review.openstack.org/#/c/75281/9/neutron/agent/linux/ovs_lib.py
>
>
>
> *From:* Martinx - ジェームズ [mailto:thiagocmartinsc at gmail.com]
> *Sent:* Tuesday, May 20, 2014 12:03 PM
> *To:* Jason Bishop
> *Cc:* openstack at lists.openstack.org
> *Subject:* Re: [Openstack] IceHouse Neutron L3 - Connectivity issue
>
>
>
> Apparently we're all facing the same issue. The fact is, Neutron is too
> much complex...
>
>
>
> Back with Havana, we faced this:
> https://bugs.launchpad.net/neutron/+bug/1252900
>
>
>
> But now, with Ubuntu 14.04 + IceHouse, seems to be a different problem.
>
>
>
> "Neutron + GRE or VXLAN", is very hard to stabilize and doesn't scale very
> well...
>
>
>
> I'm seeing that it works better with "VLAN / Flat Networks" (i.e. without
> using the Neutron Network Node as Instance's default gateway). I'll try it
> instead...
>
>
>
> Maybe James Denton, from Rackspace, might help us again!     :-P
>
>
>
> Best!
>
> Thiago
>
>
>
> On 20 May 2014 11:21, Jason Bishop <jason.bishop at gmail.com> wrote:
>
>
>
> that happened to me too.  maybe same root cause?
>
>
>
> see http://ubuntuforums.org/showthread.php?t=2223789
>
>
>
> cheers
>
> jason
>
>
>
>
>
> On Mon, May 19, 2014 at 10:59 PM, Martinx - ジェームズ <
> thiagocmartinsc at gmail.com> wrote:
>
>   I managed to login into a Ubuntu Instance via `SPICE Console` and there
> is more network outages from within it, take a look ("apt-get update" froze
> too):
>
>
>
> --
>
> http://i.imgur.com/IeEXIjR.png
>
> --
>
>
>
> Am I missing something?!
>
>
>
> On 20 May 2014 02:47, Martinx - ジェームズ <thiagocmartinsc at gmail.com> wrote:
>
> Hey Stackers!
>
>
>
> I'm facing a weird network issue here, with *IceHouse* and *Neutron L3*...
> Using "Per-Project Router with Private Networks"...
>
>
>
> I am unable to connect via SSH into an Instance, from its own "Project
> Namespace Router", look:
>
>
>
> --
>
> root at net-controller-1:~# ip netns exec
> qrouter-c252886a-1fd0-45dd-9e3b-4233786e7960 ssh -i ~/pem
> ubuntu at 192.168.1.17 -v
>
> OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
>
> debug1: Reading configuration data /etc/ssh/ssh_config
>
> debug1: /etc/ssh/ssh_config line 19: Applying options for *
>
> debug1: Connecting to 192.168.1.17 [192.168.1.17] port 22.
>
> debug1: Connection established.
>
> debug1: permanently_set_uid: 0/0
>
> debug1: identity file /root/pem type -1
>
> debug1: identity file /root/pem-cert type -1
>
> debug1: Enabling compatibility mode for protocol 2.0
>
> debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
>
> debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6p1
> Ubuntu-2ubuntu1
>
> debug1: match: OpenSSH_6.6p1 Ubuntu-2ubuntu1 pat OpenSSH_6.5*,OpenSSH_6.6*
> compat 0x14000000
>
> debug1: SSH2_MSG_KEXINIT sent
>
> ...
>
> ...
>
> ...ssh stucked!!!
>
> ...
>
> --
>
>
>
> The SSH connection froze at "debug1: SSH2_MSG_KEXINIT sent" and it never
> establishes...
>
>
>
> From its attached `Floating IP`, the symptom is the same, I can't connect
> via SSH into a Instance, I tried it with 12.04 and 14.04, same result for
> both...
>
>
>
> The Instance looks good from SPICE Consoles (but I need to login with
> "ubuntu user" via network first, using ssh key).
>
>
>
> -
>
>
>
> I'm running tcpdump on both "qr-$INT" and at the Compute Node DATA
> NETWORK, to see the GREv0 traffic too, here it is:
>
>
>
> --
>
> root at net-controller-1:~# ip netns exec
> qrouter-c252886a-1fd0-45dd-9e3b-4233786e7960 tcpdump -v -ni qr-eeb0d2f3-42
>
>
>
> http://paste.openstack.org/show/80969/
>
>
>
> NOTE: From the line ~16 to 32, SSH was already frozen...
>
> --
>
>
>
> root at compute-node-1:~# tcpdump -ni eth1 | grep -v IP6
>
>
>
> http://paste.openstack.org/show/80970/
>
> --
>
>
>
> What can I do to deep investigate this?!
>
>
>
> I double checked everything, including OVS bridges, sysctl.conf, `ethtool
> --offload ethX gro off` and etc... Don't know what's is wrong...
>
>
>
> I think that this is the last problem I'm facing with IceHouse, I would
> like to put it into prod but, I am unable to do it right now...   :-/
>
>
>
> Tks in advance!
>
>
>
> Best,
>
> Thiago
>
>
>
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140521/23c1e18b/attachment.html>