<div dir="ltr"><div><div><div><div><div><div><div><div><div><div><div>I wrote the spec for the MTU work that's in the Neutron API today. It haunts my nightmares. I learned so many nasty corner cases for MTU, and you're treading that same dark path.<br><br></div>I'd first like to point out a few things that change the implications of what you're reporting in strange ways. [1] points out even more strange ways, but these are the notable ones from what I've been reading here...<br><br></div>RFC7348: "VTEPs MUST NOT fragment VXLAN packets. ... The destination VTEP MAY silently discard such VXLAN fragments." The VXLAN VTEP implementations we use today may fragment, but it's not according to the RFC, and I wouldn't rely that every implementation you come across knows to do it. So, the largest L2 packet you can send over VXLAN is a function of path MTU.<br><br></div><div>Even if VXLAN is fragmenting, you actually actively want to avoid
it fragmenting, because - in the typical case of bulk TCP transfers
using max-MTU packets - you're *invisibly* fragmenting the packets into
two and adding about 80 bytes of overhead in the process and then
reassembling them at the far end. You've just expicitly guaranteed
that, just as you send the most data, your connection will slow down. And
the MTU problem will be undetectable to the VMs (which can't find out that a VXLAN
encapped packet has been fragmented; the packet *they* sent didn't fragment, but the one it's carried in did, not to mention the fragmentation didn't even happen at an L3 node in the virtual network so DF and therefore PMTUD wouldn't work).<br></div><div><br></div>Path MTU is not fixed, because your path can vary according to network weather (failures, congestion, whatever). It's an oddity, and perhaps a rarity, but you can get many weirdnesses: you fail over from one link to a link with a smaller MTU and the path MTU shrinks; some switches are jumbo frame and some aren't, so the path MTU might vary from host to host; and so on. Granted, these are weird cases, but the point here is that Openstack cannot *discover* this number. An installer might attempt something, knowing how to read switch config; or it might attempt to validate a number it's been given, as best it can; but even then it's best effort, it's not a guarantee. For all these reasons, the only way to really get the minimum path MTU is from the operator themselves, which is why this is a configuration parameter to Neutron (path_mtu).<br><br></div>The aim of the changes in the spec [1] were threefold:<br><br></div>1. To ensure that an app that absolutely required a certain minimum MTU to operate could guarantee it would receive it<br></div>2. To allow the network to say what the MTU was, so that the VM could be programmed accordingly<br></div>3. To ensure that the MTU for the network would - by default - settle on the optimal value, per all the stuff above.<br><br></div>So what could we do in this environment to improve matters?<br><br>1. We should advertise MTU in the RA and DHCP messages that Openstack sends. I thought we'd already done this work, but this thread suggests not.<br><br>[Note, though, that you can't reliably set an MTU higher than 1500 on IPv6 using an RA, thanks to RFC4861 referencing RFC2464 which goes with the standard, but not the practice, that the biggest ethernet packet is 1500 bytes. You've been violating the standard all these years, you bad people. Unfortunately, Linux enforces this RA rule, albeit slightly strangely.]<br><br></div>2. We should also put the MTU in any config-drive settings for VMs that don't respect such things in DHCP and RAs, or don't do DHCP. This is Nova-side, reacting to the MTU property of the network.<br><br></div>3. Installers should determine the appropriate MTU settings on interfaces and ensure they're set. Openstack can't do this in some cases (VXLAN - no interfaces) - and probably shouldn't in others (VLAN - the interface MTU is input to the MTU selection algorithm above, and the installer should set the interface MTU to match what the operator says the fabric MTU is).<br><br></div>4. We need to check the Neutron network drivers to see which ones are accepting, but not properly respecting, the MTU setting on the network. I suspect we're short of testing to make sure that veths, bridges, switches and so on are all correctly configured.<br><div><div><div><br><div><div><div><div><div><div><div>-- <br></div><div>Ian.<br></div></div></div></div></div></div></div><br>[1] <a href="https://review.openstack.org/#/c/105989/">https://review.openstack.org/#/c/105989/</a> and <a href="https://github.com/openstack/neutron-specs/blob/master/specs/kilo/mtu-selection-and-advertisement.rst">https://github.com/openstack/neutron-specs/blob/master/specs/kilo/mtu-selection-and-advertisement.rst</a><br><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 22 January 2016 at 19:13, Matt Kassawara <span dir="ltr"><<a href="mailto:mkassawara@gmail.com" target="_blank">mkassawara@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">The fun continues, now using an OpenStack deployment on physical hardware that supports jumbo frames with 9000 MTU and IPv4/IPv6. This experiment still uses Linux bridge for consistency. I'm planning to run similar experiments with Open vSwitch and Open Virtual Network (OVN) in the next week.<div><br></div><div><span style="font-size:12.8px">I highly recommend reading further, but here's the TL;DR: </span><span style="font-size:12.8px">Using physical network interfaces with MTUs larger than 1500 reveals an additional problem with </span><span style="font-size:12.8px">veth pair for the neutron router interface on the public network. Additionally</span><span style="font-size:12.8px">, </span><span style="font-size:12.8px">IP protocol version does not impact MTU calculation for Linux bridge.</span></div><div><div><br></div><div><span style="font-size:12.8px">First, review the OpenStack bits and resulting network components in the environment [1]. In the first experiment, public cloud network limitations prevented truly seeing how Linux bridge (actually the kernel) handles physical network interfaces with MTUs larger than 1500. In this experiment, we see that it automatically calculates the proper MTU for bridges and VXLAN interfaces using the MTU of parent devices. Also, see that a regular 'ping' works between the host outside of the deployment and the VM [2].</span><br></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">[1] <a href="https://gist.github.com/ionosphere80/a3725066386d8ca4c6d7" target="_blank">https://gist.github.com/ionosphere80/a3725066386d8ca4c6d7</a></span></div><div><span style="font-size:12.8px">[2] <a href="https://gist.github.com/ionosphere80/a8d601a356ac6c6274cb" target="_blank">https://gist.github.com/ionosphere80/a8d601a356ac6c6274cb</a></span></div><div><span style="font-size:12.8px"><br></span></div><div><div style="font-size:12.8px">Note: The tcpdump output in each case references up to six points: neutron router gateway on the public network (qg), namespace end of the veth pair for the neutron router interface on the private network (qr), bridge end of the veth pair for router interface on the private network (tap), controller node end of the VXLAN network (underlying interface), compute node end of the VXLAN network (underlying interface), and the bridge end of the tap for the VM (tap).</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">In the first experiment, SSH "stuck" because of a MTU mismatch on the veth pair between the router namespace and private network bridge. <span style="font-size:12.8px">In this experiment, SSH works because the VM network interface uses a 1500 MTU and all devices along the path between the host and VM use a 1500 or larger MTU. So, let's configure the VM network interface to use the proper MTU of 9000 minus the VXLAN protocol overhead of 50 bytes... 8950... and try SSH again.</span></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div style="font-size:12.8px">2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc pfifo_fast qlen 1000</div><div style="font-size:12.8px"> link/ether fa:16:3e:46:ac:d3 brd ff:ff:ff:ff:ff:ff</div><div style="font-size:12.8px"> inet <a href="http://172.16.1.3/24" target="_blank">172.16.1.3/24</a> brd 172.16.1.255 scope global eth0</div><div style="font-size:12.8px"> inet6 fd00:100:52:1:f816:3eff:fe46:acd3/64 scope global dynamic</div><div style="font-size:12.8px"> valid_lft 86395sec preferred_lft 14395sec</div><div style="font-size:12.8px"> inet6 fe80::f816:3eff:fe46:acd3/64 scope link</div><div style="font-size:12.8px"> valid_lft forever preferred_lft forever</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">SSH doesn't work with IPv4 or IPv6. Adding a slight twist to the first experiment, <span style="font-size:12.8px">I don't even see the large packet traversing the </span><span style="font-size:12.8px">neutron router gateway on the public network. So, I began a tcpdump closer to the source on the bridge</span><span style="font-size:12.8px"> end of the veth pair for the neutron router interface on the public network.</span></div><div style="font-size:12.8px"><span style="font-size:12.8px"><br></span></div><div style="font-size:12.8px"><span style="font-size:12.8px">Looking at [3], the veth pair between the router namespace and private network bridge drops the packet. The MTU changes over a layer-2 connection without a router, similar to connecting two switches with different MTUs. Even if it could participate in PMTUD, the veth pair lacks an IP address and therefore cannot originate ICMP messages.</span><span style="font-size:12.8px"><br></span></div><div style="font-size:12.8px"><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">[3] <a href="https://gist.github.com/ionosphere80/ec83d0955c79b05ea381" target="_blank">https://gist.github.com/ionosphere80/ec83d0955c79b05ea381</a></span></div><div><br></div><div><div>Using observations from the first experiment, let's configure the MTU of the interfaces in the qrouter namespace to match the other end of their respective veth pairs. The public network (gateway) interface MTU becomes 9000 and the private network router interfaces (IPv4 and IPv6) become 8950.</div><div><br></div><div><div>2: qr-49b27408-04: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000</div><div> link/ether fa:16:3e:e5:43:1c brd ff:ff:ff:ff:ff:ff</div><div>3: qr-b7e0ef22-32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000</div><div> link/ether fa:16:3e:16:01:92 brd ff:ff:ff:ff:ff:ff</div><div>4: qg-7bbe8e38-cc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000</div><div> link/ether fa:16:3e:2b:c1:fd brd ff:ff:ff:ff:ff:ff</div></div><div><br></div><div><div style="font-size:12.8px">Let's ping with a payload size of 8922 for IPv4 and 8902 for IPv6, the maximum for a VXLAN segment with 8950 MTU, and look at the tcpdump output [4]. For brevity, I'm only showing tcpdump output from the VM tap interface. Ping operates normally.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"># ping -c 1 -s 8922 -M do 10.100.52.104</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"># ping -c 1 -s 8902 -M do fd00:100:52:1:f816:3eff:fe46:acd3</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">[4] <a href="https://gist.github.com/ionosphere80/85339b587bb9b2693b07" target="_blank">https://gist.github.com/ionosphere80/85339b587bb9b2693b07</a></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Let's ping with a payload size of 8923 for IPv4 and 8903 for IPv6, one byte larger than the maximum for a VXLAN segment with 8950 MTU. <span style="font-size:12.8px">The router namespace, operating at layer-3, sees the MTU discrepancy between the two interfaces in the namespace and returns an ICMP "fragmentation needed" or "packet too big" message to the sender. The sender uses the MTU value in the ICMP packet to recalculate the length of the first packet and caches it for future packets.</span></div></div><div style="font-size:12.8px"><br></div><div><div><span style="font-size:12.8px"># ping -c 1 -s 8923 -M do 10.100.52.104</span></div><div><span style="font-size:12.8px">PING 10.100.52.104 (10.100.52.104) 8923(8951) bytes of data.</span></div><div><span style="font-size:12.8px">From 10.100.52.104 icmp_seq=1 Frag needed and DF set (mtu = 8950)</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">--- 10.100.52.104 ping statistics ---</span></div><div><span style="font-size:12.8px">1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"># ping6 -c 1 -s 8903 -M do fd00:100:52:1:f816:3eff:fe46:acd3</span></div><div><span style="font-size:12.8px">PING fd00:100:52:1:f816:3eff:fe46:acd3(fd00:100:52:1:f816:3eff:fe46:acd3) 8903 data bytes</span></div><div><span style="font-size:12.8px">From fd00:100:52::101 icmp_seq=1 Packet too big: mtu=8950</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">--- fd00:100:52:1:f816:3eff:fe46:acd3 ping statistics ---</span></div><div><span style="font-size:12.8px">1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms</span></div><div><span style="font-size:12.8px"><br></span></div><div><div style="font-size:12.8px"># ip route get to 10.100.52.104</div><div style="font-size:12.8px">10.100.52.104 dev eth1 src 10.100.52.45</div><div style="font-size:12.8px"> cache expires 596sec mtu 8950</div><div style="font-size:12.8px"><br></div><div><div><span style="font-size:12.8px"># ip route get to fd00:100:52:1:f816:3eff:fe46:acd3</span></div><div><span style="font-size:12.8px">fd00:100:52:1:f816:3eff:fe46:acd3 from :: via fd00:100:52::101 dev eth1 src fd00:100:52::45 metric 0</span></div><div><span style="font-size:12.8px"> cache expires 556sec mtu 8950</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Finally, let's try SSH.</span></div><div><span style="font-size:12.8px"><br></span></div><div><div><span style="font-size:12.8px"># ssh <a href="mailto:cirros@10.100.52.104" target="_blank">cirros@10.100.52.104</a></span></div><div><span style="font-size:12.8px"><a href="mailto:cirros@10.100.52.104" target="_blank">cirros@10.100.52.104</a>'s password:</span></div><div><span style="font-size:12.8px">$</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"># ssh cirros@fd00:100:52:1:f816:3eff:fe46:acd3</span></div><div><span style="font-size:12.8px">cirros@fd00:100:52:1:f816:3eff:fe46:acd3's password:</span></div><div><span style="font-size:12.8px">$</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">SSH works for both IPv4 and IPv6.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">This experiment reaches the same conclusion as the first experiment. However, using physical hardware that supports jumbo frames reveals an additional </span><span style="font-size:12.8px">problem with the </span><span style="font-size:12.8px">veth pair for the neutron router interface on the public network. </span><span style="font-size:12.8px">For any MTU, we can address the egress MTU disparity (from the VM) by advertising the MTU of the overlay network to the VM via DHCP/RA or using manual interface configuration. Additionally, </span><span style="font-size:12.8px">IP protocol version does not impact MTU calculation for Linux bridge.</span><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Hopefully moving to physical hardware makes this experiment easier to understand and the conclusion more useful for realistic networks.</span></div><span class="HOEnZb"><font color="#888888"><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Matt</span></div></font></span></div></div></div></div></div></div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 20, 2016 at 11:18 AM, Rick Jones <span dir="ltr"><<a href="mailto:rick.jones2@hpe.com" target="_blank">rick.jones2@hpe.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On 01/20/2016 08:56 AM, Sean M. Collins wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Tue, Jan 19, 2016 at 08:15:18AM EST, Matt Kassawara wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
No. However, we ought to determine what happens when both DHCP and RA<br>
advertise it.<br>
</blockquote>
<br>
We'd have to look at the RFCs for how hosts are supposed to behave since<br>
IPv6 has a minimum MTU of 1280 bytes while IPv4's minimum mtu is 576<br>
(what is this, an MTU for ants?).<br>
</blockquote>
<br></span>
Quibble - 576 is the IPv4 minimum, maximum MTU. That is to say a compliant IPv4 implementation must be able to reassemble datagrams of at least 576 bytes.<br>
<br>
If memory serves, the actual minimum MTU for IPv4 is 68 bytes.<span><font color="#888888"><br>
<br>
rick jones</font></span><div><div><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>
</div></div><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>