<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#70AD47">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#70AD47">Pl find my reply .<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#70AD47"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#70AD47">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#70AD47">keshava<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Ian Wells [mailto:ijw.ubuntu@cack.org.uk]
<br>
<b>Sent:</b> Tuesday, October 28, 2014 1:19 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [neutron] vm can not transport large file under neutron ml2 + linux bridge + vxlan<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On 28 October 2014 00:18, A, Keshava <<a href="mailto:keshava.a@hp.com" target="_blank">keshava.a@hp.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Currently OpenStack have any framework to notify the Tennant/Service-VM for such kind of notification
based on VM’s interest ?</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">It's possible to use DHCP or RA to notify a VM of the MTU but there are limitations (RAs don't let you increase the MTU, only decrease it, and obviously VMs must support the MTU element of DHCP) and Openstack doesn't currently use it.
You can statically configure the DHCP MTU number that DHCP transmits; this is useful to work around problems but not really the right answer to the problem.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">VM may be very much interested for such kind of notification like
</span><o:p></o:p></p>
<p style="margin-left:.75in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">1.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Path MTU.</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">This will be correctly discovered from the ICMP PMTU exceeded message, and Neutron routers should certainly be expected to send that. (In fact the namespace implementation of routers would do this if the router ever had different MTUs
on its ports; it's in the kernel network stack.) There's no requirement for a special notification, and indeed you couldn't do it that way anyway.<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#70AD47">In the network interface/router going down is common scenario. In that case the packet will take different path which may have different MTU.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#70AD47">In that case the PATH MTU calculated at the source may be different and should be notified dynamically to VM. So that VM can originate the packet with requirement
MTU size .<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#70AD47">If there is no notification mechanism ( as per this reply):<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#70AD47">If there is no such dynamic PATH MTU change notification to VM, how VM can change the packet size ?<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#70AD47">Or
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#70AD47">Do we expect ICMP too big message reaches all the way to VM ?
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#70AD47">Or
<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#70AD47">VM itself the run the PATH MTU ?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p style="margin-left:.75in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">2.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Based on specific incoming Tennant traffic, block/Allow particular traffic flow at infrastructure level itself, instead of at VM.
</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">I don't see the relevance; and you appear to be describing security groups.<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p style="margin-left:.75in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">This may require OpenStack infrastructure notification support to Tenant/Service VM.</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">Not particularly, as MTU doesn't generally change, and I think we would forbid changing the MTU of a network after creation. It's only an initial configuration thing, therefore. It might involve better cloud-init support for network configuration,
something that gets discussed periodically. <br>
<br>
-- <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Ian.<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> …</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks & regards,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Keshava</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Ian Wells [mailto:<a href="mailto:ijw.ubuntu@cack.org.uk" target="_blank">ijw.ubuntu@cack.org.uk</a>]
<br>
<b>Sent:</b> Tuesday, October 28, 2014 11:40 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [neutron] vm can not transport large file under neutron ml2 + linux bridge + vxlan</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Path MTU discovery works on a path - something with an L3 router in the way - where the outbound interface has a smaller MTU than the inbound one. You're transmitting across an L2 network
- no L3 routers present. You send a 1500 byte packet, the network fabric (which is not L3, has no address, and therefore has no means to answer you) does all that it can do with that packet - it drops it. The sender retransmits, assuming congestion, but
the same thing happens. Eventually the sender decides there's a network problem and times out.<br>
<br>
This is a common problem with Openstack deployments, although various features of the virtual networking let you get away with it, with some configs and not others. OVS used to fake a PMTU exceeded message from the destination if you tried to pass an overlarge
packet - not in spec, but it hid the problem nicely. I have a suspicion that some implementations will fragment the containing UDP packet, which is also not in spec and also solves the problem (albeit with poor performance).<br>
<br>
The right answer for you is to set the MTU in your machines to the same MTU you've given the network, that is, 1450 bytes. You can do this by setting a DHCP option for MTU, providing your VMs support that option (search the web for the solution, I don't have
it offhand) or lower the MTU by hand or by script when you start your VM.<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The right answer for everyone is to properly determine and advertise the network MTU to VMs (which, with provider networks, is not even consistent from one network to the next)
and that's the spec Kyle is referring to. We'll be fixing this in Kilo.<br>
-- <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Ian.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On 27 October 2014 20:14, Li Tianqing <<a href="mailto:jazeltq@163.com" target="_blank">jazeltq@163.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black"><br>
<br>
<br>
</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">--</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">Best</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black"> Li Tianqing</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black"> </span><o:p></o:p></p>
<pre><span style="color:black"><br>At 2014-10-27 17:42:41, "Ihar Hrachyshka" <<a href="mailto:ihrachys@redhat.com" target="_blank">ihrachys@redhat.com</a>> wrote:</span><o:p></o:p></pre>
<pre><span style="color:black">>-----BEGIN PGP SIGNED MESSAGE-----</span><o:p></o:p></pre>
<pre><span style="color:black">>Hash: SHA512</span><o:p></o:p></pre>
<pre><span style="color:black">> </span><o:p></o:p></pre>
<pre><span style="color:black">>On 27/10/14 02:18, Li Tianqing wrote:</span><o:p></o:p></pre>
<pre><span style="color:black">>> Hello, Right now, we test neutron under havana release. We</span><o:p></o:p></pre>
<pre><span style="color:black">>> configured network_device_mtu=1450 in neutron.conf, After create</span><o:p></o:p></pre>
<pre><span style="color:black">>> vm, we found the vm interface's mtu is 1500, the ping, ssh, is ok.</span><o:p></o:p></pre>
<pre><span style="color:black">>> But if we scp large file between vms then scp display 'stalled'.</span><o:p></o:p></pre>
<pre><span style="color:black">>> And iperf is also can not completed. If we configured vm's mtu to</span><o:p></o:p></pre>
<pre><span style="color:black">>> 1450, then iperf, scp all is ok. If we iperf with -M 1300, then the</span><o:p></o:p></pre>
<pre><span style="color:black">>> iperf is ok too. The vms path mtu discovery is set by default. I do</span><o:p></o:p></pre>
<pre><span style="color:black">>> not know why the vm whose mtu is 1500 can not send large file.</span><o:p></o:p></pre>
<pre><span style="color:black">> </span><o:p></o:p></pre>
<pre><span style="color:black">>There is a neutron spec currently in discussion for Kilo to finally</span><o:p></o:p></pre>
<pre><span style="color:black">>fix MTU issues due to tunneling, that also tries to propagate MTU</span><o:p></o:p></pre>
<div>
<pre><span style="color:black">>inside instances: <a href="https://review.openstack.org/#/c/105989/" target="_blank">https://review.openstack.org/#/c/105989/</a></span><o:p></o:p></pre>
</div>
<div>
<pre><span style="color:black"> </span><o:p></o:p></pre>
</div>
<div>
<pre><span style="color:black">The problem is i do not know why the vm with 1500 mtu can not send large file? </span><o:p></o:p></pre>
</div>
<div>
<pre><span style="color:black">I found the packet send out all with DF, and is it because the DF set default by linux cause the packet</span><o:p></o:p></pre>
</div>
<div>
<pre><span style="color:black">be dropped? And the application do not handle the return back icmp packet with the smaller mtu?</span><o:p></o:p></pre>
</div>
<div>
<pre><span style="color:black"> </span><o:p></o:p></pre>
</div>
<div>
<pre><span style="color:black"> </span><o:p></o:p></pre>
</div>
<pre><span style="color:black">> </span><o:p></o:p></pre>
<pre><span style="color:black">>/Ihar</span><o:p></o:p></pre>
<pre><span style="color:black">>-----BEGIN PGP SIGNATURE-----</span><o:p></o:p></pre>
<pre><span style="color:black">>Version: GnuPG/MacGPG2 v2.0.22 (Darwin)</span><o:p></o:p></pre>
<pre><span style="color:black">> </span><o:p></o:p></pre>
<pre><span style="color:black">>iQEcBAEBCgAGBQJUThORAAoJEC5aWaUY1u571u4H/3EqEVPL1Q9KgymrudLpAdRh</span><o:p></o:p></pre>
<pre><span style="color:black">>fwNarwPWT8Ed+0x7WIXAr7OFXX1P90cKRAZKTlAEEI94vOrdr0s608ZX8awMuLeu</span><o:p></o:p></pre>
<pre><span style="color:black">>+LB6IA7nMpgJammfDb8zNmYLHuTQGGatXblOinvtm3XXIcNbkNu8840MTV3y/Jdq</span><o:p></o:p></pre>
<pre><span style="color:black">>Mndtz69TrjTrjn7r9REJ4bnRIlL4DGo+gufXPD49+yax1y/woefqwZPU13kO6j6R</span><o:p></o:p></pre>
<pre><span style="color:black">>Q0+MAy13ptg2NwX26OI+Sb801W0kpDXby6WZjfekXqxqv62fY1/lPQ3oqqJBd95K</span><o:p></o:p></pre>
<pre><span style="color:black">>EFe5NuogLV7UGH5vydQJa0eO2jw5lh8qLuHSShGcDEp/N6oQWiDzXYYYoEQdUic=</span><o:p></o:p></pre>
<pre><span style="color:black">>=jRQ/</span><o:p></o:p></pre>
<pre><span style="color:black">>-----END PGP SIGNATURE-----</span><o:p></o:p></pre>
<pre><span style="color:black">> </span><o:p></o:p></pre>
<pre><span style="color:black">>_______________________________________________</span><o:p></o:p></pre>
<pre><span style="color:black">>OpenStack-dev mailing list</span><o:p></o:p></pre>
<pre><span style="color:black">><a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a></span><o:p></o:p></pre>
<pre><span style="color:black">><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></span><o:p></o:p></pre>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>