<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></a></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Armando M. [mailto:armamig@gmail.com]
<br>
<b>Sent:</b> Tuesday, June 14, 2016 12:50 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org><br>
<b>Subject:</b> Re: [openstack-dev] [neutron][ovs] The way we deal with MTU<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 13 June 2016 at 22:22, Terry Wilson <<a href="mailto:twilson@redhat.com" target="_blank">twilson@redhat.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">> So basically, as long as we try to plug ports with different MTUs into the same bridge, we are utilizing a bug in Open vSwitch, that may break us any time.<br>
><br>
> I guess our alternatives are:<br>
> - either redesign bridge setup for openvswitch to e.g. maintain a bridge per network;<br>
> - or talk to ovs folks on whether they may support that for us.<br>
><br>
> I understand the former option is too scary. It opens lots of questions, including upgrade impact since it will obviously introduce a dataplane downtime. That would be a huge shift in paradigm, probably too huge to swallow. The latter option may not fly with
vswitch folks. Any better ideas?<br>
<br>
I know I've heard from people who'd like to be able to support both<br>
DPDK and non-DPDK workloads on the same node. The current<br>
implementation with a single br-int (and thus datapath) makes that<br>
impossible to pull of with good performance. So there may be other<br>
reasons to consider introducing multiple isolated bridges: MTUs,<br>
datapath_types, etc.<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[Mooney, Sean K]
<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I just noticed this now but I just wanted to share some of the rational as to why we explicitly do not support running both datapaths on the same host today.<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">We experiment with using both datapaths during the juno cycle when we were frist upstreaming support for ovs-dpdk.<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">To efficiently enable both data paths we determined that you would have to duplicate all bridges for each data path otherwise there is a significant performance<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Penalty that degrades the performance of both datapaths.<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The only way to interconnect bridge of different data paths in ovs is to use veth pairs. Even in the case of the kernel datapath the use<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Of veth pairs is a significant performance hit compared to patchports. Adding a veth interface to the dpdk datapath is very costly from<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">A dpdk perpective to rx/tx packets as it take significantly more cpu cycles to process packet from veth interfaces then dpdk interfaces.<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">What we determined at the time was to make this configuration work effectively you would have to have 2 copies of every bridge and
<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Either modify the existing agent significantly or run two copies of the ovs agent on the same host. If you use two agents on the same host<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">With two configfiles specifying different bridge names e.g. br-int and br-int-dpdk br-tun,br-tun-dpdk and br-ex br-ex-dpdk it should be possible to support
today.<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">You might need to make minor changes to the agent and server to ensure the agents are both reported separately in the db and<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">You would need to provide some mechanism to request the use of kernel vhost or vhost-user. Unfortunately there is no construct currently in<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Neutron that can be used directly for that and also the nova scheduler does not currently have any idea regarding the vif-types or networking backend supported
on each<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Compute host.<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The scheduler side could be addressed by reusing the resource provider framework that jay pipes is working on. In essence each compute node would be a provider
of vif-types.<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">When you boot a vm you would also pass a desired vif-type and when nova is scheduling it will fileter to only host of that type. When nova asks neutron
to bind the<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Port it would pass the requested vif-type to neutron which would then use it for the port binding. Ian wells and I proposed a mechanism for this over the
last few cycles that<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Should be possible to intergrate cleanly with os-vif when nova and neutron have both adopted its uses.<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><a href="https://review.openstack.org/#/c/190917/7/specs/mitaka/approved/nova-neutron-binding-negotiation.rst">https://review.openstack.org/#/c/190917/7/specs/mitaka/approved/nova-neutron-binding-negotiation.rst</a><o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">while requesting a vif type is somewhat of a leaky abstraction it does not mean that you will know what the neutron backend is.<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">A vhost-user interface for example could be ovs-dpdk or vpp or snabb swtich or ovs-fastpath. So while it is leaking the capability
<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">To provide a vhost-user interface it does not leak the implantation which still maintains some level of abstraction and flexablity
<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">For an operator. For a tenant other then performance they cannot detect if they are using vhost-user or kernel-vhost in any way since they
<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">All they see is a virtiio-net interface in either case.<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">if there is interest in supporting both datapaths concurrently and people are open to having multiple copies of the ovs l2, and possible l3/dhcp agents
on the same host<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">then I would be happy to help with that effort but the added complexity and operator overhead of managing two copies of the neutron agents on each host
is why we<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">have not tried to enable this configuration to date.<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></i></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal">Incidentally this is something that Nova is already capable of handling (ie. wiring VM's in different bridges) thanks to [1], and with some minor additions as being discussed in the context of [2] vlan-aware-vms, we can open up the possibility
to this deployment model in a not so distant future.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></i></b></p>
</div>
<div>
<p class="MsoNormal">[1] <a href="https://blueprints.launchpad.net/nova/+spec/neutron-ovs-bridge-name">
https://blueprints.launchpad.net/nova/+spec/neutron-ovs-bridge-name</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[2] <a href="http://lists.openstack.org/pipermail/openstack-dev/2016-June/097025.html">http://lists.openstack.org/pipermail/openstack-dev/2016-June/097025.html</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><span style="color:#888888"><br>
Terry</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
On Mon, Jun 13, 2016 at 11:49 AM, Ihar Hrachyshka <<a href="mailto:ihrachys@redhat.com">ihrachys@redhat.com</a>> wrote:<br>
> Hi all,<br>
><br>
> in Mitaka, we introduced a bunch of changes to the way we handle MTU in Neutron/Nova, making sure that the whole instance data path, starting from instance internal interface, thru hybrid bridge, into the br-int; as well as router data path (qr) have proper
MTU value set on all participating devices. On hypervisor side, both Nova and Neutron take part in it, setting it with ip-link tool based on what Neutron plugin calculates for us. So far so good.<br>
><br>
> Turns out that for OVS, it does not work as expected in regards to br-int. There was a bug reported lately:
<a href="https://launchpad.net/bugs/1590397" target="_blank">https://launchpad.net/bugs/1590397</a><br>
><br>
> Briefly, when we try to set MTU on a device that is plugged into a bridge, and if the bridge already has another port with lower MTU, the bridge itself inherits MTU from that latter port, and Linux kernel (?) does not allow to set MTU on the first device
at all, making ip link calls ineffective.<br>
><br>
> AFAIU this behaviour is consistent with Linux bridging rules: you can’t have ports of different MTU plugged into the same bridge.<br>
><br>
> Now, that’s a huge problem for Neutron, because we plug ports that belong to different networks (and that hence may have different MTUs) into the same br-int bridge.<br>
><br>
> So I played with the code locally a bit and spotted that currently, we set MTU for router ports before we move their devices into router namespaces. And once the device is in a namespace, ip-link actually works. So I wrote a fix with a functional test that
proves the point: <a href="https://review.openstack.org/#/c/327651/" target="_blank">
https://review.openstack.org/#/c/327651/</a> The fix was validated by the reporter of the original bug and seems to fix the issue for him.<br>
><br>
> It’s suspicious that it works from inside a namespace but not when the device is still in the root namespace. So I reached out to Jiri Benc from our local Open vSwitch team, and here is a quote:<br>
><br>
> ===<br>
><br>
> "It's a bug in ovs-vswitchd. It doesn't see the interface that's in<br>
> other netns and thus cannot enforce the correct MTU.<br>
><br>
> We'll hopefully fix it and disallow incorrect MTU setting even across<br>
> namespaces. However, it requires significant effort and rework of ovs<br>
> name space handling.<br>
><br>
> You should not depend on the current buggy behavior. Don't set MTU of<br>
> the internal interfaces higher than the rest of the bridge, it's not<br>
> supported. Hacking this around by moving the interface to a netns is<br>
> exploiting of a bug.<br>
><br>
> We can certainly discuss whether this limitation could be relaxed.<br>
> Honestly, I don't know, it's for a discussion upstream. But as of now,<br>
> it's not supported and you should not do it.”<br>
><br>
> So basically, as long as we try to plug ports with different MTUs into the same bridge, we are utilizing a bug in Open vSwitch, that may break us any time.<br>
><br>
> I guess our alternatives are:<br>
> - either redesign bridge setup for openvswitch to e.g. maintain a bridge per network;<br>
> - or talk to ovs folks on whether they may support that for us.<br>
><br>
> I understand the former option is too scary. It opens lots of questions, including upgrade impact since it will obviously introduce a dataplane downtime. That would be a huge shift in paradigm, probably too huge to swallow. The latter option may not fly with
vswitch folks. Any better ideas?<br>
><br>
> It’s also not clear whether we want to proceed with my immediate fix. Advices are welcome.<br>
><br>
> Thanks,<br>
> Ihar<br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>