<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Awesome information!</div>
<div><br>
</div>
<div>Thanks for sharing with the rest of the OpenStack community.</div>
<div><br>
</div>
<div>Edgar</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Matt Kassawara <<a href="mailto:mkassawara@gmail.com">mkassawara@gmail.com</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, December 10, 2014 at 8:52 AM<br>
<span style="font-weight:bold">To: </span>"Vasudevan, Swaminathan (PNB Roseville)" <<a href="mailto:swaminathan.vasudevan@hp.com">swaminathan.vasudevan@hp.com</a>><br>
<span style="font-weight:bold">Cc: </span>Edgar Magana <<a href="mailto:edgar.magana@workday.com">edgar.magana@workday.com</a>>, Phil Hopkins <<a href="mailto:phil.hopkins@gmail.com">phil.hopkins@gmail.com</a>>, Nicholas Chase <<a href="mailto:nchase@mirantis.com">nchase@mirantis.com</a>>,
"<a href="mailto:openstack-docs@lists.openstack.org">openstack-docs@lists.openstack.org</a>" <<a href="mailto:openstack-docs@lists.openstack.org">openstack-docs@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: DVR network traffic flow details<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">Thanks... I'll add this as soon as possible.</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Dec 9, 2014 at 8:25 PM, Vasudevan, Swaminathan (PNB Roseville)
<span dir="ltr"><<a href="mailto:swaminathan.vasudevan@hp.com" target="_blank">swaminathan.vasudevan@hp.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Hi Matt,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">I have put together the details and functions of each entity for the East-West routing and North-South without FloatingIP.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">I will send you an update on the “North-South with FloatingIP” tomorrow.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Please let me know if you have any questions or concerns.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"><u></u> <u></u></span></p>
<p><u></u><b><span>1)<span style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 7pt; line-height: normal; font-family: 'Times New Roman';">
</span></span></b><u></u><b>East-west for instances with or without a floating IP.<u></u><u></u></b></p>
<p><b><u></u> <u></u></b></p>
<p><img width="780" height="558" src="cid:image003.jpg@01D013DD.8DFE0740"><b><u></u><u></u></b></p>
<p class="MsoNormal"><span>1 </span><span style="font-family:Wingdings">à</span><b><span></span></b><span>The frame with destination ip as ‘vm2 ip’ is sent by vm1 towards its default gateway mac for red network which is r1-red-mac. The integration bridge
forwards this frame to DVR router r1. <b>( br-int)</b><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>2 </span><span style="font-family:Wingdings">à</span><span>The DVR router r1’s red subnet interface picks this frame and it then routes the IP packet in the frame<b>. ( router-namespace)<u></u><u></u></b></span></p>
<p class="MsoNormal"><b><span>NOTE: The router_namespace arp table is populated by the l3_agent dynamically.<u></u><u></u></span></b></p>
<p class="MsoNormal"><span><u></u><u></u></span></p>
<p class="MsoNormal"><span>3 </span><span style="font-family:Wingdings">à</span><span> After routing, the DVR router r1 puts the routed frame out of its green subnet interface.
<b>(router-namespace)</b><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>4 </span><span style="font-family:Wingdings">à</span><span> This frame is switched by the integration bridge towards the tunnel bridge along with tagging the frame with green network’s local-vlan tag.
<b>(br-int)</b><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>5 </span><span style="font-family:Wingdings">à</span><span> The tunnel bridge on CN1, replaces the frame’s source mac address with a Unique DVR MAC Address of its node (one unique dvr mac address is assigned per compute node by the
controller). The resulting frame is forwarded to CN2 by this tunnel bridge. Before forwarding, it also strips the local green-vlan tag and tunnels the frame with green-vni VXLAN id.
<b>( br-tun)</b><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>6 </span><span style="font-family:Wingdings">à</span><span> The tunnel bridge on CN2, picks up the tunneled frame, de-tunnels it and strips off the green-vni tag. It then adds its local green network vlan tag to the frame and forwards
the frame to the integration bridge. <b>( br-tun)</b><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>7 </span><span style="font-family:Wingdings">à</span><span> The integration bridge on CN2, identifies the incoming frame’s source mac address is a DVR Unique MAC Address (every compute node l2-agent knows all dvr unique mac addresses
used in the cloud). It then replaces the DVR Unique MAC Address with the green subnet interface MAC address and forwards the frame to vm2.
<b>(br-int)</b><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size: 11pt; font-family: Calibri, sans-serif;">br-int:<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="font-size: 11pt; font-family: Calibri, sans-serif;">Operations:<u></u><u></u></span></b></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;">If the packet is for the local VMs residing on the same subnet then it would be a regular switching that would happen between the VMs.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;">If the packet is for the local VMs residing on a different subnet then the packet from VM1 would come with a destination address of the Default gateway on the same subnet.
Then the “br-int” will forward those packets to the “router-namespace’s”. <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;">The br-int will also receive the routed packets from the “router-namespace” and then will switch those packets to the “br-tun” since the destination is on a different node.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size: 11pt; font-family: Calibri, sans-serif;">Router_Namespace:<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="font-size: 11pt; font-family: Calibri, sans-serif;">Operations:<u></u><u></u></span></b></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;">The router_namespace has rules to route or forward the packet to the destination interface.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;">The routed packet then enters the br-int again.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size: 11pt; font-family: Calibri, sans-serif;">br-tun:<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="font-size: 11pt; font-family: Calibri, sans-serif;">Operations:<u></u><u></u></span></b></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;">The br-tun replaces the source Mac with the “DVR Mac or the LMac” for the forwarding packets.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;">Also replaces the vlan tag with the destination vni tag and forwards the packet to the destination Node through the tunnel.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;">(
<b>Note:</b> L2 population helps to identify the VNI’s that belong to different VXLAN ids and where the nodes reside).<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;">The br-tun also replace the vni tag with a corresponding vlan tag for the received packets.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size: 11pt; font-family: Calibri, sans-serif;">2. North-South for instances without a Floating IP</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;">:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><img width="780" height="546" src="cid:image005.png@01D013DD.8DFE0740"></span><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><u></u> <u></u></span></p>
<p class="MsoNormal"><span>1 </span><span style="font-family:Wingdings">à</span><b><span></span></b><span>The frame with destination ip as ‘remote external network ip’ is sent by vm1 towards its default gateway mac for red network which is r1-red-mac. The
integration bridge forwards this frame to DVR router r1. <b>( br-int)</b><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>2 </span><span style="font-family:Wingdings">à</span><span>The DVR router r1’s red subnet interface picks this frame and it then redirects the IP packet in the frame on the same-interface since it’s destination is an external network’s
ip address<b>. ( router-namespace)<u></u><u></u></b></span></p>
<p class="MsoNormal"><b><span>NOTE: The router_namespace arp table is populated by the l3_agent dynamically.<u></u><u></u></span></b></p>
<p class="MsoNormal"><span><u></u><u></u></span></p>
<p class="MsoNormal"><span>3 </span><span style="font-family:Wingdings">à</span><span> After routing, the DVR router r1 puts the routed frame out of its red subnet interface.
<b>(router-namespace)</b><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>4 </span><span style="font-family:Wingdings">à</span><span> This frame is switched by the integration bridge towards the tunnel bridge along with tagging the frame with red network’s local-vlan tag.
<b>(br-int)</b><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>5 </span><span style="font-family:Wingdings">à</span><span> The tunnel bridge on CN1, replaces the frame’s source mac address with a Unique DVR MAC Address of its node (one unique dvr mac address is assigned per compute node by the
controller). The resulting frame is forwarded to Network Node or Service Node by this tunnel bridge. Before forwarding, it also strips the local red-vlan tag and tunnels the frame with red-vni VXLAN id for the Network-Node or Service Node.
<b>( br-tun)</b><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>6 </span><span style="font-family:Wingdings">à</span><span> The tunnel bridge on Network Node or Service Node, picks up the tunneled frame, de-tunnels it and strips off the red-vni tag. It then adds its local red network vlan tag
to the frame and forwards the frame to the integration bridge. <b>( br-tun)</b><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>7 </span><span style="font-family:Wingdings">à</span><span> The integration bridge on Network Node or Service Node, identifies the incoming frame’s source mac address is a DVR Unique MAC Address (every compute node l2-agent knows
all dvr unique mac addresses used in the cloud). It then replaces the DVR Unique MAC Address with the red subnet interface MAC address and forwards the frame to router-namespace since the MAC belongs to the gateway MAC.
<b>(br-int)<u></u><u></u></b></span></p>
<p class="MsoNormal"><b><span><u></u> <u></u></span></b></p>
<p class="MsoNormal"><span>8</span><span style="font-family:Wingdings">à</span><span> The router-namespace in the Network Node will then receive the packet through the “red” interface and then will forward the packet to the “red” router-gateway interface in
the SNAT Namespace. <b>( router_namespace)</b><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>9</span><span style="font-family:Wingdings">à</span><span>The snat-namespace will then forward the packet to the External Network after replacing the source address with the Gateway address.<b> ( snat_namespace).</b><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>Similarly the return packet’s destination address when received by the “SNAT-Namespace” is replaced by the internal private address and then forwarded to the right interface that it belongs to.<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span>Thanks<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span>Swami<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span></span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif;">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif;"> Vasudevan, Swaminathan (PNB Roseville)
<br>
<b>Sent:</b> Monday, December 08, 2014 6:09 PM<br>
<b>To:</b> 'Matt Kassawara'; Edgar Magana; Phil Hopkins; Nicholas Chase; <a href="mailto:openstack-docs@lists.openstack.org" target="_blank">
openstack-docs@lists.openstack.org</a><br>
<b>Subject:</b> RE: DVR network traffic flow details<u></u><u></u></span></p>
</div>
</div>
<span class="">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Hi Matt,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Thanks for the information.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Regarding the details on “tables and flows” – I don’t think that is required for a regular user. But if we wanted to include a session on “Under the
hood” or something, then we can add those details in the admin-guide.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Let me know if you wanted to add an Under the hood session and I can let you know the top level rules and tables just added for DVR in br-int and
br-tun.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">I will review the doc with my team and will get back to your on the feedback within an couple of days.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Thanks<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Swami<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif;">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif;"> Matt Kassawara [<a href="mailto:mkassawara@gmail.com" target="_blank">mailto:mkassawara@gmail.com</a>]
<br>
<b>Sent:</b> Monday, December 08, 2014 6:06 PM<br>
<b>To:</b> Vasudevan, Swaminathan (PNB Roseville); Edgar Magana; Phil Hopkins; Nicholas Chase;
<a href="mailto:openstack-docs@lists.openstack.org" target="_blank">openstack-docs@lists.openstack.org</a><br>
<b>Subject:</b> DVR network traffic flow details<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
</span>
<div>
<p class="MsoNormal">Swami/Edgar,<u></u><u></u></p>
</div>
<div>
<div class="h5">
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">I see three distinct traffic flows with DVR:<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">1) North-south for instances without a floating IP.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2) North-south for instances with a floating IP.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">3) East-west for instances with or without a floating IP.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I drew a diagrams for each flow that includes the primary components and their connections. For simplicity (and space considerations), I left out components that don't directly apply to DVR (such as metadata).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I also wrote a series of steps describing flow #1 in detail without getting too technical. However, I'm not exactly sure about our target audience. I would expect them to know enough about networking and neutron (perhaps from other parts
of the networking guide) prior to attempting DVR, but perhaps not enough to understand tables and flows in Open vSwitch. How much detail should we provide?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I would appreciate if someone could review the diagrams for all flows and the steps describing flow #1 and provide feedback/corrections. In the interest of time, can someone also provide the steps describing flows #2 and #3 so I don't have
to reverse-engineer them too? You can follow my level of detail in flow #1 or suggest something with more or less detail based on the target audience.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Ideally, I would like to submit most if not all of this content by the end of this week for inclusion into the official networking guide. I already came across several people who stumbled upon it via search engines and successfully deployed
DVR in a test environment, so at least the configuration works.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="https://github.com/ionosphere80/openstack-networking-guide/blob/master/scenario-dvr/scenario-dvr.md" target="_blank">https://github.com/ionosphere80/openstack-networking-guide/blob/master/scenario-dvr/scenario-dvr.md</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Matt<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</span>
</body>
</html>