<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The select_all = 1 is supposed to mirror all the packets.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Referring to the documentation (<a href="http://openvswitch.org/support/dist-docs/ovs-vswitchd.conf.db.5.html">http://openvswitch.org/support/dist-docs/ovs-vswitchd.conf.db.5.html</a>),<o:p></o:p></span></p>
<pre><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">“</span><b>select_all</b>: boolean<o:p></o:p></pre>
<pre> If true, every packet arriving or departing on any port is<o:p></o:p></pre>
<pre> selected for mirroring.<o:p></o:p></pre>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">”<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">And for OVS 2.5,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<pre><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">“</span>In Open<o:p></o:p></pre>
<pre> vSwitch 2.5 and later, mirroring occurs just after a packet first<o:p></o:p></pre>
<pre> becomes eligible, using the packet as it exists at that point; …<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>in Open vSwitch 2.4, the modifications are never visible to<o:p></o:p></pre>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> mirrors, whereas in Open vSwitch 2.5 and later modifications made<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> before the first output that makes it eligible for mirroring to a par</span><span style="font-size:10.0pt;font-family:"Cambria Math",serif">‐</span><span style="font-size:10.0pt;font-family:"Courier New""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ticular destination are visible.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">”<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I believe, if the very first flow is dropping unicast packets, you might not be able to mirror them.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Maybe you can monitor the flow-tables on each OVS bridge while sending traffic and see which flows’ count increases. Something like,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">watch –n 2 “ovs-ofctl dump-flows <bridge name>”
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Kaustubh<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Sterdnot Shaken [mailto:sterdnotshaken@gmail.com]
<br>
<b>Sent:</b> Wednesday, March 22, 2017 12:24 PM<br>
<b>To:</b> Kaustubh Kelkar <kaustubh.kelkar@casa-systems.com><br>
<b>Subject:</b> Re: [Openstack] VM can receive traffic, but not send it<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Here's was my first mirror setup:<o:p></o:p></p>
<div style="margin-left:30.0pt">
<p class="MsoNormal">ip link add name dummy3 type dummy<br>
ip link set dev dummy3 up<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="margin-left:30.0pt">
<p class="MsoNormal">ovs-vsctl add-port br-ex3 dummy3<br>
<br>
ovs-vsctl -- set bridge br-ex3 mirrors=@m \<br>
-- --id=@src get port pat-ex3-bss \<br>
-- --id=@mir get port dummy3 \<br>
-- --id=@m create mirror name=ovs_mirror3 select-dst-port=@src select-src-port=@src output-port=@mir select-all=true<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">And here's the one I did by copying your example:<o:p></o:p></p>
<div style="margin-left:30.0pt">
<p class="MsoNormal">ip link add name dummy3 type dummy<br>
ip link set dev dummy3 up<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="margin-left:30.0pt">
<p class="MsoNormal">ovs-vsctl add-port br-ex3 dummy3<br>
<br>
ovs-vsctl -- set Bridge br-ex3 mirrors=@m \<br>
-- --id=@dummy3 get Port dummy3 \<br>
-- --id=@pat-ex3-bss get Port pat-ex3-bss \<br>
-- --id=@m create Mirror name=mirror0 \<br>
select-dst-port=@pat-ex3-bss select-src-port=@pat-ex3-bss \<br>
output-port=@dummy3 select_all=1<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Both yield the same results. When I tcpdump the respective dummy interface attached to br-ex3, I only see broadcast traffic for the VM in question, I never see unicast traffic (case and point, if I ping the
broadcast address on the VM, then traffic show's up in the tcpdump). I can do a tcpdump on the external interface and see the unicast traffic though, but I need to see where it's breaking in the OVS bridges.
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Is there some trick to mirror unicast dataplane traffic?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks in advance!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Mar 22, 2017 at 10:07 AM, Kaustubh Kelkar <<a href="mailto:kaustubh.kelkar@casa-systems.com" target="_blank">kaustubh.kelkar@casa-systems.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Sterdnot Shaken [mailto:<a href="mailto:sterdnotshaken@gmail.com" target="_blank">sterdnotshaken@gmail.com</a>]
<br>
<b>Sent:</b> Tuesday, March 21, 2017 8:54 PM<br>
<b>To:</b> Kaustubh Kelkar <<a href="mailto:kaustubh.kelkar@casa-systems.com" target="_blank">kaustubh.kelkar@casa-systems.com</a>><br>
<b>Cc:</b> Richard Jones <<a href="mailto:rjones@suse.com" target="_blank">rjones@suse.com</a>>;
<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
<b>Subject:</b> Re: [Openstack] VM can receive traffic, but not send it</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Thanks for everyone's kind help!
<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Steve: I will try and turn off the offload features and see if that helps. Thanks!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Neil: I will also check and make sure neither RPF nor TTL are posing any issues.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><br>
Kaustubh: Is there a reason the mirror approach only seems to work on some of the OVS bridges, but not others? if I follow your instructions, I can see traffic when I set up a mirror on some bridges, but not others... Do I need to put these OVS bridges into
promiscuous mode before the mirror will work?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[Kaustubh] I don’t recall putting the bridge in promiscuous mode, but it has been a while since I had looked
at this. How are you setting up the mirrors? You would need to mirror a specific port of the bridge, not the bridge itself.</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks!!<o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Tue, Mar 21, 2017 at 9:42 AM, Kaustubh Kelkar <<a href="mailto:kaustubh.kelkar@casa-systems.com" target="_blank">kaustubh.kelkar@casa-systems.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">You can narrow down the point where the packets are being dropped by mirroring and tracing packets
on OVS bridge ports. I use a script that does the following (as root):</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">ip link add name sniff0 type dummy</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">ip link set dev sniff0 up</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">ovs-vsctl add-port br1 sniff0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">ovs-vsctl -- set Bridge br1 mirrors=@m \</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-- --id=@sniff0 get Port sniff0 \</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-- --id=@eth0 get Port eth0 \</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-- --id=@m create Mirror name=mirror0 \</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">select-dst-port=@eth0 select-src-port=@eth0 \</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">output-port=@sniff0 select_all=1</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">and to delete,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">ovs-vsctl clear Bridge br1 mirrors</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">ovs-vsctl del-port br1 sniff0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">ip link del dev sniff0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">where eth0 is the point of packet capture and br1 is the bridge eth0 resides in. Then, you can run
tcpdump on sniff0. </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Create such mirror ports on
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">1) phy-br-ex on external OVS bridge</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">2) int-br-ex on integration bridge</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">3) qvo-xxx on integration bridge</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Also capture packets on qvb-xxx on the linux bridge having the tap interface of the VM. Hopefully,
this will provide us more clues.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Kaustubh</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Sterdnot Shaken [mailto:<a href="mailto:sterdnotshaken@gmail.com" target="_blank">sterdnotshaken@gmail.com</a>]
<br>
<b>Sent:</b> Monday, March 20, 2017 9:17 PM<br>
<b>To:</b> Richard Jones <<a href="mailto:rjones@suse.com" target="_blank">rjones@suse.com</a>><br>
<b>Cc:</b> <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
<b>Subject:</b> Re: [Openstack] VM can receive traffic, but not send it</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Wow! Thanks for answering both of my questions!<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">So, I did some things you suggested, including setting the MSS in iperf to something small (1000 bytes) and tested with no improvement. I then changed the VM running on Openstack to have
an MTU of 1000 and retested with no improvement. I noticed that the node I was testing against was reporting back to the VM on Openstack that it had an MSS of 8960, so just for the heck of it, I changed the remote node's (server outside of Openstack) MTU also
to 1000 bytes and retested with no improvement. (The effects of all of these tests were also validated by checking mss settings in the tcp header via tcpdump).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">To simplify the equation, I ditched the iperf for the time being and just did a simple "telnet 'remote server' 8080" test from the remote server to the VM in Openstack, while capturing
packets all along the way (4 different points along the network path). Every point saw the same packets, including the VM's tap interface as expected. I then reversed the test by initiating the tcp session on the VM in Openstack to the remote server while
running the packet captures at those same points having set the remote server to respond with a TCP Reset. From VM to Remote server traffic looked correct with expected TCP SYN. The TCP Reset that the remote server responded with passed all 4 points of the
network, including the external interface on the Compute node where the VM resides, but the TAP interface that connects to the VM NEVER sees the Reset. I can recreate this condition over and over.
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">So, thanks to your ideas Richard, I'm no longer convinced this is an MTU issue. What would prevent a TCP related response from being forwarded from the external interface to the intended
VM? The security group we have applied to this VM is wide open, so I can't imagine that is the cause...
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Here are 2 packet captures where I initiated a telnet to the remote server from the VM in Openstack. As said above, I set the remote server to respond with a reset. The top one is from
the physical interface on the Compute node where the VM resides and the other, the tap interface to that VM:<br>
<br>
[(openstack-mitaka) root@prv-0-18-compute user]# tcpdump -nni eth0 host x.y.120.23 and host x.y.224.45<br>
tcpdump: WARNING: eth0: no IPv4 address assigned<br>
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode<br>
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes<br>
19:10:13.143931 IP x.y.120.23.53877 > x.y.224.45.8080: Flags [S], seq 3131027441, win 8192, options [mss 960,nop,wscale 8,nop,nop,sackOK], length 0<br>
19:10:13.147951 IP x.y.224.45.8080 > x.y.120.23.53877: Flags [R.], seq 0, ack 3131027442, win 0, length 0<br>
19:10:16.156520 IP x.y.120.23.53877 > x.y.224.45.8080: Flags [S], seq 3131027441, win 8192, options [mss 960,nop,wscale 8,nop,nop,sackOK], length 0<br>
19:10:16.157693 IP x.y.224.45.8080 > x.y.120.23.53877: Flags [R.], seq 0, ack 1, win 0, length 0<br>
19:10:22.157407 IP x.y.120.23.53877 > x.y.224.45.8080: Flags [S], seq 3131027441, win 8192, options [mss 960,nop,nop,sackOK], length 0<br>
19:10:22.158682 IP x.y.224.45.8080 > x.y.120.23.53877: Flags [R.], seq 0, ack 1, win 0, length 0<br>
<br>
<br>
[(openstack-mitaka) root@prv-0-18-compute user]# tcpdump -nni tap3bbe0f9d-6b host x.y.120.23 and host x.y.224.45<br>
tcpdump: WARNING: tap3bbe0f9d-6b: no IPv4 address assigned<br>
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode<br>
listening on tap3bbe0f9d-6b, link-type EN10MB (Ethernet), capture size 65535 bytes<br>
19:10:13.143739 IP x.y.120.23.53877 > x.y.224.45.8080: Flags [S], seq 3131027441, win 8192, options [mss 960,nop,wscale 8,nop,nop,sackOK], length 0<br>
19:10:16.156499 IP x.y.120.23.53877 > x.y.224.45.8080: Flags [S], seq 3131027441, win 8192, options [mss 960,nop,wscale 8,nop,nop,sackOK], length 0<br>
19:10:22.157384 IP x.y.120.23.53877 > x.y.224.45.8080: Flags [S], seq 3131027441, win 8192, options [mss 960,nop,nop,sackOK], length 0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Any ideas? Thanks in advance for your help!!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Steve
<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Mon, Mar 20, 2017 at 4:17 PM, Richard Jones <<a href="mailto:rjones@suse.com" target="_blank">rjones@suse.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">You might consider taking a packet trace of the start of an upload to see what the TCP MSS (Maximum Segment Size) options look like and perhaps compare between the different configs.
Also, you could consider either using netperf and having it tweak the MSS to a smaller value (test-specific -G option if I recall correctly), or just try dropping the MTU of your VM before you try the upload.<br>
<br>
Another way to use netperf to "probe" without tweaking MSS or MTU settings would be to use the TCP_RR test with increasing request/response sizes. If there is indeed an MTU issue somewhere along the way, as you walk the request/response size up to the local
MTU, you should see the test performance drop off a cliff if not go fully to zero.<br>
<br>
Does the port for the VM have a security group rule permitting ICMP traffic in? Offhand I wouldn't expect that to be different between the two network setups you've described because I'd not have expected the virtual router to pay attention to an arriving
ICMP Destination Unreachable, Datagram Too Big message to have the routed version work, but it seemed a reasonable straw at which to grasp.<br>
<br>
rick jones<br>
<br>
PS perhaps iperf has a similar option to set the TCP MSS, I've not looked.<br>
<br>
>>> Sterdnot Shaken <<a href="mailto:sterdnotshaken@gmail.com" target="_blank">sterdnotshaken@gmail.com</a>> 03/20/17 3:07 PM >>><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Our info:<br>
<br>
Openstack version: Mitaka (using OVS 2.5)<br>
Firewall driver: Openvswitch<br>
<br>
Anyone know why VM's that are directly on a Flat Provider Network (so the<br>
VM would have a public IP directly assigned to it) can download data just<br>
fine, but when we try and upload anything (iperf where the VM is the client<br>
or something even like <a href="http://speedtest.net" target="_blank">speedtest.net</a> (upload portion)) the VM simply can't<br>
get data out to the intended destination? Again, download works great,<br>
upload doesn't.<br>
<br>
If I take that VM and change it's interface to be a tenant network one that<br>
has a Openstack HA virtual router, everything (upload and download) works<br>
perfectly. The problem only seems to be apparent when the VM is directly on<br>
the external network.<br>
<br>
It seems like an MTU issue, but I don't see how... Here are the MTU's of<br>
the part's at play:<br>
<br>
VM: 1500<br>
br-int (specific interface connecting to VM) - 9216<br>
br-ex - (can't tell what that MTU is set to)<br>
<br>
Any help would be GREATLY appreciated.<br>
<br>
Steve<o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>