<div dir="ltr">The tenant router has the initial delay to/from both external and internal.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Oct 15, 2013 at 4:29 PM, Narayan Desai <span dir="ltr"><<a href="mailto:narayan.desai@gmail.com" target="_blank">narayan.desai@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">This sounds like a timer in the qrouter path, since you can get to the tenant router with predictable, low latency, right?<div>
<br></div><div>This is one of the big problems with network datapaths implemented fully in software.</div>
<div> -nld</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Oct 15, 2013 at 1:06 PM, Jacob Godin <span dir="ltr"><<a href="mailto:jacobgodin@gmail.com" target="_blank">jacobgodin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Jay,<div><br></div><div>If I stop and repeat immediately, ping times are fine. However, if I wait 5+ secs, they spike up during the first packet again.</div>
<div><br></div><div>I'm running OVS 1.4.0, someone recommended upgraded to 1.9.x from the Havana repo.</div>
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Oct 15, 2013 at 2:41 PM, Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Jacob,<br>
<br>
What you are witnessing, I believe, is OVS "learning" the flows and MAC addresses of the various compute nodes involved in the communication path between the source and target interfaces.<br>
<br>
If you repeat the pings, do you see the same latency on the first ping?<br>
<br>
Best,<br>
-jay<div><br>
<br>
On 10/15/2013 10:37 AM, Jacob Godin wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
Hi folks,<br>
<br>
I'm experiencing a weird issue with OpenStack Networking + OpenvSwitch.<br>
My setup consists of several compute nodes, and a networking node (l3,<br>
OVS, dhcp, etc.). These are connected via a Gigabit switch, and it is no<br>
where near capacity.<br>
<br>
It seems that the first packet being sent through a quantum router is<br>
delayed by several hundred milliseconds. Here is some sample ping output:<br>
<br></div>
_VM(comp node 1) -> VM(comp node 2)_<div><br>
<br>
# ping 10.199.0.7<br>
PING 10.199.0.7 (10.199.0.7) 56(84) bytes of data.<br></div>
64 bytes from 10.199.0.7 <<a href="http://10.199.0.7" target="_blank">http://10.199.0.7</a>>: icmp_seq=1 ttl=64<br>
time=3.45 ms<br>
64 bytes from 10.199.0.7 <<a href="http://10.199.0.7" target="_blank">http://10.199.0.7</a>>: icmp_seq=2 ttl=64<br>
time=0.792 ms<br>
64 bytes from 10.199.0.7 <<a href="http://10.199.0.7" target="_blank">http://10.199.0.7</a>>: icmp_seq=3 ttl=64<br>
time=0.837 ms<br>
64 bytes from 10.199.0.7 <<a href="http://10.199.0.7" target="_blank">http://10.199.0.7</a>>: icmp_seq=4 ttl=64<br>
time=0.864 ms<br>
<br>
_VM -> qrouter_<div><br>
<br>
# ping 10.199.0.1<br>
PING 10.199.0.1 (10.199.0.1) 56(84) bytes of data.<br></div>
64 bytes from 10.199.0.1 <<a href="http://10.199.0.1" target="_blank">http://10.199.0.1</a>>: icmp_seq=1 ttl=64<br>
time=248 ms<br>
64 bytes from 10.199.0.1 <<a href="http://10.199.0.1" target="_blank">http://10.199.0.1</a>>: icmp_seq=2 ttl=64<br>
time=0.512 ms<br>
64 bytes from 10.199.0.1 <<a href="http://10.199.0.1" target="_blank">http://10.199.0.1</a>>: icmp_seq=3 ttl=64<br>
time=0.553 ms<br>
64 bytes from 10.199.0.1 <<a href="http://10.199.0.1" target="_blank">http://10.199.0.1</a>>: icmp_seq=4 ttl=64<br>
time=0.533 ms<br>
64 bytes from 10.199.0.1 <<a href="http://10.199.0.1" target="_blank">http://10.199.0.1</a>>: icmp_seq=5 ttl=64<br>
time=0.679 ms<br>
<br>
_qrouter -> VM_<div><br>
<br>
# ip netns exec qrouter-XXXXX ping 10.199.0.7<br>
PING 10.199.0.7 (10.199.0.7) 56(84) bytes of data.<br></div>
64 bytes from 10.199.0.7 <<a href="http://10.199.0.7" target="_blank">http://10.199.0.7</a>>: icmp_req=1 ttl=64<br>
time=576 ms<br>
64 bytes from 10.199.0.7 <<a href="http://10.199.0.7" target="_blank">http://10.199.0.7</a>>: icmp_req=2 ttl=64<br>
time=0.530 ms<br>
64 bytes from 10.199.0.7 <<a href="http://10.199.0.7" target="_blank">http://10.199.0.7</a>>: icmp_req=3 ttl=64<br>
time=0.597 ms<br>
64 bytes from 10.199.0.7 <<a href="http://10.199.0.7" target="_blank">http://10.199.0.7</a>>: icmp_req=4 ttl=64<br>
time=0.723 ms<br>
64 bytes from 10.199.0.7 <<a href="http://10.199.0.7" target="_blank">http://10.199.0.7</a>>: icmp_req=5 ttl=64<br>
time=0.677 ms<br>
<br>
_qrouter -> Internet_<div><br>
<br>
# ip netns exec qrouter-XXXXX ping 8.8.8.8<br>
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.<br></div>
64 bytes from 8.8.8.8 <<a href="http://8.8.8.8" target="_blank">http://8.8.8.8</a>>: icmp_req=1 ttl=43 time=267 ms<br>
64 bytes from 8.8.8.8 <<a href="http://8.8.8.8" target="_blank">http://8.8.8.8</a>>: icmp_req=2 ttl=43 time=37.0 ms<br>
64 bytes from 8.8.8.8 <<a href="http://8.8.8.8" target="_blank">http://8.8.8.8</a>>: icmp_req=3 ttl=43 time=37.2 ms<br>
64 bytes from 8.8.8.8 <<a href="http://8.8.8.8" target="_blank">http://8.8.8.8</a>>: icmp_req=4 ttl=43 time=37.3 ms<div><br>
<br>
<br>
Here's a tcpdump on the qrouter of a ping from a vm on that network. It<br>
doesn't appear to show the large delay:<br>
14:33:38.024040 fa:16:3e:36:8e:f2 > fa:16:3e:99:85:5d, ethertype IPv4<br></div>
(0x0800), length 98: 10.199.0.4 > 10.199.0.1 <<a href="http://10.199.0.1" target="_blank">http://10.199.0.1</a>>: ICMP<div><br>
echo request, id 29953, seq 1, length 64<br>
14:33:38.024089 fa:16:3e:99:85:5d > fa:16:3e:36:8e:f2, ethertype IPv4<br></div>
(0x0800), length 98: 10.199.0.1 > 10.199.0.4 <<a href="http://10.199.0.4" target="_blank">http://10.199.0.4</a>>: ICMP<div><br>
echo reply, id 29953, seq 1, length 64<br>
14:33:38.526725 fa:16:3e:36:8e:f2 > fa:16:3e:99:85:5d, ethertype IPv4<br></div>
(0x0800), length 98: 10.199.0.4 > 10.199.0.1 <<a href="http://10.199.0.1" target="_blank">http://10.199.0.1</a>>: ICMP<div><br>
echo request, id 29953, seq 2, length 64<br>
14:33:38.526781 fa:16:3e:99:85:5d > fa:16:3e:36:8e:f2, ethertype IPv4<br></div>
(0x0800), length 98: 10.199.0.1 > 10.199.0.4 <<a href="http://10.199.0.4" target="_blank">http://10.199.0.4</a>>: ICMP<div><br>
echo reply, id 29953, seq 2, length 64<br>
14:33:39.526943 fa:16:3e:36:8e:f2 > fa:16:3e:99:85:5d, ethertype IPv4<br></div>
(0x0800), length 98: 10.199.0.4 > 10.199.0.1 <<a href="http://10.199.0.1" target="_blank">http://10.199.0.1</a>>: ICMP<div><br>
echo request, id 29953, seq 3, length 64<br>
14:33:39.527000 fa:16:3e:99:85:5d > fa:16:3e:36:8e:f2, ethertype IPv4<br></div>
(0x0800), length 98: 10.199.0.1 > 10.199.0.4 <<a href="http://10.199.0.4" target="_blank">http://10.199.0.4</a>>: ICMP<div><br>
echo reply, id 29953, seq 3, length 64<br>
14:33:39.665664 fa:16:3e:61:ef:25 > ff:ff:ff:ff:ff:ff, ethertype ARP<br>
(0x0806), length 42: Request who-has 10.199.0.7 tell 10.199.0.9, length 28<br>
14:33:40.526963 fa:16:3e:36:8e:f2 > fa:16:3e:99:85:5d, ethertype IPv4<br></div>
(0x0800), length 98: 10.199.0.4 > 10.199.0.1 <<a href="http://10.199.0.1" target="_blank">http://10.199.0.1</a>>: ICMP<div><br>
echo request, id 29953, seq 4, length 64<br>
14:33:40.527021 fa:16:3e:99:85:5d > fa:16:3e:36:8e:f2, ethertype IPv4<br></div>
(0x0800), length 98: 10.199.0.1 > 10.199.0.4 <<a href="http://10.199.0.4" target="_blank">http://10.199.0.4</a>>: ICMP<div><br>
echo reply, id 29953, seq 4, length 64<br>
<br>
And a dump from the VM performing the ping:<br>
14:34:59.897783 fa:16:3e:36:8e:f2 > fa:16:3e:99:85:5d, ethertype IPv4<br></div>
(0x0800), length 98: 10.199.0.4 > 10.199.0.1 <<a href="http://10.199.0.1" target="_blank">http://10.199.0.1</a>>: ICMP<div><br>
echo request, id 38145, seq 1, length 64<br>
14:35:00.897569 fa:16:3e:36:8e:f2 > fa:16:3e:99:85:5d, ethertype IPv4<br></div>
(0x0800), length 98: 10.199.0.4 > 10.199.0.1 <<a href="http://10.199.0.1" target="_blank">http://10.199.0.1</a>>: ICMP<div><br>
echo request, id 38145, seq 2, length 64<br>
14:35:01.260201 fa:16:3e:99:85:5d > fa:16:3e:36:8e:f2, ethertype IPv4<br></div>
(0x0800), length 98: 10.199.0.1 > 10.199.0.4 <<a href="http://10.199.0.4" target="_blank">http://10.199.0.4</a>>: ICMP<div><br>
echo reply, id 38145, seq 1, length 64<br>
14:35:01.260229 fa:16:3e:99:85:5d > fa:16:3e:36:8e:f2, ethertype IPv4<br></div>
(0x0800), length 98: 10.199.0.1 > 10.199.0.4 <<a href="http://10.199.0.4" target="_blank">http://10.199.0.4</a>>: ICMP<div><br>
echo reply, id 38145, seq 2, length 64<br>
<br>
So the router sees a sub-millisecond delay, while the VM sees a<br>
significant delay (almost a second). This only happens during the first<br>
packet, and then responses are sub 1ms.<br>
<br>
It appears to be an issue with the router, as delays are seem with both<br>
internal and external traffic on the router itself. Any thoughts are<br>
greatly appreciated!<br>
<br>
<br></div>
______________________________<u></u>_________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.<u></u>openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-operators</a><br>
<br>
</blockquote>
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.<u></u>openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-operators</a><br>
</blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>