Open Stack

Hi All,

We have a Mirantis Mitaka deployment and are experiencing slow internet
speed on instances that are using floating IP's.

Let me start with what's not impacted:

   - Transfers between instances on the same or different tenant networks
   - Transfers between instances on separate OpenStack deployments
   - Transfers between instances and servers outside of OpenStack
   - Instances that are on VLAN provider networks

So the situation is limited only to instances with floating IP's going out
to the internet.

Here's an example of the slow speed:

Selecting best server based on latency...
Hosted by Quonix, Inc. (Philadelphia, PA) [24.88 km]: 11.956 ms
Testing download speed........................................
Download: 11.84 Mbit/s
Testing upload speed..................................................
Upload: 3.12 Mbit/s

Here's an example of the same test from one of the controllers. The
controllers are on the same network and use the same gateway.

Hosted by Quonix, Inc. (Philadelphia, PA) [24.88 km]: 3.933 ms
Testing download
speed................................................................................
Download: 624.71 Mbit/s
Testing upload
speed....................................................................................................
Upload: 213.82 Mbit/s

Now for the more interesting part...

Here's an example running a speedtest within the namespace

ip netns e qrouter-8fd1a0e9-b1cd-4ca9-89dd-e9e4fdc82778 ./speedtest-cli
--server 7519 --source 192.168.77.1
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Quonix, Inc. (Philadelphia, PA) [24.88 km]: 4006.975 ms
Testing download
speed................................................................................
Download: 2.60 Mbit/s
Testing upload
speed....................................................................................................
Upload: 2.46 Mbit/s

Here's the same example except in this case I've changed resolv.conf to
only use our primary DNS servers instead of the resolvers that are local to
OpenStack. One of these resolvers is the fuel node and the other is dnsmasq
which is also using pointed to the fuel node in its config.

ip netns e qrouter-8fd1a0e9-b1cd-4ca9-89dd-e9e4fdc82778 ./speedtest-cli
--server 7519 --source 192.168.77.1
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Quonix, Inc. (Philadelphia, PA) [24.88 km]: 3.905 ms
Testing download
speed................................................................................
Download: 547.38 Mbit/s
Testing upload
speed....................................................................................................
Upload: 283.14 Mbit/s

Running a dig against these server within the namespace fails:

ip netns e qrouter-8fd1a0e9-b1cd-4ca9-89dd-e9e4fdc82778 /usr/bin/dig @
172.17.1.6 google.com

; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> @172.17.1.6 google.com
; (1 server found)
;; global options: +cmd
;; cRunning a dig against these server within the namespace fails:

ip netns e qrouter-8fd1a0e9-b1cd-4ca9-89dd-e9e4fdc82778 /usr/bin/dig @
172.17.1.6 google.com

; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> @172.17.1.6 google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

ip netns e qrouter-8fd1a0e9-b1cd-4ca9-89dd-e9e4fdc82778 /usr/bin/dig @
172.16.1.13 google.com

; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> @172.16.1.13 google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reachedonnection timed out; no
servers could be reached

ip netns e qrouter-8fd1a0e9-b1cd-4ca9-89dd-e9e4fdc82778 /usr/bin/dig @
172.16.1.13 google.com

; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> @172.16.1.13 google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

The instance itself is using our primary DNS servers so I'm starting to
think the speed issue within the namespace may be merely a coincidence
since name resolution within the instance appears to work fine but I'm not
sure how to troubleshoot this further. I'm considering modifying the
dnsmasq config to use our primary DNS servers but I'm doubting that would
help anything.

Any help tracking this down would be greatly appreciated!

Thank You,

___

John Petrini
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20170330/89a4c873/attachment.html>