Open Stack

Hi Thiago,

It would be interesting to see the throughput results using something like
iperf between (other vms on different Hypervisors,  vms on the same
hypervisor and between a vm and the router namespace). This should help in
pin pointing the issue.

Aaron


On Tue, Oct 22, 2013 at 9:17 PM, Martinx - ジェームズ
<thiagocmartinsc at gmail.com>wrote:

> Sure, always... For example, from within a tenant instance:
>
> ------
> ubuntu at instance-1:~$ ip route
> default via 192.168.210.1 dev eth0  metric 100
> 192.168.210.0/24 dev eth0  proto kernel  scope link  src 192.168.210.2
>
> # 192.168.210.1 resides within the Tenant Namespace at the Network Node,
> ping ok:
> ubuntu at instance-1-1:~$ ping -c 1 192.168.210.1
>
>
> PING 192.168.210.1 (192.168.210.1) 56(84) bytes of data.
> 64 bytes from 192.168.210.1: icmp_req=1 ttl=64 time=0.898 ms
>
> --- 192.168.210.1 ping statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> rtt min/avg/max/mdev = 0.898/0.898/0.898/0.000 ms
>
> # Internet...
> ubuntu at instance-1:~$ ping -c 1 8.8.8.8
>
>
> PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
> 64 bytes from 8.8.8.8: icmp_req=1 ttl=48 time=133 ms
>
> --- 8.8.8.8 ping statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> rtt min/avg/max/mdev = 133.232/133.232/133.232/0.000 ms
>
> # Very slow speed:
> ubuntu at instance-1:~$ sudo aptitude update
> Get: 1 http://nova.clouds.archive.ubuntu.com precise Release.gpg [198 B]
> Get: 2 http://security.ubuntu.com precise-security Release.gpg [198 B]
> Get: 3 http://nova.clouds.archive.ubuntu.com precise-updates Release.gpg
> [198 B]
> Get: 4 http://security.ubuntu.com precise-security Release [49.6 kB]
> Get: 5 http://nova.clouds.archive.ubuntu.com precise-backports
> Release.gpg [198 B]
> Hit http://nova.clouds.archive.ubuntu.com precise Release
> Get: 6 http://nova.clouds.archive.ubuntu.com precise-updates Release
> [49.6 kB]
> 81% [6 Release 44.5 kB/49.6 kB 90%] [4 Release 35.4 kB/49.6 kB 71%]
>
>
>  *7,521 B/s 2s*
> ------
>
> 7,521 B/s ???
>
> If I run "aptitude update" from within Tenant's Namespace, the "router"
> with the IP 192.168.210.1 (tenant's gateway), it goes just fine. Look:
>
> ------
> root at net-node-1:~# ip netns exec
> qrouter-46cb8f7a-a3c5-4da7-ad69-4de63f7c34f1 ip r
> default via 172.16.0.1 dev qg-50b615b7-c2
> 172.16.0.0/20 dev qg-50b615b7-c2  proto kernel  scope link  src
> 172.16.0.2
> 192.168.210.0/24 dev qr-a1376f61-05  proto kernel  scope link  src
> 192.168.210.1
>
> # Normal speed
> root at net-node-1:~# ip netns exec
> qrouter-46cb8f7a-a3c5-4da7-ad69-4de63f7c34f1 aptitude update
> Hit http://us.archive.ubuntu.com precise Release.gpg
>
> Hit http://us.archive.ubuntu.com precise-updates Release.gpg
> Get: 1 http://us.archive.ubuntu.com precise-backports Release.gpg [198 B]
> Get: 2 http://ubuntu-cloud.archive.canonical.com precise-updates/havana
> Release.gpg [543 B]
> Get: 3 http://security.ubuntu.com precise-security Release.gpg [198 B]
> ....sniped....
> ------
>
> No idea about what's happening...    ;-(
>
> Also, the Compute Node have normal speed too, where the instance-1 is
> running.
>
> This Havana I'm trying to put into production, follow a well tested
> topology that I already have up and running, very similar with this:
> https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst
>
> I'll (double * double) check everything again. I like to test and debug...
>     :-D
>
> Best,
> Thiago
>
>
> On 23 October 2013 01:52, 止语 <menghuizhanguo at gmail.com> wrote:
>
>> Have you ever try to check the route ?
>> #route -n
>>
>>
>> ------------------ Original ------------------
>> *From: * "Martinx - ジェームズ";<thiagocmartinsc at gmail.com>;
>> *Date: * Wed, Oct 23, 2013 11:34 AM
>> *To: * "Geraint Jones"<geraint at koding.com>; **
>> *Cc: * "openstack at lists.openstack.org"<openstack at lists.openstack.org>; **
>> *Subject: * Re: [Openstack] Very slow connectivity from within tenant
>> network -GRE
>>
>> WOW! Nice move!
>>
>> But, upgrading from `openvswitch 1.10.2-0ubuntu2~cloud0` to `1.11.0-1`
>> did not solved my issue.
>>
>> Tenant Instances still have a very slow Internet connectivity.
>>
>> Thanks anyway! Nice to see your charts, pretty good improvement!
>>
>> Regards,
>> Thiago
>>
>>
>> On 22 October 2013 22:48, Geraint Jones <geraint at koding.com> wrote:
>>
>>> I have just had to tweak our grizzly network node the biggest impacts
>>> were seem from doing changing root wrap to only use sudo – not the python
>>> wrapper (its super slow) and upgrading openvswitch to 1.11
>>>
>>> This smoke ping shows the latency to one of our instances from europe
>>> before and after the openvswitch upgrade : http://d.pr/i/36v0
>>>
>>> And this graph shows the load avg on our network node, the first drop is
>>> from disabling root wrap the second is after the OVS upgrade :
>>> http://d.pr/i/xhFc
>>>
>>> I would suggest you do the same, and just make sure all MTU’s are
>>> correct.
>>>
>>> --
>>> Geraint Jones
>>> Director of Systems & Infrastructure
>>> Koding
>>> https://koding.com
>>> geraint at koding.com
>>> M (NZ) +64 22 123 4626
>>> M (US) +1 415 316 8027
>>>
>>> From: Martinx - ジェームズ <thiagocmartinsc at gmail.com>
>>> Date: Tuesday, 22 October 2013 9:00 am
>>> To: Rick Jones <rick.jones2 at hp.com>
>>> Cc: "openstack at lists.openstack.org" <openstack at lists.openstack.org>
>>> Subject: Re: [Openstack] Very slow connectivity from within tenant
>>> network - GRE
>>>
>>> Hi Rick!
>>>
>>> Back with Grizzly, I faced that problem and I was able to detect it, at
>>> the Network Node with tcpdump and fix it by running "ip link set mtu 1454
>>> dev eth0" within the Instance.
>>>
>>> Not this time... This is another problem...   ;-/
>>>
>>>
>>> On 22 October 2013 13:25, Rick Jones <rick.jones2 at hp.com> wrote:
>>>
>>>> On 10/22/2013 01:32 AM, Martinx - ジェームズ wrote:
>>>>
>>>>> Stackers,
>>>>>
>>>>> I'm trying to put my Havana into production and I'm facing a very
>>>>> strange problem.
>>>>>
>>>>> The Internet connectivity from tenant's subnet is very, very slow. It
>>>>> is
>>>>> useless in fact... I can not even use "apt-get update" from a Instance.
>>>>>
>>>>> The following command works (apt update from the tenant namespace):
>>>>>
>>>>> ---
>>>>> root at net-node-1:~# ip netns exec qrouter-XXXXXXXXX aptitude update
>>>>> ---
>>>>>
>>>>> But not from the tenant subnet...
>>>>>
>>>>> I'm following this topology:
>>>>>
>>>>> http://docs.openstack.org/**trunk/install-guide/install/**
>>>>> apt/content/section_use-cases-**tenant-router.html<http://docs.openstack.org/trunk/install-guide/install/apt/content/section_use-cases-tenant-router.html>
>>>>>
>>>>> Already tried to change MTUs (via DHCP agent)... Nothing had fixed this
>>>>> weird issue.
>>>>>
>>>>> Any thoughts?!
>>>>>
>>>>> Right now, my "aptitude safe-upgrade" will take 2 days to download
>>>>> 60MB... During this network outages, even the SSH session stops
>>>>> responding for a few seconds...
>>>>>
>>>>> Everything else seems to be working as expected, as for example, DHCP,
>>>>> Floating IPs, Security Groups...
>>>>>
>>>>> Sometimes, even the first ssh connection to the Instance Floating IP,
>>>>> have a lag.
>>>>>
>>>>
>>>> It is but a guess, but I wonder if, even with changing MTUs (to what
>>>> values?) you may still be experiencing a PathMTU+ICMP blackhole problem
>>>> accessing nodes on the Internet.  Can you access something that is a bit
>>>> "closer" but still outside your stack so you have a shot at looking at
>>>> netstat statistics on the sender and/or get packet traces on the sender?
>>>>
>>>> You could still try taking packet traces at the instance or perhaps the
>>>> namespace and try to discern packet losses at the receiving side, though it
>>>> can be a bit more difficult.
>>>>
>>>> rick jones
>>>>
>>>>
>>> _______________________________________________ Mailing list:
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to :
>>> openstack at lists.openstack.org Unsubscribe :
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>
>>
>>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131022/2d6ea5fc/attachment.html>