[Openstack] Poor instance network performance
Satish Patel
satish.txt at gmail.com
Thu Dec 17 20:08:52 UTC 2015
Its was hypervisor KVM (host)
On Thu, Dec 17, 2015 at 2:52 PM, Tim Bell <Tim.Bell at cern.ch> wrote:
> Was that the VM or the hypervisor ?
>
> These tips are good to include the upcoming hypervisor tuning guide (https://wiki.openstack.org/wiki/Documentation/HypervisorTuningGuide)
>
> Tim
>
>> -----Original Message-----
>> From: Satish Patel [mailto:satish.txt at gmail.com]
>> Sent: 17 December 2015 18:55
>> To: Vahric Muhtaryan <vahric at doruk.net.tr>
>> Cc: openstack at lists.openstack.org
>> Subject: Re: [Openstack] Poor instance network performance
>>
>> I think i found solution, it improve little bit performance
>>
>> on tapdd55b834-f8 interface default txqueuelen size was 500, I have
>> changed it to 10000
>>
>> On Thu, Dec 17, 2015 at 12:07 PM, Satish Patel <satish.txt at gmail.com>
>> wrote:
>> > On host (compute) machine tap device showing lots of TX packet loss.
>> >
>> > tapdd55b834-f8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu
>> 1500
>> > inet6 fe80::fc16:3eff:fecd:2aa4 prefixlen 64 scopeid 0x20<link>
>> > ether fe:16:3e:cd:2a:a4 txqueuelen 500 (Ethernet)
>> > RX packets 1060731031 bytes 298573599135 (278.0 GiB)
>> > RX errors 0 dropped 0 overruns 0 frame 0
>> > TX packets 503372157 bytes 81345389820 (75.7 GiB)
>> > TX errors 0 dropped 171333348 overruns 0 carrier 0
>> > collisions 0
>> >
>> > On Thu, Dec 17, 2015 at 11:45 AM, Satish Patel <satish.txt at gmail.com>
>> wrote:
>> >> I just ran iperf and here is the results, bi-directional
>> >>
>> >> [ ID] Interval Transfer Bandwidth
>> >> [ 5] 0.0-10.0 sec 1.04 GBytes 895 Mbits/sec
>> >> [ 4] 0.0-10.0 sec 1.09 GBytes 935 Mbits/sec
>> >>
>> >> Look like when guest is under load, it can't handle traffic and start
>> >> dropping packets.
>> >>
>> >>
>> >>
>> >> On Thu, Dec 17, 2015 at 11:36 AM, Vahric Muhtaryan
>> <vahric at doruk.net.tr> wrote:
>> >>> Maybe this also could be reference
>> >>>
>> >>> http://42.62.73.30/wordpress/wp-content/uploads/2013/10/Neutron-
>> perf
>> >>> ormance
>> >>> -testing.pdf
>> >>>
>> >>>
>> >>> On 17/12/15 18:32, "Satish Patel" <satish.txt at gmail.com> wrote:
>> >>>
>> >>>>We have our own custom application running on guest VM. i am just
>> >>>>checking performance of VM. but i found if load go little up machine
>> >>>>start breaking ping.
>> >>>>
>> >>>>I didn't run iperf test yet. which i am going to run in a min.
>> >>>>
>> >>>>On Thu, Dec 17, 2015 at 11:28 AM, Vahric Muhtaryan
>> >>>><vahric at doruk.net.tr>
>> >>>>wrote:
>> >>>>> Okay Satish ,
>> >>>>>
>> >>>>> Im not expert and do not want to divert you wrong direction.
>> >>>>> I want to make your test on my server to cross check , can you pls
>> >>>>>explain how and with what tool and to where you are making this
>> >>>>>test.
>> >>>>>
>> >>>>> Regards
>> >>>>> VM
>> >>>>>
>> >>>>> On 17/12/15 18:11, "Satish Patel" <satish.txt at gmail.com> wrote:
>> >>>>>
>> >>>>>>Thanks for reply,
>> >>>>>>
>> >>>>>>Following is my guest xml file.
>> >>>>>>
>> >>>>>>I am using Openstack JUNO and it use OVS.
>> >>>>>>
>> >>>>>><interface type='bridge'>
>> >>>>>> <mac address='fa:16:3e:cd:2a:a4'/>
>> >>>>>> <source bridge='qbrdd55b834-f8'/>
>> >>>>>> <target dev='tapdd55b834-f8'/>
>> >>>>>> <model type='virtio'/>
>> >>>>>> <alias name='net0'/>
>> >>>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>> >>>>>>function='0x0'/>
>> >>>>>> </interface>
>> >>>>>>
>> >>>>>>
>> >>>>>>Following is TOP command on guest machine, at this point i am
>> >>>>>>getting ping breaks
>> >>>>>>
>> >>>>>>top - 16:10:30 up 20:46, 1 user, load average: 4.63, 4.41, 3.60
>> >>>>>>Tasks: 165 total, 6 running, 159 sleeping, 0 stopped, 0 zombie
>> >>>>>>%Cpu0 : 15.1 us, 12.3 sy, 0.0 ni, 60.9 id, 0.0 wa, 0.0 hi,
>> >>>>>>0.3 si,
>> >>>>>>11.4 st
>> >>>>>>%Cpu1 : 22.9 us, 17.2 sy, 0.0 ni, 51.4 id, 0.0 wa, 0.0 hi,
>> >>>>>>0.3 si,
>> >>>>>>8.2 st
>> >>>>>>%Cpu2 : 28.8 us, 22.4 sy, 0.0 ni, 47.5 id, 0.0 wa, 0.0 hi,
>> >>>>>>1.0 si,
>> >>>>>>0.3 st
>> >>>>>>%Cpu3 : 16.6 us, 15.0 sy, 0.0 ni, 66.4 id, 0.0 wa, 0.0 hi,
>> >>>>>>0.3 si,
>> >>>>>>1.7 st
>> >>>>>>%Cpu4 : 9.8 us, 11.8 sy, 0.0 ni, 0.0 id, 75.4 wa, 0.0 hi,
>> >>>>>>0.3 si,
>> >>>>>>2.6 st
>> >>>>>>%Cpu5 : 7.6 us, 6.1 sy, 0.0 ni, 81.4 id, 0.0 wa, 0.0 hi,
>> >>>>>>4.2 si,
>> >>>>>>0.8 st
>> >>>>>>%Cpu6 : 8.1 us, 7.4 sy, 0.0 ni, 83.0 id, 0.0 wa, 0.0 hi,
>> >>>>>>1.4 si,
>> >>>>>>0.0 st
>> >>>>>>%Cpu7 : 17.8 us, 17.8 sy, 0.0 ni, 64.1 id, 0.0 wa, 0.0 hi,
>> >>>>>>0.3 si,
>> >>>>>>0.0 st
>> >>>>>>KiB Mem : 8175332 total, 4630124 free, 653284 used, 2891924
>> >>>>>>buff/cache
>> >>>>>>KiB Swap: 0 total, 0 free, 0 used. 7131540 avail
>> >>>>>>Mem
>> >>>>>>
>> >>>>>>
>> >>>>>>On Thu, Dec 17, 2015 at 11:06 AM, Vahric Muhtaryan
>> >>>>>><vahric at doruk.net.tr>
>> >>>>>>wrote:
>> >>>>>>> Hello ,
>> >>>>>>>
>> >>>>>>> Cool ! Todays I am testing to but not such way just only KVM.
>> >>>>>>>
>> >>>>>>> Wonder 2/3 things ;
>> >>>>>>>
>> >>>>>>> 1) Are you using virtio?
>> >>>>>>> <model type='virtio¹/>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> 2) is your vm saturating %100 of cpu inside guest ?
>> >>>>>>>
>> >>>>>>> 3) Using OVS or bridge ?
>> >>>>>>>
>> >>>>>>> Regards
>> >>>>>>> Vahric Muhtaryan
>> >>>>>>>
>> >>>>>>> On 17/12/15 17:41, "Satish Patel" <satish.txt at gmail.com> wrote:
>> >>>>>>>
>> >>>>>>>>I am doing some testing on Openstack VM network performance.
>> >>>>>>>>following i am doing
>> >>>>>>>>
>> >>>>>>>>Comute node: 8 CPU / 32GB mem
>> >>>>>>>>VM instance: 8vCPU / 8GB mem
>> >>>>>>>>
>> >>>>>>>>when i am running heavy load application, immediate my ping
>> >>>>>>>>started breaking for VM.. but at same time if i ping Compute
>> >>>>>>>>Host, its working fine without any packet loss.
>> >>>>>>>>
>> >>>>>>>>I have disabled GSO/TSO setting on eth0 but still same result.
>> >>>>>>>>i have notice when it reach 250mbps my ping start breaking...
>> >>>>>>>>
>> >>>>>>>>_______________________________________________
>> >>>>>>>>Mailing list:
>> >>>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> >>>>>>>>Post to : openstack at lists.openstack.org
>> >>>>>>>>Unsubscribe :
>> >>>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> >>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>>
>> >>>
>> >>>
>>
>> _______________________________________________
>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack at lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
More information about the Openstack
mailing list