<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
<div><span></span></div>
<div>It is still possible to update rx and tx queues length if your qemu and libvirt version is higher than the version recorded in [3]. （You should possible to update directly in libvirt configuration if my memory is correct）
<div>
<div><br>
</div>
<div>We also have some similar use case which run audio/vedio serivcs. They are CPU consuming and have UDP small packets. Another possible tunning is using CPU pin for the vm. you can use numa awared cpu feature to get stable cpu performance ，vm network dropped
packets sometimes because of the vm cpu is too busy，with numa cpu it works better performance，our way is similar with [a]. You need to create flavor with special metadata and dedicated Host Agg for numa awared VMs. Dedicated CPU is very good for media service.
It makes the CPU performance stable.</div>
<div><br>
</div>
<div>Another packet loss case we get is because of vm kernel, some of our app are using 32bit OS, that cause memory issue, when traffic larger then 50kpps, it dropped a lot,sometimes,it even crash. In this case, 32bit os can actually use very limited memory,
we have to add swap for the vm. Hope your app is using 64 bit OS. Because 32 bit could cause tons of trouble.</div>
<div><br>
</div>
<div>BTW,if you are using vrouter on L3, you’d better to move provider network(no vrouter). I did not tried DVR, but if you are running without DVR, the L3 node will be bottleneck very quick. Especially default iptables conntrack is 65535, you will reach to
it and drop packet on L3, even after you tun that value, it still hard to more that 1Mpps for your network node.</div>
<div><br>
</div>
<div>If your App more than 200kpps per compute node, you may be better also have a look your physical network driver tx/rx configuration. Most of the HW default value for tx/rx queues number and length are very poor,you may start to get packet on eth interface
on physical host when rx queue is full. </div>
<div><br>
</div>
<div>[a]<a href="https://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/">https://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/</a></div>
<div><br>
</div>
<div>Regards，</div>
<div>Liping Mao<br>
<div><br>
在 2018年9月16日，21:18，Satish Patel <<a href="mailto:satish.txt@gmail.com">satish.txt@gmail.com</a>> 写道：<br>
<br>
</div>
<blockquote type="cite">
<div><span>Hi Liping,</span><br>
<span></span><br>
<span>Thank you for your reply,</span><br>
<span></span><br>
<span>We notice packet drops during high load, I did try txqueue and didn't help so I believe I am going to try miltiqueue.</span><br>
<span></span><br>
<span>For SRIOV I have to look if I have support in my nic. </span><br>
<span></span><br>
<span>We are using queens so I think queue size option not possible :(</span><br>
<span></span><br>
<span>We are using voip application and traffic is udp so our pps rate is 60k to 80k per vm instance.
</span><br>
<span></span><br>
<span>I will share my result as soon as I try multiqueue. </span><br>
<span></span><br>
<span></span><br>
<span></span><br>
<span>Sent from my iPhone</span><br>
<span></span><br>
<blockquote type="cite"><span>On Sep 16, 2018, at 2:27 AM, Liping Mao (limao) <<a href="mailto:limao@cisco.com">limao@cisco.com</a>> wrote:</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span>Hi Satish,</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span>Did your packet loss happen always or it only happened when heavy load?</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span>AFAIK, if you do not tun anything, the vm tap can process about 50kpps before the tap device start to drop packets.
</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span>If it happened in heavy load, couple of things you can try:</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span>1) increase tap queue length, usually the default value is 500, you can try larger. (seems like you already tried)</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span>2) Try to use virtio multi queues feature , see [1]. Virtio use one queue for rx/tx in vm, with this feature you can get more queues. You can check
</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span>3) In rock version, you can use [2] to increase virtio queue size, the default queues size is 256/512, you may increase it to 1024, this would help to increase pps of the tap device.</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span>If all these things can not get your network performance requirement, you may need to move to use dpdk / sriov stuff to get more vm performance.</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span>I did not actually used them in our env, you may refer to [3]</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span>[1] <a href="https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-virtiomq.html">
https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-virtiomq.html</a></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span>[2] <a href="https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/libvirt-virtio-set-queue-sizes.html">
https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/libvirt-virtio-set-queue-sizes.html</a></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span>[3] <a href="https://docs.openstack.org/ocata/networking-guide/config-sriov.html">
https://docs.openstack.org/ocata/networking-guide/config-sriov.html</a></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span>Regards,</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span>Liping Mao</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span>在 2018/9/16 13:07，“Satish Patel”<<a href="mailto:satish.txt@gmail.com">satish.txt@gmail.com</a>> 写入:</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span> [root@compute-33 ~]# ifconfig tap5af7f525-5f | grep -i drop</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span> RX errors 0 dropped 0 overruns 0 frame 0</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span> TX errors 0 dropped 2528788837 overruns 0 carrier 0 collisions 0</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span> Noticed tap interface dropping TX packets and even after increasing</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span> txqueue from 1000 to 10000 nothing changed, still getting packet</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span> drops.</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span> On Sat, Sep 15, 2018 at 4:22 PM Satish Patel <<a href="mailto:satish.txt@gmail.com">satish.txt@gmail.com</a>> wrote:</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span></span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span></span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>Folks,</span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span></span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>I need some advice or suggestion to find out what is going on with my</span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>network, we have notice high packet loss on openstack instance and not</span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>sure what is going on, same time if i check on host machine and it has</span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>zero packet loss.. this is what i did for test...</span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span></span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>ping 8.8.8.8</span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span></span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>from instance: 50% packet loss</span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>from compute host: 0% packet loss</span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span></span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>I have disabled TSO/GSO/SG setting on physical compute node but still</span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>getting packet loss.</span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span></span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>We have 10G NIC on our network, look like something related to tap</span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>interface setting..</span><br>
</blockquote>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span> _______________________________________________</span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span> Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span> Post to : <a href="mailto:openstack@lists.openstack.org">
openstack@lists.openstack.org</a></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span> Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
<blockquote type="cite"><span></span><br>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>