hmm for txqueuelen, I actually followed recommendation from here: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/..., where it suggests to do this: cat <<'EOF'>/etc/udev/rules.d/71-net-txqueuelen.rules SUBSYSTEM=="net", ACTION=="add", KERNEL=="tap*", ATTR{tx_queue_len}="10000" EOF or /sbin/ip link set tap<uuid> txqueuelen 10000 I will try to bump up both libvirt/rx_queue_size and libvirt/tx_queue_size to 1024, just not sure about the difference between the above and the corresponding libvirt one. Also it seems by default rx_queue_size and tx_queue_size would be None, which means bumping them up to 1024 should help with packet drops? On Mon, Jul 19, 2021 at 1:18 PM Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2021-07-19 at 19:16 +0100, Sean Mooney wrote:
On Mon, 2021-07-19 at 12:54 -0500, hai wu wrote:
I already ensured txqueuelen for that VM's tap interface, and enabled multi-queue for the VM, but its tap rxdrop still randomly kept increasing, dropping one packet every few 10 or 20 or 60 seconds (It makes sense to me, since that would only help with txdrops, not rxdrops per my understandingj).
multi queue should help with both tx and rx drops by the way.
when you enabel multi queue we allocate 1 rx and tx queue per vm cpu. which should allow the network backend to process more packets in parallel form the vm. if the network backend is overloaded and cannot process anymore packets tehn addimg more rx queues wont help but provided you network backend is not not the bottelneck it will.
This is an idle VM. After migrating this idle VM to another idle OpenStack hypervisor, so it is only one VM running on one dedicated physical OpenStack hypervisor, and now it is dropping 0.03 rxdrops/s every 10 minutes.
I am not aware of any way to configure rxqueuelen for tap interface.
you confirure the rx queue lenght the same way you confire tx queue lenght https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.rx_... and tx queue lenght is configred by https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.tx_... which you presumabnle already have set.
I assume tap RX -> Hypervisor RX -> VM TX, correct? How to tune rx on the taps is tx from the vm yes. rx drops normally means the packets were droped by the network backend for some reason. e.g. ovs or linux bridge is discarding packets. rxqueuelen for tap interface? If logging into this idle Linux test VM, I am not seeing any drops, either in its RX or TX.
On Mon, Jul 19, 2021 at 12:03 PM Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2021-07-19 at 11:39 -0500, hai wu wrote:
There are random very slow rxdrops for certain OpenStack VMs for their network tap interfaces. Please note that this is rxdrop, NOT txdrop.
I know we could tune txqueuelen and multi-queue for tap network interface txdrop issue, but is there any known way to tune for this tap network interface rxdrop issue? i think a rx drop typically means the vswtich/kernel is droping packets so i think any tuneing you applied would have to be on the kernel side.
with that said you can configure the rxqueue lenght and enabel multi queue will also result in addtional rx queue so it may help but no i dont know of any one fix for this you will have to see what turning work in yoru env for your given traffic profile.
Thanks, Hai