Open Stack

Hello everybody,

after a lot of investigations, I finally get rid of that Kernel skb_warn_bad_offload crash. To do so you want to disable completely GSO and even CSUM from the virtio_net module by adding [1] into your preferred modprobe.d file.

You probably want to know that ethtool -K ethernet_int tso off gso off whatever off isn't sufficient and doesn't override the default of virtio_net module, which is GSO and CSUM on.

Hope it helps someone, somewhere :)

See you,
Regards

[1] options virtio_net gso=0 and csum=0
May 20 2016 11:06 AM, "Leslie-Alexandre DENIS"  wrote:

Hello,

pretty hard problem today for me, as I need to build a Docker platform test inside OpenStack with CentOS 7.2 all around and apparently the Ethernet offloading features don't like virtio_net at all !
Not even one HTTP packet is going in/out through Docker bridges and host's NIC, the Kernel stack trace refers to WARNING: at net/core/dev.c:2263 skb_warn_bad_offload+0xcd/0xda().
I detect some bad TCP checksum in the VM for HTTP traffic but it must be related to offloading, which is normally disabled...
On the contrary a second test with FTP traffic seems to work.

I disabled all the interfaces offloading features via ethtool -K, more precisely rx off tx off tso off lro off gro off gso off. It does nothing regarding to the Docker problem.

The only way to run a container is to pass the --net host argument in order to not use the bridge networking, and it works.

So my question is, does anybody is using nested Docker Engine inside KVM ? Any hint ?

*facts*

* VM net args >> -netdev tap,fd=48,id=hostnet0,vhost=on,vhostfd=52 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:c4:2a:7e,bus=pci.0,addr=0x3
* VM qemu-kvm-ev-2.3.0-31.el7_2.4.1
* VM CentOS 7.2.1511 with 3.10.0-327.18.2.el7.x86_64 Kernel
* Docker version 1.11.1, build 5604cbe
* Docker bridge options (defaults):
 "com.docker.network.bridge.default_bridge": "true",
 "com.docker.network.bridge.enable_icc": "true",
 "com.docker.network.bridge.enable_ip_masquerade": "true",
 "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
 "com.docker.network.bridge.name": "docker0",
 "com.docker.network.driver.mtu": "1500"
* Kernel TCP/IP params ok
* Offloading turned off via ethtool -K ... rx off tx off tso off lro off gro off gso off
* Kernel stack trace >

May 20 10:54:08 ccosvms0094 kernel: Call Trace:
May 20 10:54:08 ccosvms0094 kernel:  [] dump_stack+0x19/0x1b
May 20 10:54:08 ccosvms0094 kernel: [] warn_slowpath_common+0x70/0xb0
May 20 10:54:08 ccosvms0094 kernel: [] warn_slowpath_fmt+0x5c/0x80
May 20 10:54:08 ccosvms0094 kernel: [] ? ___ratelimit+0x93/0x100
May 20 10:54:08 ccosvms0094 kernel: [] skb_warn_bad_offload+0xcd/0xda
May 20 10:54:08 ccosvms0094 kernel: [] __skb_gso_segment+0x79/0xb0
May 20 10:54:08 ccosvms0094 kernel: [] validate_xmit_skb.part.86+0x135/0x2f0
May 20 10:54:08 ccosvms0094 kernel: [] dev_queue_xmit+0x4dd/0x570
May 20 10:54:08 ccosvms0094 kernel: [] ? ipv4_confirm+0x86/0x100 [nf_conntrack_ipv4]
May 20 10:54:08 ccosvms0094 kernel: [] ip_finish_output+0x53d/0x7d0
May 20 10:54:08 ccosvms0094 kernel: [] ip_output+0x6f/0xe0
May 20 10:54:08 ccosvms0094 kernel: [] ? ip_fragment+0x8b0/0x8b0
May 20 10:54:08 ccosvms0094 kernel: [] ip_forward_finish+0x66/0x80
May 20 10:54:08 ccosvms0094 kernel: [] ip_forward+0x377/0x490
May 20 10:54:08 ccosvms0094 kernel: [] ? ip_frag_mem+0x40/0x40
May 20 10:54:08 ccosvms0094 kernel: [] ip_rcv_finish+0x7d/0x350
May 20 10:54:08 ccosvms0094 kernel: [] ip_rcv+0x2b6/0x410
May 20 10:54:08 ccosvms0094 kernel: [] ? inet_del_offload+0x40/0x40
May 20 10:54:08 ccosvms0094 kernel: [] __netif_receive_skb_core+0x582/0x7d0
May 20 10:54:08 ccosvms0094 kernel: [] __netif_receive_skb+0x18/0x60
May 20 10:54:08 ccosvms0094 kernel: [] netif_receive_skb+0x40/0xc0
May 20 10:54:08 ccosvms0094 kernel: [] virtnet_poll+0x3d8/0x700 [virtio_net]
May 20 10:54:08 ccosvms0094 kernel: [] net_rx_action+0x152/0x240
May 20 10:54:08 ccosvms0094 kernel: [] __do_softirq+0xef/0x280
May 20 10:54:08 ccosvms0094 kernel: [] call_softirq+0x1c/0x30
May 20 10:54:08 ccosvms0094 kernel: [] do_softirq+0x65/0xa0
May 20 10:54:08 ccosvms0094 kernel: [] irq_exit+0x115/0x120
May 20 10:54:08 ccosvms0094 kernel: [] do_IRQ+0x58/0xf0
May 20 10:54:08 ccosvms0094 kernel: [] common_interrupt+0x6d/0x6d
May 20 10:54:08 ccosvms0094 kernel:  [] ? native_safe_halt+0x6/0x10
May 20 10:54:08 ccosvms0094 kernel: [] default_idle+0x1f/0xc0
May 20 10:54:08 ccosvms0094 kernel: [] arch_cpu_idle+0x26/0x30
May 20 10:54:08 ccosvms0094 kernel: [] cpu_startup_entry+0x245/0x290
May 20 10:54:08 ccosvms0094 kernel: [] start_secondary+0x1ba/0x230
May 20 10:54:08 ccosvms0094 kernel: ---[ end trace b2c6008796d4cce2 ]---
Thanks a lot,
Leslie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20160523/b6b33dde/attachment.html>