[Openstack] [ovs-discuss] Openvswitch network disconnect

Chris contact at progbau.de
Thu Jun 4 06:52:53 UTC 2015 

Hello Alex,

 

I will do the gdb debug when it happens again. 

 

Here the output, in this case there are 4 VMs running on the host. The physical interfaces eth2 & eth3 are in active-passive mode bonded to bond0. mgmt0 is used for the management of the host system. Anything else is created by OpenStack.

The failure happens independent from the number of VMs running on the host: 

 

ovs-vsctl show

fbbaf640-ed82-4735-99d2-fbe09f4041f1

    Bridge "br-bond0"

        Port "mgmt0"

            Interface "mgmt0"

                type: internal

        Port "br-bond0"

            Interface "br-bond0"

                type: internal

        Port "phy-br-bond0"

            Interface "phy-br-bond0"

        Port "bond0"

            Interface "eth2"

            Interface "eth3"

    Bridge br-int

        fail_mode: secure

        Port br-int

            Interface br-int

                type: internal

        Port "qvo4166dc0a-69"

            tag: 1

            Interface "qvo4166dc0a-69"

        Port "qvo6d8b70de-9c"

            tag: 1

            Interface "qvo6d8b70de-9c"

        Port "qvoe75237cc-7f"

            tag: 1

            Interface "qvoe75237cc-7f"

        Port "qvoe3b9e1fc-a5"

            tag: 1

            Interface "qvoe3b9e1fc-a5"

        Port "int-br-bond0"

            Interface "int-br-bond0"

    ovs_version: "2.3.0"

 

ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000

    link/ether 9c:b6:54:b3:67:34 brd ff:ff:ff:ff:ff:ff

3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000

    link/ether 9c:b6:54:b3:67:35 brd ff:ff:ff:ff:ff:ff

4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

    link/ether 64:51:06:f0:85:98 brd ff:ff:ff:ff:ff:ff

5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

    link/ether 64:51:06:f0:85:9c brd ff:ff:ff:ff:ff:ff

6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN

    link/ether de:76:ca:6d:57:48 brd ff:ff:ff:ff:ff:ff

7: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN

    link/ether ee:f8:2f:ef:94:42 brd ff:ff:ff:ff:ff:ff

8: br-bond0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN

    link/ether 64:51:06:f0:85:98 brd ff:ff:ff:ff:ff:ff

15: mgmt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN

    link/ether fa:e1:51:33:5d:dc brd ff:ff:ff:ff:ff:ff

    inet 10.201.195.75/24 brd 10.201.195.255 scope global mgmt0

16: qbr4166dc0a-69: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN

    link/ether be:9a:00:23:b1:f3 brd ff:ff:ff:ff:ff:ff

17: qvo4166dc0a-69: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether ae:27:12:55:09:0c brd ff:ff:ff:ff:ff:ff

18: qvb4166dc0a-69: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether be:9a:00:23:b1:f3 brd ff:ff:ff:ff:ff:ff

19: tap4166dc0a-69: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500

    link/ether fe:16:3e:d6:34:82 brd ff:ff:ff:ff:ff:ff

20: qbre75237cc-7f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN

    link/ether c2:f2:1a:30:a2:84 brd ff:ff:ff:ff:ff:ff

21: qvoe75237cc-7f: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 1e:7f:f4:57:aa:82 brd ff:ff:ff:ff:ff:ff

22: qvbe75237cc-7f: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether c2:f2:1a:30:a2:84 brd ff:ff:ff:ff:ff:ff

23: tape75237cc-7f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500

    link/ether fe:16:3e:02:a4:d7 brd ff:ff:ff:ff:ff:ff

24: qbre3b9e1fc-a5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN

    link/ether ce:c7:ec:59:fd:1a brd ff:ff:ff:ff:ff:ff

25: qvoe3b9e1fc-a5: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether c6:5f:19:25:7c:be brd ff:ff:ff:ff:ff:ff

26: qvbe3b9e1fc-a5: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether ce:c7:ec:59:fd:1a brd ff:ff:ff:ff:ff:ff

27: tape3b9e1fc-a5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500

    link/ether fe:16:3e:bb:a7:19 brd ff:ff:ff:ff:ff:ff

28: qbr6d8b70de-9c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN

    link/ether c2:45:27:31:f8:d1 brd ff:ff:ff:ff:ff:ff

29: qvo6d8b70de-9c: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 46:23:9e:4b:2a:fd brd ff:ff:ff:ff:ff:ff

30: qvb6d8b70de-9c: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether c2:45:27:31:f8:d1 brd ff:ff:ff:ff:ff:ff

31: tap6d8b70de-9c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500

    link/ether fe:16:3e:40:9e:44 brd ff:ff:ff:ff:ff:ff

32: bond0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN

    link/ether f6:83:30:bc:4c:7c brd ff:ff:ff:ff:ff:ff

35: phy-br-bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 76:51:9d:bd:24:83 brd ff:ff:ff:ff:ff:ff

36: int-br-bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 5e:fa:05:2a:1a:ef brd ff:ff:ff:ff:ff:ff

 

 

Cheers,

Chris

 

 

From: Alex Wang [mailto:alexw at nicira.com] 
Sent: Thursday, June 04, 2015 13:29
To: Chris
Cc: discuss at openvswitch.org; openstack at lists.openstack.org; Soputhi Sea
Subject: Re: [ovs-discuss] Openvswitch network disconnect

 

 

 

On Wed, Jun 3, 2015 at 11:16 PM, Chris <contact at progbau.de <mailto:contact at progbau.de> > wrote:

Hello,

We are using Openvswitch in our Openstack setup.

ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.3.0
Compiled Oct 28 2014 17:48:05
OpenFlow versions 0x1:0x1

We experience openvswitch failures from time to time. It seems to happen
random, no network traffic spikes for example.
The VM ports and the openvswitch port for the host management just stop
working.  But the openvswitch services (ovsdb-server/ ovs-vswitchd) are
still running

For debug purpose the following commands has been executed:

- "ovs-vsctl show" the interfaces are still listed
The ones below didn't show any result and just hang after the execution:
- ovs-appctl bond/show bond0
- ovs-appctl vlog/list
- ovs-ofctl dump-flows br-bond0

 

 

This seems to indicate that the ovs-vswitchd process is dead locked...

I could not find comment in branch-2.3 that relates to dead lock.  It would be

helpful if you can gdb into the running ovs-vswitchd process and provide

the backtraces when there is a failure.

 

Or could I know what your setup looks like (ovs-vsctl show output)?

 

Thanks,

Alex Wang,

 

 

 

A "service openvswitch restart" fix it, the connection from the VMs and the
host are back immediately.

Any help appreciated!

Cheers,
Chris



_______________________________________________
discuss mailing list
discuss at openvswitch.org <mailto:discuss at openvswitch.org> 
http://openvswitch.org/mailman/listinfo/discuss

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150604/f39c389e/attachment.html>

More information about the Openstack mailing list