[Openstack-operators] Network get unstable, put the whole system on halt

Andrew Glen-Young andrew.glen-young at canonical.com
Wed May 22 15:22:40 UTC 2013


Hello,

On Mon 20-05-2013 11:57 ›, Salman Toor wrote:
> Hi, 
> 
> We are working with Grizzly together with openvswitch for quantum. Following are the details of our system.. 
> 
> Controller and Compute nodes are running with Ubuntu 12.04.5, kernel 3.5 and OpenVSwitch version 1.4.0+build0 with GRE tunnels.  

Is the version of OpenVSwitch that you provided correct? When I checked
Precise the version available seems to be 1.4.0-1ubuntu1.5.

Do you require Linux 3.5 for driver compatibility reasons or can you
test with Linux 3.2 to see if you have the same problems?

> 
> The problem is with very little activity everything works very fine but as we started to increase the load on the system the kernel log started to grow on the controller and fill the entire disk space and halt the complete system. And it happen within 2 to 3 hours ... 
> 

Can you quantify what you mean by "increase the load on the system"? For
example: Does this occur when the machine becomes loaded or when the
network throughput increases?

> Most of the log is filled with the following messages 
> 
> 
> May 20 06:31:04 ukko233-cern-controller kernel: [67607.598519] Call Trace:
> May 20 06:31:04 ukko233-cern-controller kernel: [67607.598519]  <IRQ> [<ffffffff81052c9f>] warn_slowpath_common+0x7f/0xc0
> May 20 06:31:04 ukko233-cern-controller kernel: [67607.598524]  [<ffffffff81052d96>] warn_slowpath_fmt+0x46/0x50
> [...]
> May 20 06:31:04 ukko233-cern-controller kernel: [67607.598579]  [<ffffffffa03d802e>] ovs_dp_process_received_packet+0x6e/0x150 [openvswitch]
> May 20 06:31:04 ukko233-cern-controller kernel: [67607.598589]  [<ffffffffa03de4ff>] ovs_vport_receive+0x5f/0x70 [openvswitch]
> May 20 06:31:04 ukko233-cern-controller kernel: [67607.598595]  [<ffffffffa03e0e07>] patch_send+0x27/0x50 [openvswitch]
> May 20 06:31:04 ukko233-cern-controller kernel: [67607.598599]  [<ffffffffa03de52e>] ovs_vport_send+0x1e/0x50 [openvswitch]
> [...]
> 
> We are wondering does anybody else have experience this?
> 
> Or any hint which can help us to fix this problem. 
> 

Please file a kernel bug as outlined on the Ubuntu Wiki[1]? The more
information you can provide the better.

Even if this is a problem with OVS specifically and not the kernel it
will get this problem on 

[1]: https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies

Hope this helps.

-- 
Regards.
Andrew Glen-Young



More information about the OpenStack-operators mailing list