Open Stack

* Salvatore Orlando (sorlando at nicira.com) wrote:
> I've found out that several jobs are exhibiting failures like bug 1254890
> [1] and bug 1253896 [2] because openvswitch seem to be crashing the kernel.
> The kernel trace reports as offending process usually either
> neutron-ns-metadata-proxy or dnsmasq, but [3] seem to clearly point to
> ovs-vsctl.

Hmm, that actually shows dnsmasq is the running/exiting process.
The ovs-vsctl was run nearly a half-second earlier.  Looks like
ovs-vsctl successfuly added the tap device (assuming it's for
dnsmasq?).  And dnsmasq is exiting upon receiving a signal.  Shot in
the dark, has the neutron path that would end up killing dnsmasq
(Dnsmasq::reload_allocations()) changed recently?  I didn't see much.

> 254 events observed in the previous 6 days show a similar trace in the logs
> [4].

That kernel (3.2.0) is over a year old.  And there have been some network
namespace fixes since then (IIRC, refcounting related).

> This means that while this alone won't explain all the failures observed,
> it is however potentially one of the prominent root causes.
> 
> >From the logs I have little hints about the kernel running. It seems there
> has been no update in the past 7 days, but I can't be sure.
> Openvswitch builds are updated periodically. The last build I found not to
> trigger failures was the one generated on 2014/01/16 at 01:58:18.
> Unfortunately version-wise I always have only 1.4.0, no build number.
> 
> I don't know if this will require getting in touch with ubuntu, or if we
> can just prep a different image which an OVS build known to work without
> problems.
> 
> Salvatore
> 
> [1] https://bugs.launchpad.net/neutron/+bug/1254890
> [2] https://bugs.launchpad.net/neutron/+bug/1253896
> [3] http://paste.openstack.org/show/61869/
> [4] "kernel BUG at /build/buildd/linux-3.2.0/fs/buffer.c:2917" and
> filename:syslog.txt