Further update: I dusted off my gdb skills and attached it to the privsep process to try to get more details about exactly what is crashing. It looks like the segfault happens on this line: https://git.netfilter.org/libnetfilter_conntrack/tree/src/conntrack/api.c#n2... which is h->cb = cb; h being the conntrack handle and cb being the callback function. This makes me think the problem isn't the callback itself (even if we assigned a bogus pointer, which we didn't, it shouldn't cause a segfault unless you try to dereference it) but in the handle we pass in. Trying to look at h->cb results in: (gdb) print h->cb Cannot access memory at address 0x800f228 Interestingly, h itself is fine: (gdb) print h $3 = (struct nfct_handle *) 0x800f1e0 It doesn't _look_ to me like the handle should be crossing any thread boundaries or anything, so I'm not sure why it would be a problem. It gets created in the same privileged function that ultimately registers the callback: https://github.com/openstack/neutron/blob/aa8a6ea848aae6882abb631b7089836dee... So still not sure what's going on, but I thought I'd share what I've found before I stop to eat something. -Ben On 1/7/19 12:11 PM, Ben Nemec wrote:
Renamed the thread to be more descriptive.
Just to update the list on this, it looks like the problem is a segfault when the netlink_lib module makes a C call. Digging into that code a bit, it appears there is a callback being used[1]. I've seen some comments that when you use a callback with a Python thread, the thread needs to be registered somehow, but this is all uncharted territory for me. Suggestions gratefully accepted. :-)
1: https://github.com/openstack/neutron/blob/master/neutron/privileged/agent/li...
On 1/4/19 7:28 AM, Slawomir Kaplonski wrote:
Hi,
I just found that functional tests in Neutron are failing since today or maybe yesterday. See [1] I was able to reproduce it locally and it looks that it happens with oslo.privsep==1.31. With oslo.privsep==1.30.1 tests are fine.
[1] https://bugs.launchpad.net/neutron/+bug/1810518
— Slawek Kaplonski Senior software engineer Red Hat