<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Thanks Ben for digging into the detail.<div><br></div><div>I made some more tests based on your test script.</div><div>From my test result, pyroute2 and "ip" command operations against netns seems to work fine even if network namespaces of the process and thread are different.</div><div>The test script iis <a href="http://paste.openstack.org/show/742886/" target="_blank">http://paste.openstack.org/show/742886/</a> and the result is <a href="http://paste.openstack.org/show/742887/" target="_blank">http://paste.openstack.org/show/742887/</a>.<br></div><div><br></div><div>> So, to get this test passing I think we need to change [1] so it looks <br>> for the thread id and uses a replacement for [2] that allows the thread <br>> id to be injected as above.<br><br></div><div>I confirmed network namespace operations work well, so it looks safe.<br></div><div>Considering the situation, I proposed a change on the failing test to check<br></div><div>a list of network devices inside a netns.</div><div><a href="https://review.openstack.org/#/c/631654/">https://review.openstack.org/#/c/631654/</a><br></div><div><br></div><div>Thanks,</div><div>Akihiro Motoki (irc: amotoki)</div></div></div></div><br><div class="gmail_quote"><div dir="ltr">2019年1月16日(水) 7:56 Ben Nemec <<a href="mailto:openstack@nemebean.com" target="_blank">openstack@nemebean.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">TLDR: We now need to look at the thread namespace instead of the process <br>
namespace. Many, many details below.<br>
<br>
On 1/15/19 11:51 AM, Ben Nemec wrote:<br>
> <br>
> <br>
> On 1/15/19 11:16 AM, Ben Nemec wrote:<br>
>><br>
>><br>
>> On 1/15/19 6:49 AM, Doug Hellmann wrote:<br>
>>> Ben Nemec <<a href="mailto:openstack@nemebean.com" target="_blank">openstack@nemebean.com</a>> writes:<br>
>>><br>
>>>> I tried to set up a test environment for this, but I'm having some<br>
>>>> issues. My local environment is defaulting to python 3, while the gate<br>
>>>> job appears to have been running under python 2. I'm not sure why it's<br>
>>>> doing that since the tox env definition doesn't specify python 3 (maybe<br>
>>>> something to do with <a href="https://review.openstack.org/#/c/622415/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/622415/</a> ?), but<br>
>>>> either way I keep running into import issues.<br>
>>>><br>
>>>> I'll take another look tomorrow, but in the meantime I'm afraid I<br>
>>>> haven't made any meaningful progress. :-(<br>
>>><br>
>>> If no version is specified in the tox.ini then tox defaults to the<br>
>>> version of python used to install it.<br>
>>><br>
>><br>
>> Ah, good to know. I think I installed tox as just "tox" instead of <br>
>> "python-tox", which means I got the py3 version.<br>
>><br>
>> Unfortunately I'm still having trouble running the failing test (and <br>
>> not for the expected reason ;-). The daemon is failing to start with:<br>
>><br>
>> ImportError: No module named tests.functional.utils<br>
<br>
No idea why, but updating the fwaas capabilities to match core neutron <br>
by adding c.CAP_DAC_OVERRIDE and c.CAP_DAC_READ_SEARCH made this go <br>
away. Those are related to file permission checks, but the permissions <br>
on my source tree are, well, permissive, so I'm not sure why that would <br>
be a problem.<br>
<br>
>><br>
>> I'm not seeing any log output from the daemon either for some reason <br>
>> so it's hard to debug. There must be some difference between this and <br>
>> the neutron test environment because in neutron I was getting daemon <br>
>> log output in /opt/stack/logs.<br>
> <br>
> Figured this part out. tox.ini wasn't inheriting some values in the same <br>
> way as neutron. Fix proposed in <a href="https://review.openstack.org/#/c/631035/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/631035/</a><br>
<br>
Actually, I discovered that these logs were happening, they were just in <br>
/tmp. So that change is probably not necessary, especially since it's <br>
breaking ci.<br>
<br>
> <br>
> Now hopefully I can make progress on the rest of it.<br>
<br>
And sure enough, I did. :-)<br>
<br>
In short, we need to look at the thread-specific network namespace in <br>
this test instead of the process-specific one. When we change the <br>
namespace it only affects the thread, unless the call is made from the <br>
process's main thread. Here's a simple(?) example:<br>
<br>
#!/usr/bin/env python<br>
<br>
import ctypes<br>
import os<br>
import threading<br>
<br>
from pyroute2 import netns<br>
<br>
# The python threading identifier is useless here,<br>
# we need to make a syscall<br>
libc = ctypes.CDLL('libc.so.6')<br>
<br>
def do_the_thing(ns):<br>
tid = libc.syscall(186) # This id varies by platform :-/<br>
# Check the starting netns<br>
print('process %s' % os.readlink('/proc/self/ns/net'))<br>
print('thread %s' % os.readlink('/proc/self/task/%s/ns/net' % tid))<br>
# Change the netns<br>
print('changing to %s' % ns)<br>
netns.setns(ns)<br>
# Check again. It should be different<br>
print('process %s' % os.readlink('/proc/self/ns/net'))<br>
print('thread %s\n' % os.readlink('/proc/self/task/%s/ns/net' % tid))<br>
<br>
# Run in main thread<br>
do_the_thing('foo')<br>
# Run in new thread<br>
t = threading.Thread(target=do_the_thing, args=('bar',))<br>
t.start()<br>
t.join()<br>
# Run in main thread again to show difference<br>
do_the_thing('bar')<br>
<br>
# Clean up after ourselves<br>
netns.remove('foo')<br>
netns.remove('bar')<br>
<br>
And here's the output:<br>
<br>
process net:[4026531992]<br>
thread net:[4026531992]<br>
changing to foo<br>
process net:[4026532196] <- Running in the main thread changes both<br>
thread net:[4026532196]<br>
<br>
process net:[4026532196]<br>
thread net:[4026532196]<br>
changing to bar<br>
process net:[4026532196] <- Child thread only changes the thread<br>
thread net:[4026532254]<br>
<br>
process net:[4026532196]<br>
thread net:[4026532196]<br>
changing to bar<br>
process net:[4026532254] <- Main thread gets them back in sync<br>
thread net:[4026532254]<br>
<br>
So, to get this test passing I think we need to change [1] so it looks <br>
for the thread id and uses a replacement for [2] that allows the thread <br>
id to be injected as above.<br>
<br>
And it's the end of my day so I'm going to leave it there. :-)<br>
<br>
1: <br>
<a href="https://github.com/openstack/neutron-fwaas/blob/master/neutron_fwaas/privileged/tests/functional/utils.py#L23" rel="noreferrer" target="_blank">https://github.com/openstack/neutron-fwaas/blob/master/neutron_fwaas/privileged/tests/functional/utils.py#L23</a><br>
2: <br>
<a href="https://github.com/openstack/neutron-fwaas/blob/master/neutron_fwaas/privileged/utils.py#L25" rel="noreferrer" target="_blank">https://github.com/openstack/neutron-fwaas/blob/master/neutron_fwaas/privileged/utils.py#L25</a><br>
<br>
-Ben<br>
</blockquote></div></div></div>