<div dir="ltr">I'm not sure if you can test this behaviour on your own because it requires the VMware plugin and the eventlet handling of backend response.<div><br></div><div>But the issue was manifesting and had to be fixed with this mega-hack [1]. The issue was not about several workers executing the same code - the loopingcall was always started on a single thread. The issue I witnessed was that the other API workers just hang.</div><div><br></div><div>There's probably something we need to understand about how eventlet can work safely with a os.fork (I just think they're not really made to work together!).</div><div>Regardless, I did not spent too much time on it, because I thought that the multiple workers code might have been rewritten anyway by the pecan switch activities you're doing.</div><div><br></div><div>Salvatore</div><div><br></div><div><br></div><div>[1] <a href="https://review.openstack.org/#/c/180145/">https://review.openstack.org/#/c/180145/</a></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 3 June 2015 at 02:20, Kevin Benton <span dir="ltr"><<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Sorry about the long delay.<span class=""><div><br></div><div>><span style="font-size:12.8000001907349px">Even the LOG.error("KEVIN PID=%s network response: %s" % (os.getpid(), r.text)) line? Surely the server would have forked before that line was executed - so what could prevent it from executing once in each forked process, and hence generating multiple logs?</span></div><div><span style="font-size:12.8000001907349px"><br></span></div></span><div>Yes, just once. I wasn't able to reproduce the behavior you ran into. Maybe eventlet has some protection for this? Can you provide small sample code for the logging driver that does reproduce the issue? </div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Wed, May 13, 2015 at 5:19 AM, Neil Jerram <span dir="ltr"><<a href="mailto:Neil.Jerram@metaswitch.com" target="_blank">Neil.Jerram@metaswitch.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Kevin,<br>
<br>
Thanks for your response...<span><br>
<br>
On 08/05/15 08:43, Kevin Benton wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm not sure I understand the behavior you are seeing. When your<br>
mechanism driver gets initialized and kicks off processing, all of that<br>
should be happening in the parent PID. I don't know why your child<br>
processes start executing code that wasn't invoked. Can you provide a<br>
pointer to the code or give a sample that reproduces the issue?<br>
</blockquote>
<br>
</span><a href="https://github.com/Metaswitch/calico/tree/master/calico/openstack" target="_blank">https://github.com/Metaswitch/calico/tree/master/calico/openstack</a><br>
<br>
Basically, our driver's initialize method immediately kicks off a green thread to audit what is now in the Neutron DB, and to ensure that the other Calico components are consistent with that.<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I modified the linuxbridge mech driver to try to reproduce it:<br>
<a href="http://paste.openstack.org/show/216859/" target="_blank">http://paste.openstack.org/show/216859/</a><br>
<br>
In the output, I never received any of the init code output I added more<br>
than once, including the function spawned using eventlet.<br>
</blockquote>
<br></span>
Interesting. Even the LOG.error("KEVIN PID=%s network response: %s" % (os.getpid(), r.text)) line? Surely the server would have forked before that line was executed - so what could prevent it from executing once in each forked process, and hence generating multiple logs?<br>
<br>
Thanks,<br>
Neil<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The only time I ever saw anything executed by a child process was actual<br>
API requests (e.g. the create_port method).<br>
</blockquote>
<br>
<br>
<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>
On Thu, May 7, 2015 at 6:08 AM, Neil Jerram <<a href="mailto:Neil.Jerram@metaswitch.com" target="_blank">Neil.Jerram@metaswitch.com</a><br></span><span>
<mailto:<a href="mailto:Neil.Jerram@metaswitch.com" target="_blank">Neil.Jerram@metaswitch.com</a>>> wrote:<br>
<br>
Is there a design for how ML2 mechanism drivers are supposed to cope<br>
with the Neutron server forking?<br>
<br>
What I'm currently seeing, with api_workers = 2, is:<br>
<br>
- my mechanism driver gets instantiated and initialized, and<br>
immediately kicks off some processing that involves communicating<br>
over the network<br>
<br>
- the Neutron server process then forks into multiple copies<br>
<br>
- multiple copies of my driver's network processing then continue,<br>
and interfere badly with each other :-)<br>
<br>
I think what I should do is:<br>
<br>
- wait until any forking has happened<br>
<br>
- then decide (somehow) which mechanism driver is going to kick off<br>
that processing, and do that.<br>
<br>
But how can a mechanism driver know when the Neutron server forking<br>
has happened?<br>
<br>
Thanks,<br>
Neil<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe:<br>
<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br></span>
<<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><span><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
<br>
<br>
--<br>
Kevin Benton<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</span></blockquote><div><div>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div><div>Kevin Benton</div></div>
</font></span></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>