<div dir="ltr">This depends on what initialize is supposed to be doing. If it's just a one-time sync with a back-end, then I think calling it once in each child process might not be what we want.<div><br></div><div>I left a comment on Terry's patch. I think we should just use the callback manager to have a pre-fork and post-fork even to let drivers/plugins do whatever is appropriate for them.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 8, 2015 at 1:00 PM, Robert Kukura <span dir="ltr"><<a href="mailto:kukura@noironetworks.com" target="_blank">kukura@noironetworks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
From a driver's perspective, it would be simpler, and I think
sufficient, to change ML2 to call initialize() on drivers after the
forking, rather than requiring drivers to know about forking. <br>
<br>
-Bob<div><div class="h5"><br>
<br>
<div>On 6/8/15 2:59 PM, Armando M. wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Interestingly, [1] was filed a few moments ago:
<div>
<div><br>
</div>
<div>[1] <a href="https://bugs.launchpad.net/neutron/+bug/1463129" target="_blank">https://bugs.launchpad.net/neutron/+bug/1463129</a><br>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 2 June 2015 at 22:48, Salvatore
Orlando <span dir="ltr"><<a href="mailto:sorlando@nicira.com" target="_blank">sorlando@nicira.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I'm not sure if you can test this behaviour
on your own because it requires the VMware plugin and the
eventlet handling of backend response.
<div><br>
</div>
<div>But the issue was manifesting and had to be fixed
with this mega-hack [1]. The issue was not about several
workers executing the same code - the loopingcall was
always started on a single thread. The issue I witnessed
was that the other API workers just hang.</div>
<div><br>
</div>
<div>There's probably something we need to understand
about how eventlet can work safely with a os.fork (I
just think they're not really made to work together!).</div>
<div>Regardless, I did not spent too much time on it,
because I thought that the multiple workers code might
have been rewritten anyway by the pecan switch
activities you're doing.</div>
<div><br>
</div>
<div>Salvatore</div>
<div><br>
</div>
<div><br>
</div>
<div>[1] <a href="https://review.openstack.org/#/c/180145/" target="_blank">https://review.openstack.org/#/c/180145/</a></div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 3 June 2015 at 02:20,
Kevin Benton <span dir="ltr"><<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Sorry about the long delay.<span>
<div><br>
</div>
<div>><span style="font-size:12.8000001907349px">Even
the LOG.error("KEVIN PID=%s network
response: %s" % (os.getpid(), r.text))
line? Surely the server would have forked
before that line was executed - so what
could prevent it from executing once in
each forked process, and hence generating
multiple logs?</span></div>
<div><span style="font-size:12.8000001907349px"><br>
</span></div>
</span>
<div>Yes, just once. I wasn't able to reproduce
the behavior you ran into. Maybe eventlet has
some protection for this? Can you provide
small sample code for the logging driver that
does reproduce the issue? </div>
</div>
<div class="gmail_extra">
<div>
<div><br>
<div class="gmail_quote">On Wed, May 13,
2015 at 5:19 AM, Neil Jerram <span dir="ltr"><<a href="mailto:Neil.Jerram@metaswitch.com" target="_blank">Neil.Jerram@metaswitch.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Kevin,<br>
<br>
Thanks for your response...<span><br>
<br>
On 08/05/15 08:43, Kevin Benton wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm not sure I understand the
behavior you are seeing. When your<br>
mechanism driver gets initialized
and kicks off processing, all of
that<br>
should be happening in the parent
PID. I don't know why your child<br>
processes start executing code that
wasn't invoked. Can you provide a<br>
pointer to the code or give a sample
that reproduces the issue?<br>
</blockquote>
<br>
</span><a href="https://github.com/Metaswitch/calico/tree/master/calico/openstack" target="_blank">https://github.com/Metaswitch/calico/tree/master/calico/openstack</a><br>
<br>
Basically, our driver's initialize
method immediately kicks off a green
thread to audit what is now in the
Neutron DB, and to ensure that the other
Calico components are consistent with
that.<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I modified the linuxbridge mech
driver to try to reproduce it:<br>
<a href="http://paste.openstack.org/show/216859/" target="_blank">http://paste.openstack.org/show/216859/</a><br>
<br>
In the output, I never received any
of the init code output I added more<br>
than once, including the function
spawned using eventlet.<br>
</blockquote>
<br>
</span>
Interesting. Even the LOG.error("KEVIN
PID=%s network response: %s" %
(os.getpid(), r.text)) line? Surely the
server would have forked before that
line was executed - so what could
prevent it from executing once in each
forked process, and hence generating
multiple logs?<br>
<br>
Thanks,<br>
Neil<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The only time I ever saw anything
executed by a child process was
actual<br>
API requests (e.g. the create_port
method).<br>
</blockquote>
<br>
<br>
<br>
</span>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>
On Thu, May 7, 2015 at 6:08 AM, Neil
Jerram <<a href="mailto:Neil.Jerram@metaswitch.com" target="_blank">Neil.Jerram@metaswitch.com</a><br>
</span><span>
<mailto:<a href="mailto:Neil.Jerram@metaswitch.com" target="_blank">Neil.Jerram@metaswitch.com</a>>>
wrote:<br>
<br>
Is there a design for how ML2
mechanism drivers are supposed to
cope<br>
with the Neutron server forking?<br>
<br>
What I'm currently seeing, with
api_workers = 2, is:<br>
<br>
- my mechanism driver gets
instantiated and initialized, and<br>
immediately kicks off some
processing that involves
communicating<br>
over the network<br>
<br>
- the Neutron server process
then forks into multiple copies<br>
<br>
- multiple copies of my driver's
network processing then continue,<br>
and interfere badly with each
other :-)<br>
<br>
I think what I should do is:<br>
<br>
- wait until any forking has
happened<br>
<br>
- then decide (somehow) which
mechanism driver is going to kick
off<br>
that processing, and do that.<br>
<br>
But how can a mechanism driver
know when the Neutron server forking<br>
has happened?<br>
<br>
Thanks,<br>
Neil<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing
List (not for usage questions)<br>
Unsubscribe:<br>
<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
</span>
<<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><span><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
<br>
<br>
--<br>
Kevin Benton<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List
(not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</span></blockquote>
<div>
<div>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List
(not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
</div>
</div>
<span><font color="#888888">-- <br>
<div>
<div>Kevin Benton</div>
</div>
</font></span></div>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage
questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</div></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div>Kevin Benton</div></div>
</div>