<div dir="ltr">This depends on what initialize is supposed to be doing. If it's just a one-time sync with a back-end, then I think calling it once in each child process might not be what we want.<div><br></div><div>I left a comment on Terry's patch. I think we should just use the callback manager to have a pre-fork and post-fork even to let drivers/plugins do whatever is appropriate for them.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 8, 2015 at 1:00 PM, Robert Kukura <span dir="ltr"><<a href="mailto:kukura@noironetworks.com" target="_blank">kukura@noironetworks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    From a driver's perspective, it would be simpler, and I think
    sufficient, to change ML2 to call initialize() on drivers after the
    forking, rather than requiring drivers to know about forking. <br>
    <br>
    -Bob<div><div class="h5"><br>
    <br>
    <div>On 6/8/15 2:59 PM, Armando M. wrote:<br>
    </div>
    <blockquote type="cite">
      <div dir="ltr">Interestingly, [1] was filed a few moments ago:
        <div>
          <div><br>
          </div>
          <div>[1] <a href="https://bugs.launchpad.net/neutron/+bug/1463129" target="_blank">https://bugs.launchpad.net/neutron/+bug/1463129</a><br>
          </div>
        </div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On 2 June 2015 at 22:48, Salvatore
          Orlando <span dir="ltr"><<a href="mailto:sorlando@nicira.com" target="_blank">sorlando@nicira.com</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">I'm not sure if you can test this behaviour
              on your own because it requires the VMware plugin and the
              eventlet handling of backend response.
              <div><br>
              </div>
              <div>But the issue was manifesting and had to be fixed
                with this mega-hack [1]. The issue was not about several
                workers executing the same code - the loopingcall was
                always started on a single thread. The issue I witnessed
                was that the other API workers just hang.</div>
              <div><br>
              </div>
              <div>There's probably something we need to understand
                about how eventlet can work safely with a os.fork (I
                just think they're not really made to work together!).</div>
              <div>Regardless, I did not spent too much time on it,
                because I thought that the multiple workers code might
                have been rewritten anyway by the pecan switch
                activities you're doing.</div>
              <div><br>
              </div>
              <div>Salvatore</div>
              <div><br>
              </div>
              <div><br>
              </div>
              <div>[1] <a href="https://review.openstack.org/#/c/180145/" target="_blank">https://review.openstack.org/#/c/180145/</a></div>
            </div>
            <div>
              <div>
                <div class="gmail_extra"><br>
                  <div class="gmail_quote">On 3 June 2015 at 02:20,
                    Kevin Benton <span dir="ltr"><<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>></span>
                    wrote:<br>
                    <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                      <div dir="ltr">Sorry about the long delay.<span>
                          <div><br>
                          </div>
                          <div>><span style="font-size:12.8000001907349px">Even
                              the LOG.error("KEVIN PID=%s network
                              response: %s" % (os.getpid(), r.text))
                              line?  Surely the server would have forked
                              before that line was executed - so what
                              could prevent it from executing once in
                              each forked process, and hence generating
                              multiple logs?</span></div>
                          <div><span style="font-size:12.8000001907349px"><br>
                            </span></div>
                        </span>
                        <div>Yes, just once. I wasn't able to reproduce
                          the behavior you ran into. Maybe eventlet has
                          some protection for this? Can you provide
                          small sample code for the logging driver that
                          does reproduce the issue? </div>
                      </div>
                      <div class="gmail_extra">
                        <div>
                          <div><br>
                            <div class="gmail_quote">On Wed, May 13,
                              2015 at 5:19 AM, Neil Jerram <span dir="ltr"><<a href="mailto:Neil.Jerram@metaswitch.com" target="_blank">Neil.Jerram@metaswitch.com</a>></span>
                              wrote:<br>
                              <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Kevin,<br>
                                <br>
                                Thanks for your response...<span><br>
                                  <br>
                                  On 08/05/15 08:43, Kevin Benton wrote:<br>
                                  <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                    I'm not sure I understand the
                                    behavior you are seeing. When your<br>
                                    mechanism driver gets initialized
                                    and kicks off processing, all of
                                    that<br>
                                    should be happening in the parent
                                    PID. I don't know why your child<br>
                                    processes start executing code that
                                    wasn't invoked. Can you provide a<br>
                                    pointer to the code or give a sample
                                    that reproduces the issue?<br>
                                  </blockquote>
                                  <br>
                                </span><a href="https://github.com/Metaswitch/calico/tree/master/calico/openstack" target="_blank">https://github.com/Metaswitch/calico/tree/master/calico/openstack</a><br>
                                <br>
                                Basically, our driver's initialize
                                method immediately kicks off a green
                                thread to audit what is now in the
                                Neutron DB, and to ensure that the other
                                Calico components are consistent with
                                that.<span><br>
                                  <br>
                                  <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                    I modified the linuxbridge mech
                                    driver to try to reproduce it:<br>
                                    <a href="http://paste.openstack.org/show/216859/" target="_blank">http://paste.openstack.org/show/216859/</a><br>
                                    <br>
                                    In the output, I never received any
                                    of the init code output I added more<br>
                                    than once, including the function
                                    spawned using eventlet.<br>
                                  </blockquote>
                                  <br>
                                </span>
                                Interesting.  Even the LOG.error("KEVIN
                                PID=%s network response: %s" %
                                (os.getpid(), r.text)) line?  Surely the
                                server would have forked before that
                                line was executed - so what could
                                prevent it from executing once in each
                                forked process, and hence generating
                                multiple logs?<br>
                                <br>
                                Thanks,<br>
                                        Neil<span><br>
                                  <br>
                                  <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                    The only time I ever saw anything
                                    executed by a child process was
                                    actual<br>
                                    API requests (e.g. the create_port
                                    method).<br>
                                  </blockquote>
                                  <br>
                                  <br>
                                  <br>
                                </span>
                                <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>
                                    On Thu, May 7, 2015 at 6:08 AM, Neil
                                    Jerram <<a href="mailto:Neil.Jerram@metaswitch.com" target="_blank">Neil.Jerram@metaswitch.com</a><br>
                                  </span><span>
                                    <mailto:<a href="mailto:Neil.Jerram@metaswitch.com" target="_blank">Neil.Jerram@metaswitch.com</a>>>
                                    wrote:<br>
                                    <br>
                                        Is there a design for how ML2
                                    mechanism drivers are supposed to
                                    cope<br>
                                        with the Neutron server forking?<br>
                                    <br>
                                        What I'm currently seeing, with
                                    api_workers = 2, is:<br>
                                    <br>
                                        - my mechanism driver gets
                                    instantiated and initialized, and<br>
                                        immediately kicks off some
                                    processing that involves
                                    communicating<br>
                                        over the network<br>
                                    <br>
                                        - the Neutron server process
                                    then forks into multiple copies<br>
                                    <br>
                                        - multiple copies of my driver's
                                    network processing then continue,<br>
                                        and interfere badly with each
                                    other :-)<br>
                                    <br>
                                        I think what I should do is:<br>
                                    <br>
                                        - wait until any forking has
                                    happened<br>
                                    <br>
                                        - then decide (somehow) which
                                    mechanism driver is going to kick
                                    off<br>
                                        that processing, and do that.<br>
                                    <br>
                                        But how can a mechanism driver
                                    know when the Neutron server forking<br>
                                        has happened?<br>
                                    <br>
                                        Thanks,<br>
                                                 Neil<br>
                                    <br>
                                       
__________________________________________________________________________<br>
                                        OpenStack Development Mailing
                                    List (not for usage questions)<br>
                                        Unsubscribe:<br>
                                        <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
                                  </span>
                                      <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><span><br>
                                        <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
                                    <br>
                                    <br>
                                    <br>
                                    <br>
                                    --<br>
                                    Kevin Benton<br>
                                    <br>
                                    <br>
__________________________________________________________________________<br>
                                    OpenStack Development Mailing List
                                    (not for usage questions)<br>
                                    Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
                                    <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
                                    <br>
                                  </span></blockquote>
                                <div>
                                  <div>
                                    <br>
__________________________________________________________________________<br>
                                    OpenStack Development Mailing List
                                    (not for usage questions)<br>
                                    Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
                                    <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
                                  </div>
                                </div>
                              </blockquote>
                            </div>
                            <br>
                            <br clear="all">
                            <div><br>
                            </div>
                          </div>
                        </div>
                        <span><font color="#888888">-- <br>
                            <div>
                              <div>Kevin Benton</div>
                            </div>
                          </font></span></div>
                      <br>
__________________________________________________________________________<br>
                      OpenStack Development Mailing List (not for usage
                      questions)<br>
                      Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
                      <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
                      <br>
                    </blockquote>
                  </div>
                  <br>
                </div>
              </div>
            </div>
            <br>
__________________________________________________________________________<br>
            OpenStack Development Mailing List (not for usage questions)<br>
            Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
            <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
            <br>
          </blockquote>
        </div>
        <br>
      </div>
      <br>
      <fieldset></fieldset>
      <br>
      <pre>__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
    </blockquote>
    <br>
  </div></div></div>

<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div>Kevin Benton</div></div>
</div>