Open Stack

This depends on what initialize is supposed to be doing. If it's just a
one-time sync with a back-end, then I think calling it once in each child
process might not be what we want.

I left a comment on Terry's patch. I think we should just use the callback
manager to have a pre-fork and post-fork even to let drivers/plugins do
whatever is appropriate for them.

On Mon, Jun 8, 2015 at 1:00 PM, Robert Kukura <kukura at noironetworks.com>
wrote:

>  From a driver's perspective, it would be simpler, and I think sufficient,
> to change ML2 to call initialize() on drivers after the forking, rather
> than requiring drivers to know about forking.
>
> -Bob
>
>
> On 6/8/15 2:59 PM, Armando M. wrote:
>
> Interestingly, [1] was filed a few moments ago:
>
>  [1] https://bugs.launchpad.net/neutron/+bug/1463129
>
> On 2 June 2015 at 22:48, Salvatore Orlando <sorlando at nicira.com> wrote:
>
>> I'm not sure if you can test this behaviour on your own because it
>> requires the VMware plugin and the eventlet handling of backend response.
>>
>>  But the issue was manifesting and had to be fixed with this mega-hack
>> [1]. The issue was not about several workers executing the same code - the
>> loopingcall was always started on a single thread. The issue I witnessed
>> was that the other API workers just hang.
>>
>>  There's probably something we need to understand about how eventlet can
>> work safely with a os.fork (I just think they're not really made to work
>> together!).
>> Regardless, I did not spent too much time on it, because I thought that
>> the multiple workers code might have been rewritten anyway by the pecan
>> switch activities you're doing.
>>
>>  Salvatore
>>
>>
>>  [1] https://review.openstack.org/#/c/180145/
>>
>> On 3 June 2015 at 02:20, Kevin Benton <blak111 at gmail.com> wrote:
>>
>>> Sorry about the long delay.
>>>
>>>  >Even the LOG.error("KEVIN PID=%s network response: %s" %
>>> (os.getpid(), r.text)) line?  Surely the server would have forked before
>>> that line was executed - so what could prevent it from executing once in
>>> each forked process, and hence generating multiple logs?
>>>
>>>  Yes, just once. I wasn't able to reproduce the behavior you ran into.
>>> Maybe eventlet has some protection for this? Can you provide small sample
>>> code for the logging driver that does reproduce the issue?
>>>
>>> On Wed, May 13, 2015 at 5:19 AM, Neil Jerram <Neil.Jerram at metaswitch.com
>>> > wrote:
>>>
>>>> Hi Kevin,
>>>>
>>>> Thanks for your response...
>>>>
>>>> On 08/05/15 08:43, Kevin Benton wrote:
>>>>
>>>>> I'm not sure I understand the behavior you are seeing. When your
>>>>> mechanism driver gets initialized and kicks off processing, all of that
>>>>> should be happening in the parent PID. I don't know why your child
>>>>> processes start executing code that wasn't invoked. Can you provide a
>>>>> pointer to the code or give a sample that reproduces the issue?
>>>>>
>>>>
>>>> https://github.com/Metaswitch/calico/tree/master/calico/openstack
>>>>
>>>> Basically, our driver's initialize method immediately kicks off a green
>>>> thread to audit what is now in the Neutron DB, and to ensure that the other
>>>> Calico components are consistent with that.
>>>>
>>>>  I modified the linuxbridge mech driver to try to reproduce it:
>>>>> http://paste.openstack.org/show/216859/
>>>>>
>>>>> In the output, I never received any of the init code output I added
>>>>> more
>>>>> than once, including the function spawned using eventlet.
>>>>>
>>>>
>>>>  Interesting.  Even the LOG.error("KEVIN PID=%s network response: %s" %
>>>> (os.getpid(), r.text)) line?  Surely the server would have forked before
>>>> that line was executed - so what could prevent it from executing once in
>>>> each forked process, and hence generating multiple logs?
>>>>
>>>> Thanks,
>>>>         Neil
>>>>
>>>>  The only time I ever saw anything executed by a child process was
>>>>> actual
>>>>> API requests (e.g. the create_port method).
>>>>>
>>>>
>>>>
>>>>
>>>>  On Thu, May 7, 2015 at 6:08 AM, Neil Jerram <
>>>>> Neil.Jerram at metaswitch.com
>>>>>  <mailto:Neil.Jerram at metaswitch.com>> wrote:
>>>>>
>>>>>     Is there a design for how ML2 mechanism drivers are supposed to
>>>>> cope
>>>>>     with the Neutron server forking?
>>>>>
>>>>>     What I'm currently seeing, with api_workers = 2, is:
>>>>>
>>>>>     - my mechanism driver gets instantiated and initialized, and
>>>>>     immediately kicks off some processing that involves communicating
>>>>>     over the network
>>>>>
>>>>>     - the Neutron server process then forks into multiple copies
>>>>>
>>>>>     - multiple copies of my driver's network processing then continue,
>>>>>     and interfere badly with each other :-)
>>>>>
>>>>>     I think what I should do is:
>>>>>
>>>>>     - wait until any forking has happened
>>>>>
>>>>>     - then decide (somehow) which mechanism driver is going to kick off
>>>>>     that processing, and do that.
>>>>>
>>>>>     But how can a mechanism driver know when the Neutron server forking
>>>>>     has happened?
>>>>>
>>>>>     Thanks,
>>>>>              Neil
>>>>>
>>>>>
>>>>> __________________________________________________________________________
>>>>>     OpenStack Development Mailing List (not for usage questions)
>>>>>     Unsubscribe:
>>>>>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>      <
>>>>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>>>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Kevin Benton
>>>>>
>>>>>
>>>>>
>>>>> __________________________________________________________________________
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>>
>>>   --
>>>  Kevin Benton
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Kevin Benton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150608/d44fa827/attachment.html>