Open Stack

Interestingly, [1] was filed a few moments ago:

[1] https://bugs.launchpad.net/neutron/+bug/1463129

On 2 June 2015 at 22:48, Salvatore Orlando <sorlando at nicira.com> wrote:

> I'm not sure if you can test this behaviour on your own because it
> requires the VMware plugin and the eventlet handling of backend response.
>
> But the issue was manifesting and had to be fixed with this mega-hack [1].
> The issue was not about several workers executing the same code - the
> loopingcall was always started on a single thread. The issue I witnessed
> was that the other API workers just hang.
>
> There's probably something we need to understand about how eventlet can
> work safely with a os.fork (I just think they're not really made to work
> together!).
> Regardless, I did not spent too much time on it, because I thought that
> the multiple workers code might have been rewritten anyway by the pecan
> switch activities you're doing.
>
> Salvatore
>
>
> [1] https://review.openstack.org/#/c/180145/
>
> On 3 June 2015 at 02:20, Kevin Benton <blak111 at gmail.com> wrote:
>
>> Sorry about the long delay.
>>
>> >Even the LOG.error("KEVIN PID=%s network response: %s" % (os.getpid(),
>> r.text)) line?  Surely the server would have forked before that line was
>> executed - so what could prevent it from executing once in each forked
>> process, and hence generating multiple logs?
>>
>> Yes, just once. I wasn't able to reproduce the behavior you ran into.
>> Maybe eventlet has some protection for this? Can you provide small sample
>> code for the logging driver that does reproduce the issue?
>>
>> On Wed, May 13, 2015 at 5:19 AM, Neil Jerram <Neil.Jerram at metaswitch.com>
>> wrote:
>>
>>> Hi Kevin,
>>>
>>> Thanks for your response...
>>>
>>> On 08/05/15 08:43, Kevin Benton wrote:
>>>
>>>> I'm not sure I understand the behavior you are seeing. When your
>>>> mechanism driver gets initialized and kicks off processing, all of that
>>>> should be happening in the parent PID. I don't know why your child
>>>> processes start executing code that wasn't invoked. Can you provide a
>>>> pointer to the code or give a sample that reproduces the issue?
>>>>
>>>
>>> https://github.com/Metaswitch/calico/tree/master/calico/openstack
>>>
>>> Basically, our driver's initialize method immediately kicks off a green
>>> thread to audit what is now in the Neutron DB, and to ensure that the other
>>> Calico components are consistent with that.
>>>
>>>  I modified the linuxbridge mech driver to try to reproduce it:
>>>> http://paste.openstack.org/show/216859/
>>>>
>>>> In the output, I never received any of the init code output I added more
>>>> than once, including the function spawned using eventlet.
>>>>
>>>
>>> Interesting.  Even the LOG.error("KEVIN PID=%s network response: %s" %
>>> (os.getpid(), r.text)) line?  Surely the server would have forked before
>>> that line was executed - so what could prevent it from executing once in
>>> each forked process, and hence generating multiple logs?
>>>
>>> Thanks,
>>>         Neil
>>>
>>>  The only time I ever saw anything executed by a child process was actual
>>>> API requests (e.g. the create_port method).
>>>>
>>>
>>>
>>>
>>>  On Thu, May 7, 2015 at 6:08 AM, Neil Jerram <Neil.Jerram at metaswitch.com
>>>> <mailto:Neil.Jerram at metaswitch.com>> wrote:
>>>>
>>>>     Is there a design for how ML2 mechanism drivers are supposed to cope
>>>>     with the Neutron server forking?
>>>>
>>>>     What I'm currently seeing, with api_workers = 2, is:
>>>>
>>>>     - my mechanism driver gets instantiated and initialized, and
>>>>     immediately kicks off some processing that involves communicating
>>>>     over the network
>>>>
>>>>     - the Neutron server process then forks into multiple copies
>>>>
>>>>     - multiple copies of my driver's network processing then continue,
>>>>     and interfere badly with each other :-)
>>>>
>>>>     I think what I should do is:
>>>>
>>>>     - wait until any forking has happened
>>>>
>>>>     - then decide (somehow) which mechanism driver is going to kick off
>>>>     that processing, and do that.
>>>>
>>>>     But how can a mechanism driver know when the Neutron server forking
>>>>     has happened?
>>>>
>>>>     Thanks,
>>>>              Neil
>>>>
>>>>
>>>> __________________________________________________________________________
>>>>     OpenStack Development Mailing List (not for usage questions)
>>>>     Unsubscribe:
>>>>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>     <
>>>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Kevin Benton
>>>>
>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>>
>> --
>> Kevin Benton
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150608/b693d0d2/attachment.html>