Open Stack

I'm not sure if you can test this behaviour on your own because it requires
the VMware plugin and the eventlet handling of backend response.

But the issue was manifesting and had to be fixed with this mega-hack [1].
The issue was not about several workers executing the same code - the
loopingcall was always started on a single thread. The issue I witnessed
was that the other API workers just hang.

There's probably something we need to understand about how eventlet can
work safely with a os.fork (I just think they're not really made to work
together!).
Regardless, I did not spent too much time on it, because I thought that the
multiple workers code might have been rewritten anyway by the pecan switch
activities you're doing.

Salvatore


[1] https://review.openstack.org/#/c/180145/

On 3 June 2015 at 02:20, Kevin Benton <blak111 at gmail.com> wrote:

> Sorry about the long delay.
>
> >Even the LOG.error("KEVIN PID=%s network response: %s" % (os.getpid(),
> r.text)) line?  Surely the server would have forked before that line was
> executed - so what could prevent it from executing once in each forked
> process, and hence generating multiple logs?
>
> Yes, just once. I wasn't able to reproduce the behavior you ran into.
> Maybe eventlet has some protection for this? Can you provide small sample
> code for the logging driver that does reproduce the issue?
>
> On Wed, May 13, 2015 at 5:19 AM, Neil Jerram <Neil.Jerram at metaswitch.com>
> wrote:
>
>> Hi Kevin,
>>
>> Thanks for your response...
>>
>> On 08/05/15 08:43, Kevin Benton wrote:
>>
>>> I'm not sure I understand the behavior you are seeing. When your
>>> mechanism driver gets initialized and kicks off processing, all of that
>>> should be happening in the parent PID. I don't know why your child
>>> processes start executing code that wasn't invoked. Can you provide a
>>> pointer to the code or give a sample that reproduces the issue?
>>>
>>
>> https://github.com/Metaswitch/calico/tree/master/calico/openstack
>>
>> Basically, our driver's initialize method immediately kicks off a green
>> thread to audit what is now in the Neutron DB, and to ensure that the other
>> Calico components are consistent with that.
>>
>>  I modified the linuxbridge mech driver to try to reproduce it:
>>> http://paste.openstack.org/show/216859/
>>>
>>> In the output, I never received any of the init code output I added more
>>> than once, including the function spawned using eventlet.
>>>
>>
>> Interesting.  Even the LOG.error("KEVIN PID=%s network response: %s" %
>> (os.getpid(), r.text)) line?  Surely the server would have forked before
>> that line was executed - so what could prevent it from executing once in
>> each forked process, and hence generating multiple logs?
>>
>> Thanks,
>>         Neil
>>
>>  The only time I ever saw anything executed by a child process was actual
>>> API requests (e.g. the create_port method).
>>>
>>
>>
>>
>>  On Thu, May 7, 2015 at 6:08 AM, Neil Jerram <Neil.Jerram at metaswitch.com
>>> <mailto:Neil.Jerram at metaswitch.com>> wrote:
>>>
>>>     Is there a design for how ML2 mechanism drivers are supposed to cope
>>>     with the Neutron server forking?
>>>
>>>     What I'm currently seeing, with api_workers = 2, is:
>>>
>>>     - my mechanism driver gets instantiated and initialized, and
>>>     immediately kicks off some processing that involves communicating
>>>     over the network
>>>
>>>     - the Neutron server process then forks into multiple copies
>>>
>>>     - multiple copies of my driver's network processing then continue,
>>>     and interfere badly with each other :-)
>>>
>>>     I think what I should do is:
>>>
>>>     - wait until any forking has happened
>>>
>>>     - then decide (somehow) which mechanism driver is going to kick off
>>>     that processing, and do that.
>>>
>>>     But how can a mechanism driver know when the Neutron server forking
>>>     has happened?
>>>
>>>     Thanks,
>>>              Neil
>>>
>>>
>>> __________________________________________________________________________
>>>     OpenStack Development Mailing List (not for usage questions)
>>>     Unsubscribe:
>>>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>     <
>>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>
>>> --
>>> Kevin Benton
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
> Kevin Benton
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150603/1e1a5dc8/attachment.html>