[openstack-dev] [Oslo][Neutron] Fork() safety and oslo.messaging
sileht at sileht.net
Tue Nov 25 10:38:35 UTC 2014
I think the main issue is the behavior of the API
of oslo-incubator/openstack/common/service.py, specially:
And then the MyService have this behavior:
# CODE DONE BEFORE os.fork()
# CODE DONE AFTER os.fork()
So if an application created a FD inside MyService.__init__ or
before ProcessLauncher.launch_service, it will be shared between
processes and we got this kind of issues...
For the rabbitmq/qpid driver, the first connection is created when the
rpc server is started or when the first rpc call/cast/... is done.
So if the application doesn't do that inside MyService.__init__ or
before ProcessLauncher.launch_service everything works as expected.
But if the issue is raised I think this is an application issue (rpc
stuff done before the os.fork())
For the amqp1 driver case, I think this is the same things, it seems
to do lazy creation of the connection too.
I will take a look to the neutron code, if I found a rpc usage
before the os.fork().
Personally, I don't like this API, because the behavior difference between
'__init__' and 'start' is too implicit.
mail: sileht at sileht.net
Le 2014-11-24 20:27, Ken Giusti a écrit :
> Hi all,
> As far as oslo.messaging is concerned, should it be possible for the
> main application to safely os.fork() when there is already an active
> connection to a messaging broker?
> I ask because I'm hitting what appears to be fork-related issues with
> the new AMQP 1.0 driver. I think the same problems have been seen
> with the older impl_qpid driver as well 
> Both drivers utilize a background threading.Thread that handles all
> async socket I/O and protocol timers.
> In the particular case I'm trying to debug, rpc_workers is set to 4 in
> neutron.conf. As far as I can tell, this causes neutron.service to
> os.fork() four workers, but does so after it has created a listener
> (and therefore a connection to the broker).
> This results in multiple processes all select()'ing the same set of
> networks sockets, and stuff breaks :(
> Even without the background process, wouldn't this use still result in
> sockets being shared across the parent/child processes? Seems
>  https://bugs.launchpad.net/oslo.messaging/+bug/1330199
More information about the OpenStack-dev