Open Stack

Thank you for bringing it up, Marian!

On Mon, Aug 17, 2015 at 5:29 PM, mhorban <mhorban at mirantis.com> wrote:

> Most of openstack services use ProcessLauncher(located in oslo.services)
> to run services, fork new worker processes, reload configuration, etc.
> Initialization of services in master process usually contains opening of
> sockets, so that socket will be inherited in child processes. Then
> master(parent) process spawns(with fork call) children. Communication
> between master process and children is implemented with signals, for
> example when master process wants to shutdown children it sends SIGTERM
> signal to children, to reload children config master process sends SIGHUP
> signal.
>
> I would like to discuss three things:
>

The first two points also apply to ServiceLauncher which is used to start
services in case a number of workers is 0 or 1:


> 1. Behaviour of reloading configuration in children processes.


If a service receives SIGHUP it is restarted which implies calling stop,
reset and then start.

2. Additional way to control of master process.
>

 ServiceLauncher handles the same signals as ProcessLauncher.


> 3. Communication between master and children processes.


> 1. Behaviour of reloading configuration in children processes.
> Now we can reload processes by sending SIGHUP to master process. Master
> process reloads its own configuration and sends SIGHUP signal to children.
> When child process receives SIGHUP signal it loads config file, stops and
> starts service. During stopping-starting service new config options usually
> don't applied because there should be written a lot of code to manage
> cofiguration changes. rpodolyaka expressed idea to shutdown children during
> reloading configuration and respawn new workers. This approach frees us of
> implementing a huge amount of service-related code for reloading
> configuration of specific objects. Apache and NGINX uses the same reloading
> approach: gracefully stop children and start new child instances.
>

>From my point of view, this is a double-edged sword. On the one hand,
respawning workers would indeed free us from writing reset logic for each
service and save a lot of effort and time. On the other hand, the idea
behind using SIGHUPs was to have an ability to make changing configuration
as instantaneous and invisible to users as possible, ideally without
stopping services. If we choose not to stop services at all (i.e. getting
rid of calling stop and start on receiving SIGHUP) we'll be able to
implement flexible configuration reloading (for example, changes to the RPC
section of a config will invoke only re-initialization of RPC connections
while db connections wouldn't be affected in any way).
But this approach is really quite complex so I think we have to weight the
pros and cons before making a decision.

>
> 2. Additional way to control of master process.
> Right now we can control ProcessLauncher by sending signals to master
> process. It is the only way to manage it. The problem is that such approach
> is not platform independent. We already faced an issue: Windows doesn't
> support SIGHUP signal, so part of functionality is inaccessible in Windows
> :(. Usually process containers like ProcessLauncher could be managed by CLI
> too. What do you think about creating listening interface for incoming
> commands?
>

So, the main problem here is the lack of functionality on Windows. I see
several ways how to solve this problem:

1. Send commands to a master process like Marian suggests. As a transport
we can use sockets or pipes (need to research how are pipes implemented on
Windows).
2. Accept one of the signals available on Windows as SIGHUP. There are
quite a few of them [1], [2] but it'll still require some research on how
Windows implements signals.
3. Do nothing. The fact that oslo.service currently doesn't support SIGHUP
on Windows is documented and Windows users can simply restart a service if
they need to reload config files.


> 3. Communication between master and children processes.
> Master process uses signals to control children. Since many signals are
> not supported on some platforms I suggest to replace signal mechanism with
> pipes. Each of children will listen to input commands on one side of pipe
> and master process will write commands on the other side of pipe.
>

Pipes might be a good idea, but again we shouldn't remove handlers in
children at least for those signals that are supported on all platforms
(SIGINT and SIGTERM). There can be situations when we need to send a
termination signal to some of the workers directly.

Just my 2 cents :)



>
> Any idea?
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

[1] https://docs.python.org/2/library/signal.html#signal.signal
[2] https://msdn.microsoft.com/en-us/library/xdkz3x12.aspx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150818/1bd36af4/attachment.html>