Open Stack

I like #3 for windows. ("Do nothing.") :)

-- dims

On Tue, Aug 18, 2015 at 6:37 AM, Elena Ezhova <eezhova at mirantis.com> wrote:

> Thank you for bringing it up, Marian!
>
> On Mon, Aug 17, 2015 at 5:29 PM, mhorban <mhorban at mirantis.com> wrote:
>
>> Most of openstack services use ProcessLauncher(located in oslo.services)
>> to run services, fork new worker processes, reload configuration, etc.
>> Initialization of services in master process usually contains opening of
>> sockets, so that socket will be inherited in child processes. Then
>> master(parent) process spawns(with fork call) children. Communication
>> between master process and children is implemented with signals, for
>> example when master process wants to shutdown children it sends SIGTERM
>> signal to children, to reload children config master process sends SIGHUP
>> signal.
>>
>> I would like to discuss three things:
>>
>
> The first two points also apply to ServiceLauncher which is used to start
> services in case a number of workers is 0 or 1:
>
>
>> 1. Behaviour of reloading configuration in children processes.
>
>
> If a service receives SIGHUP it is restarted which implies calling stop,
> reset and then start.
>
> 2. Additional way to control of master process.
>>
>
>  ServiceLauncher handles the same signals as ProcessLauncher.
>
>
>> 3. Communication between master and children processes.
>
>
>> 1. Behaviour of reloading configuration in children processes.
>> Now we can reload processes by sending SIGHUP to master process. Master
>> process reloads its own configuration and sends SIGHUP signal to children.
>> When child process receives SIGHUP signal it loads config file, stops and
>> starts service. During stopping-starting service new config options usually
>> don't applied because there should be written a lot of code to manage
>> cofiguration changes. rpodolyaka expressed idea to shutdown children during
>> reloading configuration and respawn new workers. This approach frees us of
>> implementing a huge amount of service-related code for reloading
>> configuration of specific objects. Apache and NGINX uses the same reloading
>> approach: gracefully stop children and start new child instances.
>>
>
> From my point of view, this is a double-edged sword. On the one hand,
> respawning workers would indeed free us from writing reset logic for each
> service and save a lot of effort and time. On the other hand, the idea
> behind using SIGHUPs was to have an ability to make changing configuration
> as instantaneous and invisible to users as possible, ideally without
> stopping services. If we choose not to stop services at all (i.e. getting
> rid of calling stop and start on receiving SIGHUP) we'll be able to
> implement flexible configuration reloading (for example, changes to the RPC
> section of a config will invoke only re-initialization of RPC connections
> while db connections wouldn't be affected in any way).
> But this approach is really quite complex so I think we have to weight the
> pros and cons before making a decision.
>
>>
>> 2. Additional way to control of master process.
>> Right now we can control ProcessLauncher by sending signals to master
>> process. It is the only way to manage it. The problem is that such approach
>> is not platform independent. We already faced an issue: Windows doesn't
>> support SIGHUP signal, so part of functionality is inaccessible in Windows
>> :(. Usually process containers like ProcessLauncher could be managed by CLI
>> too. What do you think about creating listening interface for incoming
>> commands?
>>
>
> So, the main problem here is the lack of functionality on Windows. I see
> several ways how to solve this problem:
>
> 1. Send commands to a master process like Marian suggests. As a transport
> we can use sockets or pipes (need to research how are pipes implemented on
> Windows).
> 2. Accept one of the signals available on Windows as SIGHUP. There are
> quite a few of them [1], [2] but it'll still require some research on how
> Windows implements signals.
> 3. Do nothing. The fact that oslo.service currently doesn't support SIGHUP
> on Windows is documented and Windows users can simply restart a service if
> they need to reload config files.
>
>
>> 3. Communication between master and children processes.
>> Master process uses signals to control children. Since many signals are
>> not supported on some platforms I suggest to replace signal mechanism with
>> pipes. Each of children will listen to input commands on one side of pipe
>> and master process will write commands on the other side of pipe.
>>
>
> Pipes might be a good idea, but again we shouldn't remove handlers in
> children at least for those signals that are supported on all platforms
> (SIGINT and SIGTERM). There can be situations when we need to send a
> termination signal to some of the workers directly.
>
> Just my 2 cents :)
>
>
>
>>
>> Any idea?
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> [1] https://docs.python.org/2/library/signal.html#signal.signal
> [2] https://msdn.microsoft.com/en-us/library/xdkz3x12.aspx
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Davanum Srinivas :: https://twitter.com/dims
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150818/8a93b8b4/attachment.html>