[openstack-dev] [oslo][oslo.config][oslo.service] Dynamic Reconfiguration of OpenStack Services

Joshua Harlow harlowja at fastmail.com
Wed Nov 4 22:13:33 UTC 2015


Agreed, it'd be nice to have an 'audit' of the various projects configs 
and try to categorize which ones should be reloadable (and the priority 
to make it reloadable) and which ones are service discovery configs (and 
probably shouldn't be in config in the first place) and which ones are 
nice to haves to be configurable... (like rpc_response_timeout).

The side-effects of making a few things configurable will likely cause a 
whole bunch of other issues anyway (like how an application/library... 
gets notified that a config has changed and it may need to re-adjust 
itself to those new values which may include for example unloading a 
driver, stopping a thread, starting a new driver...), so I'm thinking we 
should start small and grow as needed (based on above prioritization).

Log levels are high on my known list.

Fox, Kevin M wrote:
> Ah, I hadn't considered the rabbit_hosts (discovery) case. yeah, that would be a useful thing to be able to tweak live. I haven't needed that feature yet, but could see how that flexibility could come in handy.
>
> Thanks,
> Kevin
> ________________________________________
> From: Joshua Harlow [harlowja at fastmail.com]
> Sent: Wednesday, November 04, 2015 11:34 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [oslo][oslo.config][oslo.service] Dynamic Reconfiguration of OpenStack Services
>
> Along this line, thinks like the following are likely more changeable
> (and my guess is operators would want to change them when things start
> going badly), for example from a nova.conf that I have laying around...
>
> [DEFAULT]
>
> rabbit_hosts=...
> rpc_response_timeout=...
> default_notification_level=...
> default_log_levels=...
>
> [glance]
>
> api_servers=...
>
> (and more)
>
> Some of those I think should have higher priority as being
> reconfigurable, but I think operators should be asked what they think
> would be useful and prioritize those.
>
> Some of those really are service discovery 'types' (rabbit_hosts,
> glance/api_servers, keystone/api_servers) but fixing this is likely a
> longer term goal (see conversations in keystone).
>
> Joshua Harlow wrote:
>> gord chung wrote:
>>> we actually had a solution implemented in Ceilometer to handle this[1].
>>>
>>> that said, based on the results of our survey[2], we found that most
>>> operators *never* update configuration files after the initial setup and
>>> if they did it was very rarely (monthly updates). the question related
>>> to Ceilometer and its pipeline configuration file so the results might
>>> be specific to Ceilometer. I think you should definitely query operators
>>> before undertaking any work. the last thing you want to do is implement
>>> a feature no one really needs/wants.
>>>
>>> [1]
>>> http://specs.openstack.org/openstack/ceilometer-specs/specs/liberty/reload-file-based-pipeline-configuration.html
>>>
>>> [2]
>>> http://lists.openstack.org/pipermail/openstack-dev/2015-September/075628.html
>>>
>> So my general though on the above is yes, definitely consult operators
>> to see if they would use this, although if a feature doesn't exist and
>> has never existed (say outside of ceilometer) then it's sort of hard to
>> get an accurate survey result from a group of people that have never had
>> the feature in the first place... Either way it should be done, just to
>> get more knowledge...
>>
>> I know operators (at yahoo!) want to be able to dynamically change the
>> logging level, and that's not a monthly task, but more of an 'as-needed'
>> one that would be very helpful when things start going badly... So
>> perhaps the set of reloadable configuration should start out small and
>> not encompass all the things...
>>
>>> On 04/11/2015 10:00 AM, Marian Horban wrote:
>>>> Hi guys,
>>>>
>>>> Unfortunately I haven't been on Tokio summit but I know that there was
>>>> discussion about dynamic reloading of configuration.
>>>> Etherpad refs:
>>>> https://etherpad.openstack.org/p/mitaka-cross-project-dynamic-config-services,
>>>>
>>>>
>>>> https://etherpad.openstack.org/p/mitaka-oslo-security-logging
>>>>
>>>> In this thread I want to discuss agreements reached on the summit and
>>>> discuss
>>>> implementation details.
>>>>
>>>> Some notes taken from etherpad and my remarks:
>>>>
>>>> 1. "Adding "mutable" parameter for each option."
>>>> "Do we have an option mutable=True on CfgOpt? Yes"
>>>> ---------------------------------------------------------
>>>> As I understood 'mutable' parameter must indicate whether service
>>>> contains
>>>> code responsible for reloading of this option or not.
>>>> And this parameter should be one of the arguments of cfg.Opt
>>>> constructor.
>>>> Problems:
>>>> 1. Library's options.
>>>> SSL options ca_file, cert_file, key_file taken from oslo.service library
>>>> could be reloaded in nova-api so these options should be mutable...
>>>> But for some projects that don't need SSL support reloading of SSL
>>>> options
>>>> doesn't make sense. For such projects this option should be non mutable.
>>>> Problem is that oslo.service - single and there are many different
>>>> projects
>>>> which use it in different way.
>>>> The same options could be mutable and non mutable in different contexts.
>>>> 2. Support of config options on some platforms.
>>>> Parameter "mutable" could be different for different platforms. Some
>>>> options
>>>> make sense only for specific platforms. If we mark such options as
>>>> mutable
>>>> it could be misleading on some platforms.
>>>> 3. Dependency of options.
>>>> There are many 'workers' options(osapi_compute_workers, ec2_workers,
>>>> metadata_workers, workers). These options specify number of workers for
>>>> OpenStack API services.
>>>> If value of the 'workers' option is greater than '1' instance of
>>>> ProcessLauncher is created otherwise instance of ServiceLauncher is
>>>> created.
>>>> When ProcessLauncher receives SIGHUP it reloads it own configuration,
>>>> gracefully terminates children and respawns new children.
>>>> This mechanism allows to reload many config options implicitly.
>>>> But if value of the 'workers' option equals '1' instance of
>>>> ServiceLauncher
>>>> is created.
>>>> ServiceLauncher starts everything in single process and in this case we
>>>> don't have such implicit reloading.
>>>>
>>>> I think that mutability of options is a complicated feature and I
>>>> think that
>>>> adding of 'mutable' parameter into cfg.Opt constructor could just add
>>>> mess.
>>>>
>>>> 2. "oslo.service catches SIGHUP and calls oslo.config"
>>>> ---------------------------------------------------------
>>>>  From my point of view every service should register list of hooks to
>>>> reload
>>>> config options. oslo.service should catch SIGHUP and call list of
>>>> registered
>>>> hooks one by one with specified order.
>>>> Discussion of such implementation was started in ML:
>>>> http://lists.openstack.org/pipermail/openstack-dev/2015-September/074558.html.
>>>>
>>>> Raw reviews:
>>>> https://review.openstack.org/#/c/228892/,
>>>> https://review.openstack.org/#/c/223668/.
>>>>
>>>> 3. "oslo.config is responsible to log changes which were ignored on
>>>> SIGHUP"
>>>> ---------------------------------------------------------
>>>> Some config options could be changed using API(for example quotas)
>>>> that's why
>>>> oslo.config doesn't know actual configuration of service and can't log
>>>> changes of configuration.
>>>>
>>>> Regards, Marian Horban
>>>>
>>>>
>>>> __________________________________________________________________________
>>>>
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> --
>>> gord
>>>
>>> __________________________________________________________________________
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list