[openstack-dev] [oslo][oslo.config][oslo.service] Dynamic Reconfiguration of OpenStack Services
Joshua Harlow
harlowja at fastmail.com
Wed Nov 4 19:34:32 UTC 2015
Along this line, thinks like the following are likely more changeable
(and my guess is operators would want to change them when things start
going badly), for example from a nova.conf that I have laying around...
[DEFAULT]
rabbit_hosts=...
rpc_response_timeout=...
default_notification_level=...
default_log_levels=...
[glance]
api_servers=...
(and more)
Some of those I think should have higher priority as being
reconfigurable, but I think operators should be asked what they think
would be useful and prioritize those.
Some of those really are service discovery 'types' (rabbit_hosts,
glance/api_servers, keystone/api_servers) but fixing this is likely a
longer term goal (see conversations in keystone).
Joshua Harlow wrote:
> gord chung wrote:
>> we actually had a solution implemented in Ceilometer to handle this[1].
>>
>> that said, based on the results of our survey[2], we found that most
>> operators *never* update configuration files after the initial setup and
>> if they did it was very rarely (monthly updates). the question related
>> to Ceilometer and its pipeline configuration file so the results might
>> be specific to Ceilometer. I think you should definitely query operators
>> before undertaking any work. the last thing you want to do is implement
>> a feature no one really needs/wants.
>>
>> [1]
>> http://specs.openstack.org/openstack/ceilometer-specs/specs/liberty/reload-file-based-pipeline-configuration.html
>>
>> [2]
>> http://lists.openstack.org/pipermail/openstack-dev/2015-September/075628.html
>>
>
> So my general though on the above is yes, definitely consult operators
> to see if they would use this, although if a feature doesn't exist and
> has never existed (say outside of ceilometer) then it's sort of hard to
> get an accurate survey result from a group of people that have never had
> the feature in the first place... Either way it should be done, just to
> get more knowledge...
>
> I know operators (at yahoo!) want to be able to dynamically change the
> logging level, and that's not a monthly task, but more of an 'as-needed'
> one that would be very helpful when things start going badly... So
> perhaps the set of reloadable configuration should start out small and
> not encompass all the things...
>
>>
>> On 04/11/2015 10:00 AM, Marian Horban wrote:
>>> Hi guys,
>>>
>>> Unfortunately I haven't been on Tokio summit but I know that there was
>>> discussion about dynamic reloading of configuration.
>>> Etherpad refs:
>>> https://etherpad.openstack.org/p/mitaka-cross-project-dynamic-config-services,
>>>
>>>
>>> https://etherpad.openstack.org/p/mitaka-oslo-security-logging
>>>
>>> In this thread I want to discuss agreements reached on the summit and
>>> discuss
>>> implementation details.
>>>
>>> Some notes taken from etherpad and my remarks:
>>>
>>> 1. "Adding "mutable" parameter for each option."
>>> "Do we have an option mutable=True on CfgOpt? Yes"
>>> ---------------------------------------------------------
>>> As I understood 'mutable' parameter must indicate whether service
>>> contains
>>> code responsible for reloading of this option or not.
>>> And this parameter should be one of the arguments of cfg.Opt
>>> constructor.
>>> Problems:
>>> 1. Library's options.
>>> SSL options ca_file, cert_file, key_file taken from oslo.service library
>>> could be reloaded in nova-api so these options should be mutable...
>>> But for some projects that don't need SSL support reloading of SSL
>>> options
>>> doesn't make sense. For such projects this option should be non mutable.
>>> Problem is that oslo.service - single and there are many different
>>> projects
>>> which use it in different way.
>>> The same options could be mutable and non mutable in different contexts.
>>> 2. Support of config options on some platforms.
>>> Parameter "mutable" could be different for different platforms. Some
>>> options
>>> make sense only for specific platforms. If we mark such options as
>>> mutable
>>> it could be misleading on some platforms.
>>> 3. Dependency of options.
>>> There are many 'workers' options(osapi_compute_workers, ec2_workers,
>>> metadata_workers, workers). These options specify number of workers for
>>> OpenStack API services.
>>> If value of the 'workers' option is greater than '1' instance of
>>> ProcessLauncher is created otherwise instance of ServiceLauncher is
>>> created.
>>> When ProcessLauncher receives SIGHUP it reloads it own configuration,
>>> gracefully terminates children and respawns new children.
>>> This mechanism allows to reload many config options implicitly.
>>> But if value of the 'workers' option equals '1' instance of
>>> ServiceLauncher
>>> is created.
>>> ServiceLauncher starts everything in single process and in this case we
>>> don't have such implicit reloading.
>>>
>>> I think that mutability of options is a complicated feature and I
>>> think that
>>> adding of 'mutable' parameter into cfg.Opt constructor could just add
>>> mess.
>>>
>>> 2. "oslo.service catches SIGHUP and calls oslo.config"
>>> ---------------------------------------------------------
>>> From my point of view every service should register list of hooks to
>>> reload
>>> config options. oslo.service should catch SIGHUP and call list of
>>> registered
>>> hooks one by one with specified order.
>>> Discussion of such implementation was started in ML:
>>> http://lists.openstack.org/pipermail/openstack-dev/2015-September/074558.html.
>>>
>>> Raw reviews:
>>> https://review.openstack.org/#/c/228892/,
>>> https://review.openstack.org/#/c/223668/.
>>>
>>> 3. "oslo.config is responsible to log changes which were ignored on
>>> SIGHUP"
>>> ---------------------------------------------------------
>>> Some config options could be changed using API(for example quotas)
>>> that's why
>>> oslo.config doesn't know actual configuration of service and can't log
>>> changes of configuration.
>>>
>>> Regards, Marian Horban
>>>
>>>
>>> __________________________________________________________________________
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> --
>> gord
>>
>> __________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
More information about the OpenStack-dev
mailing list