Open Stack

On Tue, Oct 11, 2016 at 11:32 AM, Alex Schultz <aschultz at redhat.com> wrote:
> On Tue, Oct 11, 2016 at 1:39 AM, Thomas Bechtold <tbechtold at suse.com> wrote:
>> Hi,
>>
>> On Mon, Oct 10, 2016 at 10:07:05AM -0600, Alex Schultz wrote:
>>> On Mon, Oct 10, 2016 at 5:03 AM, Thomas Bechtold <tbechtold at suse.com> wrote:
>>> > Hi,
>>> >
>>> > in the rpm-packaging project we started to package the services and are
>>> > currently discussing a possible schema for configuration files and
>>> > snippets used by the systemd .service files (but this would also affect
>>> > OCF resource agents).
>>> > This affects packagers, endusers and config management systems (Chef,
>>> > Puppet, Ansible, Salt, ...) so I want to discuss the proposal here on
>>> > the list.
>>>
>>> This also affects deployment tools so you may want to include tripleo,
>>> kolla, fuel as they are downstream consumers and may have their own
>>> assumptions about how services are launched.
>>
>> Done.
>>
>>> > Most services (at least for SUSE and RDO) are using a single config
>>> > (e.g. /etc/nova/nova.conf) when starting the service. Some services
>>> > (e.g. neutron services like neutron-l3-agent) use multiple config files.
>>> >
>>> > There are multiple problems with that approach:
>>> > - when using a config-mgmt-system, users may want to override a config
>>> > option (for a feature that is not yet supported) but the
>>> > config-mgmt-system will override the config later again.
>>>
>>> Just to chime in here from a puppet standpoint, this is not a problem
>>> because we provide a way for a user to add any extra options they wish
>>> using the provider so it always ends up in the correct configuration
>>> file.
>>
>> Does that also work if you need extra configuration files for 3rd party
>> plugins (e.g. neutron plugins) ? I guess you could just copy the 3rd
>> party config file content to the needed neutron config but that's ugly imo.
>>
>
> Plugins also have their own .ini files and are handled differently.
> They get weird but we put the service configs in
> /etc/neutron/neutron.conf and agent configs get put in .ini files.
>
>>> > - when users adjust /etc/$service/$service.conf and a release update is
>>> > done (e.g. mitaka->newton) /etc/$service/$service.conf wil not be
>>> > overridden. That's good because the user changed something but otoh the
>>> > file (with all it's config options and comments) is no longer correct.
>>>
>>> Depending on the configuration management tool, the 'default' options
>>> and comments may not even be there so I'm not sure this is actually
>>> that much of a concern.  Also when you upgrade there is usually some
>>> sort of upgrade process that has to go along with your configuration
>>> management tool which should take care of this for you. So i'm not
>>> sure this needs to be a packaging concern.
>>>
>>> > - when config-mgmt-systems use templates for the $service.conf file,
>>> > updating theses templates is a lot of work.
>>>
>>> Yes which is why tools that don't use templates in the configuration
>>> management tool makes this a non-issue.  I'm not sure this needs to be
>>> a concern of packagers as it's an issue with the configuration
>>> management tool of choice and many of these tools have switched away
>>> from templates or are currently handling this.  If the configuration
>>> management tool doesn't support this or is suffering from this, simply
>>> adding conf.d support might help but then you also run into issues
>>> about ensuring things are removed and cleaned up.
>>
>> There *are* config-mgmt-tools still using templates. And having the
>> possibility to add config snippets simplifies the process here without
>> any downside for available solutions. Just don't use it if you don't
>> need it.
>>
>
> Yes there are but that's not a packaging concern other than providing
> a simple way to not have to deal with constantly managing
> $service.conf.  I'm ok with a conf.d folder, but I guess that's more
> of a question for folks who have to manage those templates today as to
> what their prefered method is. Speaking from experience with a tool
> that we try not to use templates, my preference is still to have
> $service.conf but don't have it replaced on package update.

>From the perspective of Chef, we stopped using erubis templates for
the service configs, adopting an attribute-driven config instead. This
got rid of the impossible to manage, thousand line long templates, and
gave us just what is needed to get the service off the ground.

Moving to a snippets model is feasible from our point of view, but it
doesn't make sense if only rpm does this and deb does not follow suit.
Having to maintain different config models would result in having to
maintain two lines of code to achieve the same desired result.

>
>>> > - setting configuration options per service (let's say cinder-volume
>>> > needs other config options than cinder-api) is not possible.
>>>
>>> So I agree this is more likely a real problem, but i'm not sure this
>>> should be solved by packaging as this probably needs to be addressed
>>> in upstream.  Unless this is already a thing and it's just never been
>>> properly implemented when deployed via packages. The issue I have with
>>> only solving this via rpm packaging is that for tools that support
>>> both rpms and debs this would lead to 2 different configuration
>>> mechanisms.  So I would vote not to do anything different then what
>>> debs also do unless both packaging methods are updated at the same
>>> time.
>>
>> Don't you have already a lot of different implementations for rpm
>> vs. deb, different apache config dir styles, different package name, ...
>> Improving the rpm side only if the deb side also changes is not going to
>> work I think.
>
> Package name and configuration file locations have been handled. What
> I want to prevent is even more configuration files and default
> assumptions.  My concern is more around the increase of default
> assumptions with configurations.
>

My concerns are the increase of default assumptions as well. I don't
want to assume there is always going to be an /etc or a /var, because
there won't always be an /etc or /var. I'm working on Habitat packages
as a side project, and that has none of the traditional hierarchy.
It's astonishing at just how much software assumes these days.

>>
>>> Do we have any examples or instances where an end user would
>>> specifically want to configure two of the services in a conflicting
>>> fashion?  Or are there configuration options that fall into this
>>> pattern? I thought the service specific items where in their own
>>> configuration namespace to allow for such things. I would assume that
>>> the bigger issue would be wanting to run two of the same service on
>>> the same host with different configurations I would think that's where
>>> something like containers would handle this case better than trying to
>>> have multiple configuration files.
>>
>> In HA case you may want to set the hostname for all cinder-volumes to
>> the same name but to something different for cinder-api (if both
>> services run on the same host). I'm sure there are other examples.
>
> That seems like a deployer issue for how to tackle that instead of a
> packaging.  Considering this is probably not a new deployment issues,
> I guess the question would be is this something that would be solved
> by packaging?  Based on the proposed configuration I'm not sure how
> that gets achieved via files and it seems like needs to get fixed
> upstream to support such a configuration on the same host.  It would
> be easier to have the cinder-api look in the cinder-api configuration
> section rather than splitting files.
>

This issue seems like it would be much better solved via
automation/configuration management than via packaging. From a
deployer's perspective, the less "magic" a package does, the easier it
is to consume and support.

>>
>>> > So here's the proposal to fix theses problems. The proposal is based on
>>> > what RDO is already doing with neutron and extends it a bit. Let's do it
>>> > for Nova as an example:
>>> >
>>> > - /usr/share/nova/nova-dist.conf
>>> > This is the configuration file a distribution (openSUSE, RDO, ...) can
>>> > modify. It must not be modified by endusers and will be overridden with
>>> > package updates
>>> >
>>> > - /etc/nova/nova.conf
>>> > This is an empty file . Users/config-mgmt-systems can modify it and it
>>> > will not be overridden (if changed) with a package update.
>>> >
>>> > - /etc/nova/conf.d/common/
>>> > In this directory, config snippets can be added. By convention,
>>> > config-mgmt-systems would install files starting with 100-XXX.conf,
>>> > endusers would install files starting with 500-XXX.conf . Also this
>>> > directory is used by all services (nova-api, nova-compute, ...).
>>> >
>>> > - /etc/nova/conf.d/$service/ (e.g. /etc/nova/conf.d/nova-compute/)
>>> > Also a dir for config snippets (with same rules as for
>>> > /etc/nova/conf.d/common/ ) but this dir is only used by $service (in
>>> > this case nova-compute)
>>> >
>>> > - /usr/share/doc/packages/openstack-nova/nova.conf.sample
>>> > The unmodified sample config from upstream
>>> >
>>> > - /etc/nova/README
>>> > Explaining the configuration structure and where to find the whole
>>> > sample config.
>>> >
>>>
>>> From a packaging standpoint it's probably better to provide less and
>>> not more default configuration files as the tooling usually has to go
>>> an clean this stuff up as they have their own ways of configuring the
>>> services.  Currently they may align with the existing packaging files
>>> or they may completely remove what's provided via packaging and setup
>>> their own structure.  Speaking from experience, having to cleanup
>>> package provided configuration files is a pain[0][1].
>>
>> There are not more config files with this proposal. It's the same
>> amount, just more possibilities. the nova-dist.conf is already in use
>> for RDO and neutron [2]. This would be just used for all services.
>> /etc/$service/$service.conf is the usual thing and also read by default
>> by oslo.config . The conf.d directory delivered by a package would be
>> completly empty. So nothing to cleanup there.
>>
>> [2]
>> https://github.com/rdo-packages/neutron-distgit/blob/rpm-master/neutron-server.service
>>
>
> I wasn't aware of the *-dist.conf files in RDO and am saddened by this
> enlightenment.  So we had a chat in #puppet-openstack around this
> topic this morning[0] and I went looking at the -dist files and I'm
> not even sure some of the settings should even be there.  My concern
> with these files is that within configuration management tool X, it
> works on $service.conf. It unsets the configuration for foobar because
> the end user has told it to remove this line from the config file. The
> user knows what they are doing and gets the expected result. The
> packager updates $service-dist.conf to define foobar as 12345 in the
> next release. User upgrades and now $service is broken for his
> configuration because the packager has injected a value that the end
> user did not want.
>
> I found an example for heat in RDO. User can define within puppet to
> remove the log_dir from the configuration file, which should fall back
> to whatever is defaulted in python code.  But what the user gets is
> log_dir to to /var/log/heat because it's in heat-dist.conf.  This is
> not what they wanted.  This is also not the same via deb packages
> where they would get their expected configuration.  This is a real use
> case.  It would be one thing if $service-dist.conf provided only
> required parameters to start. But they seem to be inconsistent with
> what values are in there or if there's even a reason to have them.
> That being said, log_dir is a weird one but actual users have run into
> this problem and it fits nicely in my example but I think there's a
> bug upstream around this.
>
> Another example of dangerous defaults in *-dist.conf is nova and
> max_retries. The default in python is 10, RDO has -1 in
> nova-dist.git[2].   So once again, the end user might want to default
> to whats being provided via python and assumes the tooling is
> enforcing this when the packager is injecting additional
> configurations that were not expected.
>
> I would like packagers to get out of the business of defining these
> types of configuration defaults unless they truly know what they are
> doing and I don't think packagers spend enough time actually deploying
> these services in such ways to understand the implications when they
> specify these kinda of defaults for the various scenarios or know when
> these defaults change upstream.  Maybe it is my misunderstanding, but
> in puppet we've been assuming that we'd be falling back to the
> defaults provided via python for any values not defined in
> configuration files because want the upstream defaults. The only thing
> I'm looking to the packager to provide is the initial stuff around
> specific file locations for configs, pid paths, lock files, etc and
> having those defined in an overridable fashion.   Currently they are
> overridable but not undefinable unless you know they exist. So by
> default you get them if you want them or not.  I'd prefer in the case
> of these $flavor specific configurations is that they get put in in
> $service.conf not $service-dist.conf so that you're only managing one
> place with defaults.
>
> In openstack the upstream provides many defaults in python so the
> packagers for openstack services shouldn't need to provide a whole
> bunch of extra stuff.  We already have to fight debian packages for
> configuration & starting of services.  I'd rather not also have to do
> it for rpm packaging.
>
> [0] http://eavesdrop.openstack.org/irclogs/%23puppet-openstack/%23puppet-openstack.2016-10-11.log.html#t2016-10-11T13:52:06
> [1] https://github.com/rdo-packages/heat-distgit/blob/rpm-master/heat-dist.conf#L7
> [2] https://github.com/rdo-packages/nova-distgit/blob/rpm-master/nova-dist.conf#L20
>
>
>>>
>>> [0] https://github.com/openstack/puppet-horizon/blob/master/manifests/wsgi/apache.pp#L115-L128
>>> [1] https://github.com/openstack/puppet-tripleo/blob/master/manifests/ui.pp#L94-L107
>>>
>>> >
>>> > So starting nova-api would be:
>>> > nova-api --config-file /usr/share/nova/nova-dist.conf
>>> >          --config-file /etc/nova/nova.conf
>>> >          --config-dir /etc/nova/conf.d/common/
>>> >          --config-dir /etc/nova/conf.d/nova-api/
>>> >
>>>
>>> So from a puppet standpoint we actually go out of our way to default
>>> to the python defaults and we may purge all values not explicitly set
>>> via puppet. This would break our assumptions around this.  I
>>> personally do not agree with forcing the *-dist.conf into the loading
>>> path as this may also cause issues with unwanted values being injected
>>> by packagers.
>>
>> This is already happening in RDO+neutron. See [2]. So nothing new here.
>>
>
> So I'm not against the config-dir (provided they are empty by default)
> and I think it's actually beneficial.  My real issue is with
> $service-dist.conf. My personal preference would be to have the items
> in $service-dist.conf in $service.conf so that there's only one place
> to manage this.  I guess from a puppet standpoint we could just nuke
> all $service-dist.conf files but that's not ideal.  Would it be better
> to not provide $service.conf and only start with $service-dist.conf if
> $service.conf is not present? So use packager's defaults only if the
> user hasn't specified anything.
>

My preference is to have an empty $service.conf and more emphasis on
documentation of the defaults with some non-dangerous recommendations.
>From a Chef perspective and a deployer's perspective, a
$service-dist.conf is cruft that serves to confuse and conflate.

>>> The danger for people who use puppet is that we
>>> currently expected some values to not be defined in the configuration
>>> files thus falling back to the python defaults. If they now get
>>> defined in service-dist.conf and we undefine it in service.conf
>>> (because that's what we currently configure),  the end user is now
>>> going to get a value they did not expect or want.
>>
>> It's not about providing a complete config (which wouldn't work anyway)
>> but setting for some values sane defaults. See [3].
>>
>> [3]
>> https://github.com/rdo-packages/neutron-distgit/blob/rpm-master/neutron-dist.conf
>
> But are you sure they are sane?  I already mentioned the max_retries =
> -1 for nova.  I'll have to go looking at the others.  Reducing the
> amount of places that we need to manage defaults is the most ideal
> thing as we seem to have upstream manage defaults, packaging managing
> defaults, configuration management tools managing defaults, and
> deployment tools managing defaults.  From a puppet standpoint we've
> started to try and not manage any defaults and fall back to upstream
> as much as possible.  But for the most part we assume that upstream is
> python not packaging.

Chef also considers upstream to be python, at least for configuration.
For obvious reasons, upstream is also rpm and deb packaging, but for
service config defaults we defer to python.

>
>>
>>> >
>>> > The order of command line switches (--config-file/--config-dir) is
>>> > important here. Also --config-dir is ordering the files. So adding a
>>> > config snippet to /etc/nova/conf.d/nova-api/ with
>>> > e.g. [DEFAULT]bind_port would override the option from the previous
>>> > configs (last found option wins).
>>> >
>>>
>>> This gets complicated to follow and may lead to issues on the user
>>> side when ordering is a problem or attempting to debug issues. I think
>>> this can lead to more issues than it's solving.
>>
>> Most services print the complete config when running in debug mode. So
>> getting th used config is not complicated. Also adding theses switches
>> makes it more explicit when doing e.g. "ps awxu|grep nova-api" because
>> you see then what Also knowing which files/dirs are used is just "ps
>> awxu|grep $service".
>> And afaik oslo.config already loads implicit config files if they are
>> present.
>>
>
> I think this assumes knowledge of how these services run. You also
> have to consider the junior sysadmin who has no idea this is how
> openstack services work and they go into /etc/$service to find a pile
> of configurations and are not quite sure how they are all loaded.  Or
> maybe the only view into the system is their configuration management
> tool so they aren't seeing this stuff.   Like i said i'm not against
> the main service.conf and a config directory.  I think those are
> beneficial, but I'm a fan of "simple is better".  It reduces the risk
> of misconfiguration and simplifies the troubleshooting.
>
> Since RDO already uses $service-dist.conf, if you start with that it'd
> be ok but I'd really prefer some very strict policy around what goes
> in there.  Nothing more than necessary to be $flavor specific.
> Probably only paths or package name related items.  Ideally leverage
> something that allows the $service.conf to include the $flavor
> specific configuration items in place so you don't have to manage your
> own templates.  Anything else should be handled by the end user and
> not managed in packaging.
>
> Thanks,
> -Alex
>

I concur. There shouldn't be any flavor-specific items in
$service-dist.conf. I would go one farther by saying this should align
as closely to upstream python as possible, platform nuances addressed,
of course.

In Chef, we lay down a very spartan $service.conf from node attributes
from their respective cookbooks. As I alluded to above, we used to
follow a path of managing $service.conf derived from an upstream
python $service.conf.sample. Each release was spent carefully combing
through the configs, making sure each value was as it should. Today,
each $service.conf is driven from a set of node attributes that wind
up giving a 15-20 line $service.conf that can be consumed and
understood by even novice deployers whether it be a deb or rpm
packaging system under the hood. It would be unfortunate to sacrifice
that flexibility just to wind up having to manage effectively two
lines of code to achieve the same end.

Best,

Samuel

>>> Personally unless this structure is also followed by the deb
>>> packaging, I'd prefer not to switch to this as it may lead to even
>>> more fragmentation when it comes to trying to configure OpenStack
>>> deployed via packages.  Has this been requested by an end user to
>>> solve a specific problem?  What exactly is the problem that's trying
>>> to be solved other than trying to allow for two of the same project's
>>> services being configured in (currently) conflicting fashions?
>>
>> See my anwers above. It's not only about different configs for different
>> service.
>>
>> Thanks for the feedback!
>>
>> Cheers,
>>
>> Tom
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev