Open Stack

Flavio: I think your right. k8s configmaps and confd are doing very similar things. The one thing confd seems to add is dynamic templates on the host side. This is still accomplished in k8s with a sidecar watching for config changes with the templating engine in it and an emptyDir. or statically with an init container and an emptyDir (kolla-kubernetes does the latter)

But, for k8s, I actually prefer a fully atomic container config model, where you do a rolling upgrade any time you want to do a configmap change. k8s gives you the plumbing to do that and you can more easily roll forward/backward, allowing you versioning too.

So, I think your right. etcd/confd is more suited to the non k8s deployments.

Thanks,
Kevin
________________________________
From: Flavio Percoco [flavio at redhat.com]
Sent: Thursday, June 08, 2017 3:28 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd



On Thu, Jun 8, 2017, 19:14 Doug Hellmann <doug at doughellmann.com<mailto:doug at doughellmann.com>> wrote:
Excerpts from Flavio Percoco's message of 2017-06-08 18:27:51 +0200:
> On 08/06/17 18:23 +0200, Flavio Percoco wrote:
> >On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:
> >>On 06.06.2017 18:08, Emilien Macchi wrote:
> >>>Another benefit is that confd will generate a configuration file when
> >>>the application will start. So if etcd is down *after* the app
> >>>startup, it shouldn't break the service restart if we don't ask confd
> >>>to re-generate the config. It's good for operators who were concerned
> >>>about the fact the infrastructure would rely on etcd. In that case, we
> >>>would only need etcd at the initial deployment (and during lifecycle
> >>>actions like upgrades, etc).
> >>>
> >>>The downside is that in the case of containers, they would still have
> >>>a configuration file within the container, and the whole goal of this
> >>>feature was to externalize configuration data and stop having
> >>>configuration files.
> >>
> >>It doesn't look a strict requirement. Those configs may (and should) be
> >>bind-mounted into containers, as hostpath volumes. Or, am I missing
> >>something what *does* make embedded configs a strict requirement?..
> >
> >mmh, one thing I liked about this effort was possibility of stop bind-mounting
> >config files into the containers. I'd rather find a way to not need any
> >bindmount and have the services get their configs themselves.
>
> Probably sent too early!
>
> If we're not talking about OpenStack containers running in a COE, I guess this
> is fine. For k8s based deployments, I think I'd prefer having installers
> creating configmaps directly and use that. The reason is that depending on files
> that are in the host is not ideal for these scenarios. I hate this idea because
> it makes deployments inconsistent and I don't want that.
>
> Flavio
>

I'm not sure I understand how a configmap is any different from what is
proposed with confd in terms of deployment-specific data being added to
a container before it launches. Can you elaborate on that?


Unless I'm missing something, to use confd with an OpenStack deployment on k8s, we'll have to do something like this:

* Deploy confd in every node where we may want to run a pod (basically wvery node)
* Configure it to download all configs from etcd locally (we won't be able to download just some of them because we don't know what services may run in specific nodes. Except, perhaps, in the case of compute nodes and some other similar nodes)
* Enable hostpath volumes (iirc it's disabled by default) so that we can mount these files in the pod
* Run the pods and mount the files assuming the files are there.

All of the above is needed because  confd syncs files locally from etcd. Having a centralized place to manage these configs allows for controlling the deployment better. For example, if a configmap doesn't exist, then stop everything.

Not trying to be negative but rather explain why I think confd may not work well for the k8s based deployments. I think it's a good fit for the rest of the deployments.

Am I missing something? Am I overcomplicating things?

Flavio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170609/9c43a341/attachment.html>