Open Stack

On Tue, 2017-04-04 at 18:03 +0200, Bogdan Dobrelya wrote:
> On 03.04.2017 21:01, Dan Prince wrote:
> > On Mon, 2017-04-03 at 16:15 +0200, Bogdan Dobrelya wrote:
> > > Let's please re-evaluate configuration of containerized non-
> > > openstack,
> > > like database, message queue, key value, web, services for
> > > tripleo
> > > heat
> > > templates and Kolla. Here is an example containerized etcd patch
> > > [0].
> > > 
> > > tl;dr use kolla images and bootsrap OR upstream images with
> > > direct
> > > commands:
> > > 
> > > .. code-block:: yaml
> > > kolla_config:
> > >     /var/lib/kolla/config_files/foo.json
> > >       command: /usr/bin/foo
> > > 
> > > vs
> > > 
> > > .. code-block:: yaml
> > > foo:
> > >     image: upstream/foo:latest
> > >     command: /usr/bin/foo
> > > 
> > > Note, tht already doesn't use configs [1] copied into the images
> > > by
> > > kolla build time. The next and logical step might be to omit
> > > kolla's
> > > bootstrap, where applicable, as well.
> > 
> > The kolla config file copying proved to be a bit pedantic. So we
> > removed it. A good example of this would be how this played out for
> > the
> > keystone service:
> > 
> > http://git.openstack.org/cgit/openstack/tripleo-heat-templates/comm
> > it/?
> > id=332e8ec10345ad5c8bf10a532f6f6003da682b68
> > 
> > > 
> > > There is a two options:
> > > * use kolla images and bootstrap as t-h-t does now for all
> > > services
> > > being containerized
> > 
> > We do not use KOLLA_BOOTSTRAP for all services now. Only 4 services
> > use
> > it currently I think. I think the general consensus is we should
> > not be
> > using it unless there is a functional requirement to do so.
> > Services
> > like Mysql and Rabbitmq have some extra initialization that needs
> > to
> > execute in-container before the services startup. We could
> > duplicate
> > this in tripleo-heat-templates, run it with docker-cmd perhaps and
> > do
> > it that way but we initially made exception for just a few
> > services.
> > 
> > Other services like Glance are using it, but that is just
> > historical.
> > There is already a patch to remove the use of KOLLA_BOOTSTRAP for
> > this
> > service:
> > 
> > https://review.openstack.org/#/c/440884/1
> > 
> > So in summary the consensus is we'd prefer not to be using the
> > KOLLA_BOOTSTRAP environment variables because in some cases there
> > is a
> > 'Kolla' flavor to these things that don't match how TripleO deploys
> > things.
> > 
> > It is worth pointing out that while we aren't using KOLLA_BOOTSTRAP
> > we
> > are using the kolla startup systems in many cases. This gives some
> > features around file permissions, extra sudoers files, etc. We may
> > be
> > able to stop using this for some services but I also think we are
> > getting value out of the interfaces today. They aren't nearly as
> > verbose as the Kolla config copying stuff so we could go either
> > way.
> 
> That's interesting, there is related topic with hostpath mounted
> persistent log dirs [0]. The issue is:
> 
> [tl;dr] kolla_config is a bad fit for logs, let's run containers'
> steps
> as 'user: root' in a user namespace remapped [1] for a
> tripleocontainers
> system user. Thoughts? And would that work for stock centos7/rhel7
> kernels?
> 
> 
> So, the issue is when log/data dirs created by host_prep_tasks, we
> must
> configure host permissions in a simple non-opinionated way.
> What I can see for t-h-t to cover it on its own:
> * Custom entrypoints for containers adds complexity and head ache.

Good point. But the entry points Kolla uses for many containers don't
match what our systemd services already use on baremetal. As we are
striving for update path that does not break end users upgrading from
baremetal to containers we have to have a mechanism that gives us
configuration parity across the implementions. Controlling the entry
point either by injecting it into the container (via something like
Kolla's template overrides mechanism) or via tripleo-heat-templates
direction (much more hackable) is where we ended up.

In general we like Kolla images at the moment for what they provide.
But there are some cases where we need to control things that have too
much of a "kolla flavor" and would potentially break upgrades/features
if we used them directly.

> * Mount logs world writable is a non starter.

I don't think we advocating we make log directories world writable
anywhere. One idea is that use jistr's host_prep_tasks to create
directories on the host and then log there. Another idea would be to
use 'init tasks' via docker-cmd to do the same thing.

The nice thing about either of these approaches is that it allows us to
integrate with TripleO's existing fluentd integration which uses host
logging. I think this is a nice first step in a world where we are
temporarily supporting both baremetal and containers deployment with
the same architecture. In the future perhaps we could move towards
stdout logging for services in containers and then integrate with more
standard containerized logging features.

> * Running containers as a user root is a security issue.

We don't need to do this. We run containers as the service level user
and it should be fine.

> 
> What I can see for Kolla to accomplish that task:
> * the extended start does not cover logs and there are nuances with
> recursive chowning and permissions of new files [2].
> * log directories doesn't fit well into the kolla_config's volumes
> and
> permissions defined in services' config.json. Using those for log
> volumes would look like a dirty hack. This also would undo the recent
> changes with using hostpath mounts instead of the kolla_config
> volumes.
> 
> That's why I came to the option with 'user: root' and remapped user
> namespace for docker daemon. Please share your thoughts.
> 
> [0] https://review.openstack.org/#/c/442603/
> [1] https://goo.gl/HtDQYk
> [2] https://review.openstack.org/#/c/452742/
> 
> > 
> > > pros: same way to template everything, kolla build/start just
> > > works.
> > > risks: non-openstack services, eventually, may stop being
> > > supported
> > > by
> > > Kolla for number of reasons. Kolla bootstrap changes aren't
> > > tested in
> > > tripleo CI and might be breaking

Like I mentioned above. I think the problem here is that some of the
Kolla build/start stuff is opinionated in a way that could break
TripleO upgrades. This is also why the "why can't TripleO just consume
kolla-ansible directly" debate which has occurred several times never
ended well either. In the end TripleO has some deployment areas that it
needs to control to prevent breaking these things.

> > > cons: locking in to the kolla opinionated bootstrap entry points
> > > and
> > > kolla_config's config.json and command.
> > > 
> > > * if applicable to the service, use upstream images (etcd example
> > > [2])
> > > w/o any kolla parts.
> > > pros: less moving parts like custom entry points, no locking in
> > > onto opinionated Kolla config/bootstrap
> > > risks: Upstream image changes aren't tested in tripleo CI and
> > > might
> > > be
> > > breaking
> > > cons: different ways to template openstack/non-openstack
> > > services,
> > > kolla
> > > build/start doesn't work for the latter.
> > > 
> > > [0] https://review.openstack.org/#/c/447627/
> > > [1] https://review.openstack.org/#/c/451366
> > > [2]
> > > https://review.openstack.org/#/c/445883/2/contrib/overcloud_conta
> > > iner
> > > s.yaml
> > > 
> > 
> > ___________________________________________________________________
> > _______
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsu
> > bscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 
>