Open Stack

I'm not arguing that everything should be managed by one systemd, I'm just saying, for certain types of containers, a single docker container with systemd in it might be preferable to trying to slice it unnaturally into several containers.

Systemd has invested a lot of time/effort to be able to relaunch failed services, support spawning and maintaining unix sockets and services across them, etc, that you'd have to push out of and across docker containers. All of that can be done, but why reinvent the wheel? Like you said, pacemaker can be made to make it all work, but I have yet to see a way to deploy pacemaker services anywhere near as easy as systemd+yum makes it. (Thanks be to redhat. :)

The answer seems to be, its not "dockerish". Thats ok. I just wanted to understand the issue for what it is. If there is a really good reason for not wanting to do it, or that its just "not the way things are done". I've had kind of the opposite feeling regarding docker containers. Docker use to do very bad things when killing the container. nasty if you wanted your database not to go corrupt. killing pid 1 is a bit sketchy then forcing the container down after 10 seconds was particularly bad. having something like systemd in place allows the database to be notified, then shutdown properly. Sure you can script up enough shell to make this work, but you have to do some difficult code, over and over again... Docker has gotten better more recently but it still makes me a bit nervous using it for statefull things.

As for recovery, systemd can do the recovery too. I'd argue at this point in time, I'd expect systemd recovery to probably work better then some custom shell scripts when it comes to doing the right thing recovering at bring up. The other thing is, recovery is not just about pid1 going away. often it sticks around and other badness is going on. Its A way to know things are bad, but you can't necessarily rely on it to know the container's healty. You need more robust checks for that.

Thanks,
Kevin

________________________________________
From: David Vossel [dvossel at redhat.com]
Sent: Tuesday, October 14, 2014 4:52 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns

----- Original Message -----
> Ok, why are you so down on running systemd in a container?

It goes against the grain.

>From a distributed systems view, we gain quite a bit of control by maintaining
"one service per container". Containers can be re-organised and re-purposed dynamically.
If we have systemd trying to manage an entire stack of resources within a container,
we lose this control.

>From my perspective a containerized application stack needs to be managed externally
by whatever is orchestrating the containers to begin with. When we take a step back
and look at how we actually want to deploy containers, systemd doesn't make much sense.
It actually limits us in the long run.

Also... recovery. Using systemd to manage a stack of resources within a single container
makes it difficult for whatever is externally enforcing the availability of that container
to detect the health of the container.  As it is now, the actual service is pid 1 of a
container. If that service dies, the container dies. If systemd is pid 1, there can
be all kinds of chaos occurring within the container, but the external distributed
orchestration system won't have a clue (unless it invokes some custom health monitoring
tools within the container itself, which will likely be the case someday.)

-- Vossel


> Pacemaker works, but its kind of a pain to setup compared just yum installing
> a few packages and setting init to systemd. There are some benefits for
> sure, but if you have to force all the docker components onto the same
> physical machine anyway, why bother with the extra complexity?
>
> Thanks,
> Kevin
>
> ________________________________________
> From: David Vossel [dvossel at redhat.com]
> Sent: Tuesday, October 14, 2014 3:14 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns
>
> ----- Original Message -----
> > Same thing works with cloud init too...
> >
> >
> > I've been waiting on systemd working inside a container for a while. it
> > seems
> > to work now.
>
> oh no...
>
> > The idea being its hard to write a shell script to get everything up and
> > running with all the interactions that may need to happen. The init
> > system's
> > already designed for that. Take a nova-compute docker container for
> > example,
> > you probably need nova-compute, libvirt, neutron-openvswitch-agent, and the
> > celiometer-agent all backed in. Writing a shell script to get it all
> > started
> > and shut down properly would be really ugly.
> >
> > You could split it up into 4 containers and try and ensure they are
> > coscheduled and all the pieces are able to talk to each other, but why?
> > Putting them all in one container with systemd starting the subprocesses is
> > much easier and shouldn't have many drawbacks. The components code is
> > designed and tested assuming the pieces are all together.
>
> What you need is a dependency model that is enforced outside of the
> containers. Something
> that manages the order containers are started/stopped/recovered in. This
> allows
> you to isolate your containers with 1 service per container, yet still
> express that
> container with service A needs to start before container with service B.
>
> Pacemaker does this easily. There's even a docker resource-agent for
> Pacemaker now.
> https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/docker
>
> -- Vossel
>
> ps. don't run systemd in a container... If you think you should, talk to me
> first.
>
> >
> > You can even add a ssh server in there easily too and then ansible in to do
> > whatever other stuff you want to do to the container like add other
> > monitoring and such....
> >
> > Ansible or puppet or whatever should work better in this arrangement too
> > since existing code assumes you can just systemctl start foo;
> >
> > Kevin
> > ________________________________________
> > From: Lars Kellogg-Stedman [lars at redhat.com]
> > Sent: Tuesday, October 14, 2014 12:10 PM
> > To: OpenStack Development Mailing List (not for usage questions)
> > Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns
> >
> > On Tue, Oct 14, 2014 at 02:45:30PM -0400, Jay Pipes wrote:
> > > With Docker, you are limited to the operating system of whatever the
> > > image
> > > uses.
> >
> > See, that's the part I disagree with.  What I was saying about ansible
> > and puppet in my email is that I think the right thing to do is take
> > advantage of those tools:
> >
> >   FROM ubuntu
> >
> >   RUN apt-get install ansible
> >   COPY my_ansible_config.yaml /my_ansible_config.yaml
> >   RUN ansible /my_ansible_config.yaml
> >
> > Or:
> >
> >   FROM Fedora
> >
> >   RUN yum install ansible
> >   COPY my_ansible_config.yaml /my_ansible_config.yaml
> >   RUN ansible /my_ansible_config.yaml
> >
> > Put the minimal instructions in your dockerfile to bootstrap your
> > preferred configuration management tool. This is exactly what you
> > would do when booting, say, a Nova instance into an openstack
> > environment: you can provide a shell script to cloud-init that would
> > install whatever packages are required to run your config management
> > tool, and then run that tool.
> >
> > Once you have bootstrapped your cm environment you can take advantage
> > of all those distribution-agnostic cm tools.
> >
> > In other words, using docker is no more limiting than using a vm or
> > bare hardware that has been installed with your distribution of
> > choice.
> >
> > > [1] Is there an official MySQL docker image? I found 553 Dockerhub
> > > repositories for MySQL images...
> >
> > Yes, it's called "mysql".  It is in fact one of the official images
> > highlighted on https://registry.hub.docker.com/.
> >
> > > >I have looked into using Puppet as part of both the build and runtime
> > > >configuration process, but I haven't spent much time on it yet.
> > >
> > > Oh, I don't think Puppet is any better than Ansible for these things.
> >
> > I think it's pretty clear that I was not suggesting it was better than
> > ansible.  That is hardly relevant to this discussion.  I was only
> > saying that is what *I* have looked at, and I was agreeing that *any*
> > configuration management system is probably better than writing shells
> > cript.
> >
> > > How would I go about essentially transferring the ownership of the RPC
> > > exchanges that the original nova-conductor container managed to the new
> > > nova-conductor container? Would it be as simple as shutting down the old
> > > container and starting up the new nova-conductor container using things
> > > like
> > > --link rabbitmq:rabbitmq in the startup docker line?
> >
> > I think that you would not necessarily rely on --link for this sort of
> > thing.  Under kubernetes, you would use a "service" definition, in
> > which kubernetes maintains a proxy that directs traffic to the
> > appropriate place as containers are created and destroyed.
> >
> > Outside of kubernetes, you would use some other service discovery
> > mechanism; there are many available (etcd, consul, serf, etc).
> >
> > But this isn't particularly a docker problem.  This is the same
> > problem you would face running the same software on top of a cloud
> > environment in which you cannot predict things like ip addresses a
> > priori.
> >
> > --
> > Lars Kellogg-Stedman <lars at redhat.com> | larsks @ {freenode,twitter,github}
> > Cloud Engineering / OpenStack          | http://blog.oddbit.com/
> >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev