Open Stack

Hi,

Kolla-ansible went through this process a few years ago, and ended up with
a solution involving heka pulling logs from files in a shared docker volume
(kolla_logs). Heka was recently switched for fluentd due to the
disappearance of upstream support. I suspect kolla-kubernetes has been
through a similar process recently.

I'd recommend soliciting input from the kolla team on the factors that lead
to the current design, and whether with hindsight anything would be done
differently.

Cheers,
Marl

On 19 July 2017 at 09:26, Bogdan Dobrelya <bdobreli at redhat.com> wrote:

> On 18.07.2017 21:27, Lars Kellogg-Stedman wrote:
> > Our current model for logging in a containerized deployment has pretty
> > much everything logging to files in a directory that has been
> > bind-mounted from the host.  This has some advantages: primarily, it
> > makes it easy for an operator on the local system to find logs,
> > particularly if they have had some previous exposure to
> > non-containerized deployments.
> >
> > There is strong demand for a centralized logging solution.  We've got
> > one potential solution right now in the form of the fluentd service
> > introduced in Newton, but this requires explicit registration of log
> > files for every service.  I don't think it's an ideal solution, and I
> > would like to explore some alternatives.
> >
> > Logging via syslog
> > ==============
> >
> > For the purposes of the following, I'm going to assume that we're
> > deploying on an EL-variant (RHEL/CentOS/etc), which means (a) journald
> > owns /dev/log and (b) we're running rsyslog on the host and using the
> > omjournal plugin to read messages from journald.
> >
> > If we bind mount /dev/log into containers and configure openstack
> > services to log via syslog rather than via files, we get the following
> > for free:
> >
> > - We get message-based rather than line-based logging.  This means that
> > multiline tracebacks are handled correctly.
> >
> > - A single point of collection for logs.  If your host has been
> > configured to ship logs to a centralized collector, logs from all of
> > your services will be sent there without any additional configuration.
> >
> > - We get per-service message rate limiting from journald.
> >
> > - Log messages are annotated by journald with a variety of useful
> > metadata, including the container id and a high resolution timestamp.
> >
> > - We can configure the syslog service on the host to continue to write
> > files into legacy locations, so an operator looking to run grep against
> > local log files will still have that ability.
> >
> > - Ryslog itself can send structured messages directly to an Elastic
> > instance, which means that in a many deployments we would not require
> > fluentd and its dependencies.
> >
> > - This plays well in environments where some services are running in
> > containers and others are running on the host, because everything simply
> > logs to /dev/log.
>
> Plus it solves the log rotation (still have to be addressed [0] for Pike
> though), out-of-box.
>
> >
> > Logging via stdin/stdout
> > ==================
> >
> > A common pattern in the container world is to log everything to
> > stdout/stderr.  This has some of the advantages of the above:
> >
> > - We can configure the container orchestration service to send logs to
> > the journal or to another collector.
> >
> > - We get a different set of annotations on log messages.
> >
> > - This solution may play better with frameworks like Kubernetes that
> > tend to isolate containers from the host a little more than using Docker
> > or similar tools straight out of the box.
> >
> > But there are some disadvantages:
> >
> > - Some services only know how to log via syslog (e.g., swift and haproxy)
> >
> > - We're back to line-based vs. message-based logging.
> >
> > - It ends up being more difficult to expose logs at legacy locations.
> >
> > - The container orchestration layer may not implement the same message
> > rate limiting we get with fluentd.
> >
> > Based on the above, I would like to suggest exploring a syslog-based
> > logging model moving forward. What do people think about this idea? I've
> > started putting together a spec
> > at https://review.openstack.org/#/c/484922/ and I would welcome your
> input.
>
> My vote goes for this option, but TBD for Queens. It won't make it for
> Pike, as it looks too late for such amount of drastic changes, like
> switching all OpenStack services to syslog, deploying additional
> required components, and so on.
>
> [0] https://bugs.launchpad.net/tripleo/+bug/1700912
> [1] https://review.openstack.org/#/c/462900/
>
> >
> > Cheers,
> >
> > --
> > Lars Kellogg-Stedman <lars at redhat.com <mailto:lars at redhat.com>>
> >
> >
> >
> > ____________________________________________________________
> ______________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
> unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> --
> Best regards,
> Bogdan Dobrelya,
> Irc #bogdando
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170719/12fa9b1c/attachment.html>