Open Stack

Our current model for logging in a containerized deployment has pretty much
everything logging to files in a directory that has been bind-mounted from
the host.  This has some advantages: primarily, it makes it easy for an
operator on the local system to find logs, particularly if they have had
some previous exposure to non-containerized deployments.

There is strong demand for a centralized logging solution.  We've got one
potential solution right now in the form of the fluentd service introduced
in Newton, but this requires explicit registration of log files for every
service.  I don't think it's an ideal solution, and I would like to explore
some alternatives.

Logging via syslog
==============

For the purposes of the following, I'm going to assume that we're deploying
on an EL-variant (RHEL/CentOS/etc), which means (a) journald owns /dev/log
and (b) we're running rsyslog on the host and using the omjournal plugin to
read messages from journald.

If we bind mount /dev/log into containers and configure openstack services
to log via syslog rather than via files, we get the following for free:

- We get message-based rather than line-based logging.  This means that
multiline tracebacks are handled correctly.

- A single point of collection for logs.  If your host has been configured
to ship logs to a centralized collector, logs from all of your services
will be sent there without any additional configuration.

- We get per-service message rate limiting from journald.

- Log messages are annotated by journald with a variety of useful metadata,
including the container id and a high resolution timestamp.

- We can configure the syslog service on the host to continue to write
files into legacy locations, so an operator looking to run grep against
local log files will still have that ability.

- Ryslog itself can send structured messages directly to an Elastic
instance, which means that in a many deployments we would not require
fluentd and its dependencies.

- This plays well in environments where some services are running in
containers and others are running on the host, because everything simply
logs to /dev/log.

Logging via stdin/stdout
==================

A common pattern in the container world is to log everything to
stdout/stderr.  This has some of the advantages of the above:

- We can configure the container orchestration service to send logs to the
journal or to another collector.

- We get a different set of annotations on log messages.

- This solution may play better with frameworks like Kubernetes that tend
to isolate containers from the host a little more than using Docker or
similar tools straight out of the box.

But there are some disadvantages:

- Some services only know how to log via syslog (e.g., swift and haproxy)

- We're back to line-based vs. message-based logging.

- It ends up being more difficult to expose logs at legacy locations.

- The container orchestration layer may not implement the same message rate
limiting we get with fluentd.

Based on the above, I would like to suggest exploring a syslog-based
logging model moving forward. What do people think about this idea? I've
started putting together a spec at https://review.openstack.org/#/c/484922/
and I would welcome your input.

Cheers,

-- 
Lars Kellogg-Stedman <lars at redhat.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170718/e2ee5668/attachment.html>