Open Stack

FYI, in kolla-kubernes, I've been playing with fluent-bit as a log shipper. Works very similar to fluentd but is much lighter weight. I used this: https://github.com/kubernetes/charts/tree/master/stable/fluent-bit

I fought with getting log rolling working properly with log files and its kind of a pain. A lot of things that can go wrong.

I ended up getting the following to work pretty well:
1. configure docker to roll its own log files based on size.
2. switch containers to use stderror/stdout instead of log files.
3. use fluent-bit to follow docker logs, add k8s pod info and ship to central server.

Thanks,
Kevin
________________________________________
From: Bogdan Dobrelya [bdobreli at redhat.com]
Sent: Wednesday, July 19, 2017 1:26 AM
To: openstack-dev at lists.openstack.org
Subject: Re: [openstack-dev] [tripleo] Logging in containerized services

On 18.07.2017 21:27, Lars Kellogg-Stedman wrote:
> Our current model for logging in a containerized deployment has pretty
> much everything logging to files in a directory that has been
> bind-mounted from the host.  This has some advantages: primarily, it
> makes it easy for an operator on the local system to find logs,
> particularly if they have had some previous exposure to
> non-containerized deployments.
>
> There is strong demand for a centralized logging solution.  We've got
> one potential solution right now in the form of the fluentd service
> introduced in Newton, but this requires explicit registration of log
> files for every service.  I don't think it's an ideal solution, and I
> would like to explore some alternatives.
>
> Logging via syslog
> ==============
>
> For the purposes of the following, I'm going to assume that we're
> deploying on an EL-variant (RHEL/CentOS/etc), which means (a) journald
> owns /dev/log and (b) we're running rsyslog on the host and using the
> omjournal plugin to read messages from journald.
>
> If we bind mount /dev/log into containers and configure openstack
> services to log via syslog rather than via files, we get the following
> for free:
>
> - We get message-based rather than line-based logging.  This means that
> multiline tracebacks are handled correctly.
>
> - A single point of collection for logs.  If your host has been
> configured to ship logs to a centralized collector, logs from all of
> your services will be sent there without any additional configuration.
>
> - We get per-service message rate limiting from journald.
>
> - Log messages are annotated by journald with a variety of useful
> metadata, including the container id and a high resolution timestamp.
>
> - We can configure the syslog service on the host to continue to write
> files into legacy locations, so an operator looking to run grep against
> local log files will still have that ability.
>
> - Ryslog itself can send structured messages directly to an Elastic
> instance, which means that in a many deployments we would not require
> fluentd and its dependencies.
>
> - This plays well in environments where some services are running in
> containers and others are running on the host, because everything simply
> logs to /dev/log.

Plus it solves the log rotation (still have to be addressed [0] for Pike
though), out-of-box.

>
> Logging via stdin/stdout
> ==================
>
> A common pattern in the container world is to log everything to
> stdout/stderr.  This has some of the advantages of the above:
>
> - We can configure the container orchestration service to send logs to
> the journal or to another collector.
>
> - We get a different set of annotations on log messages.
>
> - This solution may play better with frameworks like Kubernetes that
> tend to isolate containers from the host a little more than using Docker
> or similar tools straight out of the box.
>
> But there are some disadvantages:
>
> - Some services only know how to log via syslog (e.g., swift and haproxy)
>
> - We're back to line-based vs. message-based logging.
>
> - It ends up being more difficult to expose logs at legacy locations.
>
> - The container orchestration layer may not implement the same message
> rate limiting we get with fluentd.
>
> Based on the above, I would like to suggest exploring a syslog-based
> logging model moving forward. What do people think about this idea? I've
> started putting together a spec
> at https://review.openstack.org/#/c/484922/ and I would welcome your input.

My vote goes for this option, but TBD for Queens. It won't make it for
Pike, as it looks too late for such amount of drastic changes, like
switching all OpenStack services to syslog, deploying additional
required components, and so on.

[0] https://bugs.launchpad.net/tripleo/+bug/1700912
[1] https://review.openstack.org/#/c/462900/

>
> Cheers,
>
> --
> Lars Kellogg-Stedman <lars at redhat.com <mailto:lars at redhat.com>>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


--
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev