<div dir="ltr">I like the idea of using Heka. You and I have discussed this on IRC before. So my vote for this is +1. I can't think of any downside. I would like to hear Alicja Kwasniewska's view on this as she has done the majority of work with Logstash up until this point.</div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div>Sam Yaple</div></div></div></div>
<br><div class="gmail_quote">On Mon, Jan 11, 2016 at 3:16 PM, Eric LEMOINE <span dir="ltr"><<a href="mailto:elemoine@mirantis.com" target="_blank">elemoine@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi<br>
<br>
As discussed on IRC the other day [1] we want to propose a distributed<br>
logs processing architecture based on Heka [2], built on Alicja<br>
Kwasniewska's ELK work with<br>
<<a href="https://review.openstack.org/#/c/252968/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/252968/</a>>. Please take a look at the<br>
design document I've started working on [3]. The document is still<br>
work-in-progress, but the "Problem statement" and "Proposed change"<br>
sections should provide you with a good overview of the architecture<br>
we have in mind.<br>
<br>
In the proposed architecture each cluster node runs an instance of<br>
Heka for collecting and processing logs. And instead of sending the<br>
processed logs to a centralized Logstash instance, logs are directly<br>
sent to Elasticsearch, which itself can be distributed across multiple<br>
nodes for high-availability and scaling. The proposed architecture is<br>
based on Heka, and it doesn't use Logstash.<br>
<br>
That being said, it is important to note that the intent of this<br>
proposal is not strictly directed at replacing Logstash by Heka. The<br>
intent is to propose a distributed architecture with Heka running on<br>
each cluster node rather than having Logstash run as a centralized<br>
logs processing component. For such a distributed architecture we<br>
think that Heka is more appropriate, with a smaller memory footprint<br>
and better performances in general. In addition, Heka is also more<br>
than a logs processing tool, as it's designed to process streams of<br>
any type of data, including events, logs and metrics.<br>
<br>
Some elements of comparison between Heka and Logstash:<br>
<br>
* Logstash was designed for logs processing. Heka is a "unified data<br>
processing" software, designed to process streams of any type of data.<br>
So Heka is about running one service on each box instead of many.<br>
Using a single service for processing different types of data also<br>
makes it possible to do correlations, and derive metrics from logs and<br>
events. See Rob Miller's presentation [4] for more details.<br>
<br>
* The virtual size of the Logstash Docker image is 447 MB, while the<br>
virtual size of an Heka image built from the same base image<br>
(debian:jessie) is 177 MB. For comparison the virtual size of the<br>
Elasticsearch image is 345 MB.<br>
<br>
* Heka is written in Go and has no dependencies. Go programs are<br>
compiled to native code. This in contrast to Logstash which uses<br>
JRuby and as such requires running a Java Virtual Machine. Besides<br>
this native versus interpreted code aspect, this also can raise the<br>
question of which JVM to use (Oracle, OpenJDK?) and which version<br>
(6,7,8?).<br>
<br>
* There are six types of Heka plugins: Inputs, Splitters, Decoders,<br>
Filters, Encoders, and Outputs. Heka plugins are written in Go or<br>
Lua. When written in Lua their executions are sandbox'ed, where<br>
misbehaving plugins may be shut down by Heka. Lua plugins may also be<br>
dynamically added to Heka with no config changes or Heka restart. This<br>
is an important property on container environments such as Mesos,<br>
where workloads are changed dynamically.<br>
<br>
* To avoid losing logs under high load it is often recommend to use<br>
Logstash together with Redis [5]. Redis plays the role of a buffer,<br>
where logs are queued when Logstash or Elasticsearch cannot keep up<br>
with the load. Heka, as a "unified data processing" software,<br>
includes its own resilient message queue, making it unnecessary to use<br>
an external queue (Redis for example).<br>
<br>
* Heka is faster than Logstash for processing logs, and its memory<br>
footprint is smaller. I ran tests, where 3,400,000 log messages were<br>
read from 500 input files and then written to a single output file.<br>
Heka processed the 3,400,000 log messages in 12 seconds, consuming<br>
500M of RAM. Logstash processed the 3,400,000 log messages in 1mn<br>
35s, consuming 1.1G of RAM. Adding a grok filter to parse and<br>
structure logs, Logstash processed the 3,400,000 log messages in 2mn<br>
15s, consuming 1.5G of RAM. Using an equivalent filtering plugin, Heka<br>
processed the 3,400,000 log messages in 27s, consuming 730M of RAM.<br>
See my GitHub repo [6] for more information about the test<br>
environment.<br>
<br>
Also, I want to say that our team has been using Heka in production<br>
for about a year, in clusters of up to 200 nodes. Heka has proven to<br>
be very robust, efficient and flexible enough to address our logs<br>
processing and monitoring use-cases. We've also acquired a solid<br>
experience with it.<br>
<br>
Any comments are welcome!<br>
<br>
Thanks.<br>
<br>
<br>
[1] <<a href="http://eavesdrop.openstack.org/meetings/kolla/2016/kolla.2016-01-06-16.32.html" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/meetings/kolla/2016/kolla.2016-01-06-16.32.html</a>><br>
[2] <<a href="http://hekad.readthedocs.org" rel="noreferrer" target="_blank">http://hekad.readthedocs.org</a>><br>
[3] <<a href="https://docs.google.com/document/d/1RdckXedts4THPb6giAZvoy3ESiJ5GXau3PYIgbGR-fA/edit?usp=sharing" rel="noreferrer" target="_blank">https://docs.google.com/document/d/1RdckXedts4THPb6giAZvoy3ESiJ5GXau3PYIgbGR-fA/edit?usp=sharing</a>><br>
[4] <<a href="http://www.slideshare.net/devopsdays/heka-rob-miller" rel="noreferrer" target="_blank">http://www.slideshare.net/devopsdays/heka-rob-miller</a>><br>
[5] <<a href="http://blog.sematext.com/2015/09/28/recipe-rsyslog-redis-logstash/" rel="noreferrer" target="_blank">http://blog.sematext.com/2015/09/28/recipe-rsyslog-redis-logstash/</a>><br>
[6] <<a href="https://github.com/elemoine/heka-logstash-comparison" rel="noreferrer" target="_blank">https://github.com/elemoine/heka-logstash-comparison</a>><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><br></div>