<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I would vote for fluentd because: 1 – it has been around since 2011 so it is hard to call it green. 2 – There is constant development of new features/filters,
3 – As it was mentioned at Kubecon by fluentd people, they are deeply committed to Open Source community so rumors that they would go private does not sound reliable. We use it in kolla kubernetes and so far we could share only positive feedback.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Serguei<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></a></p>
<span style="mso-bookmark:_MailEndCompose"></span>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Steven Dake (stdake)
<br>
<b>Sent:</b> Sunday, November 27, 2016 3:46 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org><br>
<b>Subject:</b> Re: [openstack-dev] [kolla] the alternative of log processing tool<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">Jeffrey,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">Logstash-forwarder is deprecated upstream, so we can’t rely on that. Elastic's replacement is filebeat.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">I’m not sure which one meets the requirements – filebeat or fluentd. In kolla-kubernetes fluentd is being used, and is well maintained. Both implementations are
pretty green IMO. Not sure if fluentd also does log processing. I think its crucial to pick a component that just does log forwarding since that is the part that was deprecated.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">Our system has no log stash at all in it, and I’d like to keep it that way. Logstash is unnecessary for our use case. What we want is forwarder->es->cabana. Whatever
forwarder is chosen, recommend picking the best of the two choices. I’d start with defining best as “does it solve the same problem as Heka does in our current implementation” then sprinkle throughput and minimal cpu and network utilization on top. If we
can’t make a decision from there, not sure I have any further suggestions as I am not writing the code.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">Regards<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">-steve<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:"Calibri",sans-serif;color:black">From:
</span></b><span style="font-family:"Calibri",sans-serif;color:black">Jeffrey Zhang <<a href="mailto:zhang.lei.fly@gmail.com">zhang.lei.fly@gmail.com</a>><br>
<b>Reply-To: </b>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<b>Date: </b>Sunday, November 27, 2016 at 9:40 AM<br>
<b>To: </b>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<b>Subject: </b>Re: [openstack-dev] [kolla] the alternative of log processing tool<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">So filebeat is working with Logstash right? We need split the logs into pieces by using logstash. IMU, Filebeat do not a variety of processing plugins, like Logstash[0].<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">[0] <a href="https://www.elastic.co/guide/en/logstash/current/filter-plugins.html">
https://www.elastic.co/guide/en/logstash/current/filter-plugins.html</a> <o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">On Sun, Nov 27, 2016 at 11:30 PM, Ian Cordasco <<a href="mailto:sigmavirus24@gmail.com" target="_blank">sigmavirus24@gmail.com</a>> wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">File beat is maintained be elastic and a part of their product line just like ELK. It's a fantastic tool and quite flexible given its age and size of codebase<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">On Nov 26, 2016 11:59 PM, "Jeffrey Zhang" <<a href="mailto:zhang.lei.fly@gmail.com" target="_blank">zhang.lei.fly@gmail.com</a>> wrote:<o:p></o:p></span></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we have a</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat. </span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">For Filebeat, it is just a replacement of logstash-forward[2]. It is not intent</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">to replace the Logstash at all.</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">> Filebeat is based on the Logstash Forwarder source code and replaces Logstash</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">> Forwarder as the method to use for tailing log files and forwarding them to</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">> Logstash.</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">Fillebeat is a log transport tool rather than log processing too. I do not</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">treat it as an alternative at all.</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">To be honest, I'd like back to Logstash, and Logstash 5.x is released with high</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">performance improvement[4].</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">> In our performance testing, we've seen consistent throughput increases</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">> across multiple configurations. In some cases, we observed up to 75%</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">> increase in events processed through Logstash.</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">another benefit to using Logstash is the whole ELK stack is maintained by one</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">community/company. It is well tested and easy to upgrade the whole stack at the</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">same time. Using other tools may force us on certain elasticsearch release.</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">So, I think we have to alternative tools.</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">* Fluentd</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">* Logstash</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">IMO, we need to make the decision and at least prepare the migration solution now.</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">[1]
<a href="https://blueprints.launchpad.net/kolla/+spec/heka-deprecation" target="_blank">
https://blueprints.launchpad.net/kolla/+spec/heka-deprecation</a></span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">[2]
<a href="https://www.elastic.co/guide/en/beats/filebeat/current/migrating-from-logstash-forwarder.html" target="_blank">
https://www.elastic.co/guide/en/beats/filebeat/current/migrating-from-logstash-forwarder.html</a></span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">[3]
<a href="http://www.fluentd.org/" target="_blank">http://www.fluentd.org/</a></span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New";color:black">[4]
<a href="https://www.elastic.co/blog/logstash-5-0-0-released" target="_blank">https://www.elastic.co/blog/logstash-5-0-0-released</a></span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">--
<o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Regards,</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Jeffrey Zhang</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New";color:black">Blog: </span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><a href="http://xcodest.me/" target="_blank"><span style="font-size:9.5pt;font-family:"Courier New"">http://xcodest.me</span></a><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></span></p>
</blockquote>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></span></p>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><br>
<br clear="all">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">--
<o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Regards,</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Jeffrey Zhang</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Courier New";color:black">Blog: </span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><a href="http://xcodest.me/" target="_blank"><span style="font-size:9.5pt;font-family:"Courier New"">http://xcodest.me</span></a><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>