Open Stack

Jeffrey,

Logstash-forwarder is deprecated upstream, so we can’t rely on that.  Elastic's replacement is filebeat.

I’m not sure which one meets the requirements – filebeat or fluentd.  In kolla-kubernetes fluentd is being used, and is well maintained.  Both implementations are pretty green IMO.  Not sure if fluentd also does log processing.  I think its crucial to pick a component that just does log forwarding since that is the part that was deprecated.

Our system has no log stash at all in it, and I’d like to keep it that way.  Logstash is unnecessary for our use case.  What we want is forwarder->es->cabana.  Whatever forwarder is chosen, recommend picking the best of the two choices.  I’d start with defining best as “does it solve the same problem as Heka does in our current implementation” then sprinkle throughput and minimal cpu and network utilization on top.  If we can’t make a decision from there, not sure I have any further suggestions as I am not writing the code.

Regards
-steve


From: Jeffrey Zhang <zhang.lei.fly at gmail.com<mailto:zhang.lei.fly at gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Sunday, November 27, 2016 at 9:40 AM
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: Re: [openstack-dev] [kolla] the alternative of log processing tool

So filebeat is working with Logstash right? We need split the logs into pieces by using logstash. IMU, Filebeat do not a variety of processing plugins, like Logstash[0].

[0] https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

On Sun, Nov 27, 2016 at 11:30 PM, Ian Cordasco <sigmavirus24 at gmail.com<mailto:sigmavirus24 at gmail.com>> wrote:

File beat is maintained be elastic and a part of their product line just like ELK. It's a fantastic tool and quite flexible given its age and size of codebase

On Nov 26, 2016 11:59 PM, "Jeffrey Zhang" <zhang.lei.fly at gmail.com<mailto:zhang.lei.fly at gmail.com>> wrote:
Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we have a
blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat.

For Filebeat, it is just a replacement of logstash-forward[2]. It is not intent
to replace the Logstash at all.

> Filebeat is based on the Logstash Forwarder source code and replaces Logstash
> Forwarder as the method to use for tailing log files and forwarding them to
> Logstash.

Fillebeat is a log transport tool rather than log processing too. I do not
treat it as an alternative at all.

To be honest, I'd like back to Logstash, and Logstash 5.x is released with high
performance improvement[4].

>  In our performance testing, we've seen consistent throughput increases
>  across multiple configurations. In some cases, we observed up to 75%
>  increase in events processed through Logstash.

another benefit to using Logstash is the whole ELK stack is maintained by one
community/company. It is well tested and easy to upgrade the whole stack at the
same time. Using other tools may force us on certain elasticsearch release.

So, I think we have to alternative tools.

* Fluentd
* Logstash

IMO, we need to make the decision and at least prepare the migration solution now.

[1] https://blueprints.launchpad.net/kolla/+spec/heka-deprecation
[2] https://www.elastic.co/guide/en/beats/filebeat/current/migrating-from-logstash-forwarder.html
[3] http://www.fluentd.org/
[4] https://www.elastic.co/blog/logstash-5-0-0-released

--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me<http://xcodest.me/>

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me<http://xcodest.me/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161127/78a50a25/attachment.html>