Open Stack

Logstash-forward/Filebeat just cut logs in preparation for processing
elsewhere. It doesn't process logs just forward it to another processor (
Logstash / Heka / Fluentd ). It do not have any processing filter like
Logstash. At least, we need some thing tool like grok, syslog intput etc.

what we need is:

* listen on syslog like socket to collect logs
* processing plugin, like logstash grok does.

I do not think fielbeat meet this requirement. So finally, we need

<service> -> filebeat ( maybe, log forward ) -> Logstash/heka/Fluentd ( log
processing ) -> ES ( log storage ) -> grafana ( log ui )



On Mon, Nov 28, 2016 at 4:45 AM, Steven Dake (stdake) <stdake at cisco.com>
wrote:

> Jeffrey,
>
> Logstash-forwarder is deprecated upstream, so we can’t rely on that.
> Elastic's replacement is filebeat.
>
> I’m not sure which one meets the requirements – filebeat or fluentd.  In
> kolla-kubernetes fluentd is being used, and is well maintained.  Both
> implementations are pretty green IMO.  Not sure if fluentd also does log
> processing.  I think its crucial to pick a component that just does log
> forwarding since that is the part that was deprecated.
>
> Our system has no log stash at all in it, and I’d like to keep it that
> way.  Logstash is unnecessary for our use case.  What we want is
> forwarder->es->cabana.  Whatever forwarder is chosen, recommend picking the
> best of the two choices.  I’d start with defining best as “does it solve
> the same problem as Heka does in our current implementation” then sprinkle
> throughput and minimal cpu and network utilization on top.  If we can’t
> make a decision from there, not sure I have any further suggestions as I am
> not writing the code.
>
> Regards
> -steve
>
>
> From: Jeffrey Zhang <zhang.lei.fly at gmail.com>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev at lists.openstack.org>
> Date: Sunday, November 27, 2016 at 9:40 AM
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev at lists.openstack.org>
> Subject: Re: [openstack-dev] [kolla] the alternative of log processing
> tool
>
> So filebeat is working with Logstash right? We need split the logs into
> pieces by using logstash. IMU, Filebeat do not a variety of processing
> plugins, like Logstash[0].
>
> [0] https://www.elastic.co/guide/en/logstash/current/filter-plugins.html
>
> On Sun, Nov 27, 2016 at 11:30 PM, Ian Cordasco <sigmavirus24 at gmail.com>
> wrote:
>
>> File beat is maintained be elastic and a part of their product line just
>> like ELK. It's a fantastic tool and quite flexible given its age and size
>> of codebase
>>
>> On Nov 26, 2016 11:59 PM, "Jeffrey Zhang" <zhang.lei.fly at gmail.com>
>> wrote:
>>
>>> Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we
>>> have a
>>> blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat.
>>>
>>> For Filebeat, it is just a replacement of logstash-forward[2]. It is not
>>> intent
>>> to replace the Logstash at all.
>>>
>>> > Filebeat is based on the Logstash Forwarder source code and replaces
>>> Logstash
>>> > Forwarder as the method to use for tailing log files and forwarding
>>> them to
>>> > Logstash.
>>>
>>> Fillebeat is a log transport tool rather than log processing too. I do
>>> not
>>> treat it as an alternative at all.
>>>
>>> To be honest, I'd like back to Logstash, and Logstash 5.x is released
>>> with high
>>> performance improvement[4].
>>>
>>> >  In our performance testing, we've seen consistent throughput increases
>>> >  across multiple configurations. In some cases, we observed up to 75%
>>> >  increase in events processed through Logstash.
>>>
>>> another benefit to using Logstash is the whole ELK stack is maintained
>>> by one
>>> community/company. It is well tested and easy to upgrade the whole stack
>>> at the
>>> same time. Using other tools may force us on certain elasticsearch
>>> release.
>>>
>>> So, I think we have to alternative tools.
>>>
>>> * Fluentd
>>> * Logstash
>>>
>>> IMO, we need to make the decision and at least prepare the migration
>>> solution now.
>>>
>>> [1] https://blueprints.launchpad.net/kolla/+spec/heka-deprecation
>>> [2] https://www.elastic.co/guide/en/beats/filebeat/current/migra
>>> ting-from-logstash-forwarder.html
>>> [3] http://www.fluentd.org/
>>> [4] https://www.elastic.co/blog/logstash-5-0-0-released
>>>
>>> --
>>> Regards,
>>> Jeffrey Zhang
>>> Blog: http://xcodest.me
>>>
>>> ____________________________________________________________
>>> ______________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.op
>>> enstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>> ____________________________________________________________
>> ______________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Regards,
> Jeffrey Zhang
> Blog: http://xcodest.me
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Regards,
Jeffrey Zhang
Blog: http://xcodest.me
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161128/abcabe4f/attachment.html>