[openstack-dev] [kolla] the alternative of log processing tool

Jeffrey Zhang zhang.lei.fly at gmail.com
Sun Nov 27 16:52:07 UTC 2016


After reading the docs in github, i do not think Snap can handle logs very
well.  Snap introduce itself as "The open telemetry framework". I think it
is
more like ceilometer/collectd/zabbix. I also can not find how to create a
mock
syslog socket to collect logs[0], how to use regexp to split the logs?.
I do
not think we have enough time and energy to waiting until Snap fix it.



It may be a cool tool. But it is not mature, imo. Why not use a well used
and
mature software, even though it may be slower than others? But it works very
well and lots of guys are familiar with it.
​​



[0] https://github.com/intelsdi-x/snap/issues/1117

On Sun, Nov 27, 2016 at 9:46 PM, Michał Jastrzębski <inc007 at gmail.com>
wrote:

> Hey,
>
> I am also working with Snap community to enable log forwarding with it [1].
> Snap is super lightweight and additional benefit of this solution
> would be that it can also handle monitoring, which was it's initial
> role. One service to handle both would be elegant. I'll keep you
> posted but let's not throw away this idea just yet.
>
> Cheers,
> Michal
>
> [1] https://github.com/intelsdi-x/snap
>
> On 26 November 2016 at 23:55, Jeffrey Zhang <zhang.lei.fly at gmail.com>
> wrote:
> > Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we
> have a
> > blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat.
> >
> > For Filebeat, it is just a replacement of logstash-forward[2]. It is not
> > intent
> > to replace the Logstash at all.
> >
> >> Filebeat is based on the Logstash Forwarder source code and replaces
> >> Logstash
> >> Forwarder as the method to use for tailing log files and forwarding them
> >> to
> >> Logstash.
> >
> > Fillebeat is a log transport tool rather than log processing too. I do
> not
> > treat it as an alternative at all.
> >
> > To be honest, I'd like back to Logstash, and Logstash 5.x is released
> with
> > high
> > performance improvement[4].
> >
> >>  In our performance testing, we've seen consistent throughput increases
> >>  across multiple configurations. In some cases, we observed up to 75%
> >>  increase in events processed through Logstash.
> >
> > another benefit to using Logstash is the whole ELK stack is maintained by
> > one
> > community/company. It is well tested and easy to upgrade the whole stack
> at
> > the
> > same time. Using other tools may force us on certain elasticsearch
> release.
> >
> > So, I think we have to alternative tools.
> >
> > * Fluentd
> > * Logstash
> >
> > IMO, we need to make the decision and at least prepare the migration
> > solution now.
> >
> > [1] https://blueprints.launchpad.net/kolla/+spec/heka-deprecation
> > [2]
> > https://www.elastic.co/guide/en/beats/filebeat/current/
> migrating-from-logstash-forwarder.html
> > [3] http://www.fluentd.org/
> > [4] https://www.elastic.co/blog/logstash-5-0-0-released
> >
> > --
> > Regards,
> > Jeffrey Zhang
> > Blog: http://xcodest.me
> >
> > ____________________________________________________________
> ______________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
> unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Regards,
Jeffrey Zhang
Blog: http://xcodest.me
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161128/4f945796/attachment.html>


More information about the OpenStack-dev mailing list