Open Stack

As a user of logstash myself, thanks for the contribution.  I'm currently using the configuration at [0] with some minor modification to collect from more log files.  These two configurations for logstash seem similar, are they both going to be kept in sync?

[0]: https://github.com/openstack/osops-tools-logging/blob/master/logstash/basic/logstash.conf 

-----Original Message-----
From: Jonathan Harker [mailto:jesusaurus at hpe.com] 
Sent: Friday, May 13, 2016 10:12
To: openstack-operators at lists.openstack.org
Cc: openstack-infra at lists.openstack.org
Subject: [Openstack-operators] [logging] Announcing openstack-infra/logstash-filters

The openstack-infra team has put a lot of effort into creating logstash filters to parse openstack logs. These filters are primarily used to parse service logs from devstack runs, but should work for production deployments as well. Yesterday I worked with Clark Boylan to move these filters out of puppet and into their own project called openstack-infra/logstash-filters to make them easy to reconsume. This project has three files in the filters/ directory: an example input section, an example output section, and the filters section used to index devstack service log data into logstash.openstack.org. Using conf.d style logstash configs, you can easily drop these filters into your own config while using custom input and output config sections. You can see how this is done for logstash.openstack.org using puppet at [1].

These filters work by switching on tags for the different log formats.
In order for them to parse the logs correctly, the correct tags need to be applied to the logs before they reach the filters. The tags applied to the devstack service logs can be viewed at [2]. Most service logs use the "oslofmt" tag, but some require the "apachecombined" tag instead.
The filters also understand the "libvirt" and "syslog" tags for their respective standard log formats.

[1]
https://review.openstack.org/#/c/310052/6/modules/openstack_project/manifests/logstash_worker.pp
[2]
http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/logstash/jenkins-log-client.yaml#n31

--
Jonathan Harker

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators