Open Stack

On 28 May 2015 at 22:45, Nick Jones <nick.jones at datacentred.co.uk> wrote:
>
> On 28 May 2015, at 19:10, Tim Bell <Tim.Bell at cern.ch> wrote:
>
> Using UDP is a great workaround but it does not feel like a fix... can't
> the daemons realise that the syslog socket is not alive and reconnect.
> Given it affects most of the OpenStack projects, a fix inside one of the
> oslo logging libraries (if the error can be detected there) would be great.
>
> We too have been bitten hard by this issue in the past - way before Juno -
> when using rsyslog logging to a remote target (i.e Logstash).  We
> eventually went down the route of using log-courier [1] on x86 and beaver
> [2] on ARM (due to the lack of support for Go).
>
> Both have worked out well for us - if you’re using Logstash it might be
> worth looking into either of these as a solution instead of switching to
> UDP and hoping that you don’t lose any messages that you might care about.
>
> [1] https://github.com/driskell/log-courier
> [2] https://github.com/josegonzalez/python-beaver
>
>
We also tried using direct syslog logging from OpenStack services and hit
the same issues. We've opted to rather let the services log natively, then
have python-beaver forward the logs. This scales well, provides a
consistent log forwarding method and also allows us to do multi-line event
consolidation at the source (necessary because logstash doesn't scale well
if you try to do it there). You can find our work, use it, derive from it
and contribute feedback here: https://github.com/rcbops/rpc-extras/pull/123
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20150529/08949d36/attachment.html>