Open Stack

I'd like to introduce an option to use asynchronous logging to the common logging module (openstack.common.log). It would default to disabled.  Mark McLoughlin recommended that I bring this to the mailing list for discussion and feedback.

Currently, whenever a log message is sent, the calling greenthread blocks. Where eventlet isn't used at all, the thread blocks. Using eventlet, the other greenthreads get processed while this blocks, but the calling greenthread is blocked all the same.

My patch addresses this by allowing calling threads/greenthreads to send non-blocking calls to the logger. Log messages will be pushed into the I/O bound handlers in the background.  This is done by creating a new subclass of Logger, pushing messages into a process-local, global queue.  It creates a thread to pop messages out and send them to the logging handlers. The standard Python Queue module is used, for compatibility and safety with both eventlet and standard Python threading.

It would trivially attach to the existing logging system. It currently activates with the following addition to log.setup():

  if CONF.logging_async:
      logging.setLoggerClass(ThreadedLogger)


Two immediate concerns with my general approach:
 * Log timestamps could be skewed from the time they were emitted.
    - I am looking at generating the messages sooner, but sending them at a delay.
    - Potentially, some logging backends might discard in-message timestamps.
 * Messages will be lost at the destruction of the process unless a wait/cleanup is performed.
    - this can happen now anyway, but less severely.
    - we add a cleanup method.
    - cleanup method doesn't help with more catastrophic failures.

I'm welcome to hear other questions, concerns and feedback.

Regards,
Eric Windisch

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121105/90787e5e/attachment.html>