[openstack-dev] Oslo logging eats system level tracebacks by default
Sean Dague
sean at dague.net
Wed May 28 14:38:42 UTC 2014
When attempting to build a new tool for Tempest, I found that my python
syntax errors were being completely eaten. After 2 days of debugging I
found that oslo log.py does the following *very unexpected* thing.
- replaces the sys.excepthook with it's own function
- eats the execption traceback unless debug or verbose are set to True
- sets debug and verbose to False by default
- prints out a completely useless summary log message at Critical
([CRITICAL] [-] 'id' was my favorite of these)
This is basically for an exit level event. Something so breaking that
your program just crashed.
Note this has nothing to do with preventing stack traces that are
currently littering up the logs that happen at many logging levels, it's
only about removing the stack trace of a CRITICAL level event that's
going to very possibly result in a crashed daemon with no information as
to why.
So the process of including oslo log makes the code immediately
undebuggable unless you change your config file to not the default.
Whether or not there was justification for this before, one of the
things we heard loud and clear from the operator's meetup was:
- Most operators are running at DEBUG level for all their OpenStack
services because you can't actually do problem determination in
OpenStack for anything < that.
- Operators reacted negatively to the idea of removing stack traces
from logs, as that's typically the only way to figure out what's going
on. It took a while of back and forth to explain that our initiative to
do that wasn't about removing them per say, but having the code
correctly recover.
So the current oslo logging behavior seems inconsistent (we spew
exceptions at INFO and WARN levels, and hide all the important stuff
with a legitimately uncaught system level crash), undebuggable, and
completely against the prevailing wishes of the operator community.
I'd like to change that here - https://review.openstack.org/#/c/95860/
-Sean
--
Sean Dague
http://dague.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140528/ff03d972/attachment.pgp>
More information about the OpenStack-dev
mailing list