<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>An idea, and one that I've tried to apply in taskflow.</div>
<div><br>
</div>
<div>Since 3.0 added with <a href="http://legacy.python.org/dev/peps/pep-3134">http://legacy.python.org/dev/peps/pep-3134</a> if we can *simulate* except chaining in our projects this would likely help even more with traceability and debugging. I have a common
exception that that I've been using to approximate this (since chaining doesn’t work/exist in 2.7 and 2.6) and it might be useful for others to have similar types of exceptions (and try to print out as much of it as possible when errors occur).</div>
<div><br>
</div>
<div><a href="https://github.com/openstack/taskflow/blob/master/taskflow/exceptions.py#L22">https://github.com/openstack/taskflow/blob/master/taskflow/exceptions.py#L22</a> (see pformat() method that dumps a large string containing all connected causes).</div>
<div><br>
</div>
<div>Might be useful for others (if there are better approaches/libraries that do similar things let me know),</div>
<div><br>
</div>
<div>-Josh</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Morgan Fainberg <<a href="mailto:morgan.fainberg@gmail.com">morgan.fainberg@gmail.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, May 28, 2014 at 8:53 AM<br>
<span style="font-weight:bold">To: </span>Jay Pipes <<a href="mailto:jaypipes@gmail.com">jaypipes@gmail.com</a>>, "OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] Oslo logging eats system level tracebacks by default<br>
</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div><style>body{font-family:Helvetica,Arial;font-size:13px}</style>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">
+1 Providing service crashing information is very valuable. In general we need to provide as much information about why the service exited (critically/traceback/unexpectedly) for our operators.</div>
<div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">
<br>
</div>
<div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">
—Morgan</div>
<div id="bloop_sign_1401292313484024064" class="bloop_sign">
<p><strong>—</strong><br>
<strong>Morgan Fainberg</strong></p>
</div>
<div style="color:black"><br>
From: <span style="color:black">Jay Pipes</span> <a href="mailto:jaypipes@gmail.com">
jaypipes@gmail.com</a><br>
Reply: <span style="color:black">OpenStack Development Mailing List (not for usage questions)</span><a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a><br>
Date: <span style="color:black">May 28, 2014 at 08:50:25</span><br>
To: <span style="color:black"><a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a></span>
<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a><br>
Subject: <span style="color:black"> Re: [openstack-dev] Oslo logging eats system level tracebacks by default
<br>
</span></div>
<br>
<blockquote type="cite" class="clean_bq"><span>
<div>
<div></div>
<div>On 05/28/2014 11:39 AM, Doug Hellmann wrote: <br>
> On Wed, May 28, 2014 at 10:38 AM, Sean Dague <<a href="mailto:sean@dague.net">sean@dague.net</a>> wrote:
<br>
>> When attempting to build a new tool for Tempest, I found that my python <br>
>> syntax errors were being completely eaten. After 2 days of debugging I <br>
>> found that oslo log.py does the following *very unexpected* thing. <br>
>> <br>
>> - replaces the sys.excepthook with it's own function <br>
>> - eats the execption traceback unless debug or verbose are set to True <br>
>> - sets debug and verbose to False by default <br>
>> - prints out a completely useless summary log message at Critical <br>
>> ([CRITICAL] [-] 'id' was my favorite of these) <br>
>> <br>
>> This is basically for an exit level event. Something so breaking that <br>
>> your program just crashed. <br>
>> <br>
>> Note this has nothing to do with preventing stack traces that are <br>
>> currently littering up the logs that happen at many logging levels, it's <br>
>> only about removing the stack trace of a CRITICAL level event that's <br>
>> going to very possibly result in a crashed daemon with no information as <br>
>> to why. <br>
>> <br>
>> So the process of including oslo log makes the code immediately <br>
>> undebuggable unless you change your config file to not the default. <br>
>> <br>
>> Whether or not there was justification for this before, one of the <br>
>> things we heard loud and clear from the operator's meetup was: <br>
>> <br>
>> - Most operators are running at DEBUG level for all their OpenStack <br>
>> services because you can't actually do problem determination in <br>
>> OpenStack for anything < that. <br>
>> - Operators reacted negatively to the idea of removing stack traces <br>
>> from logs, as that's typically the only way to figure out what's going <br>
>> on. It took a while of back and forth to explain that our initiative to <br>
>> do that wasn't about removing them per say, but having the code <br>
>> correctly recover. <br>
>> <br>
>> So the current oslo logging behavior seems inconsistent (we spew <br>
>> exceptions at INFO and WARN levels, and hide all the important stuff <br>
>> with a legitimately uncaught system level crash), undebuggable, and <br>
>> completely against the prevailing wishes of the operator community. <br>
>> <br>
>> I'd like to change that here - <a href="https://review.openstack.org/#/c/95860/">
https://review.openstack.org/#/c/95860/</a> <br>
>> <br>
>> -Sean <br>
> <br>
> I agree, we should dump as much detail as we can when we encounter an <br>
> unhandled exception that causes an app to die. <br>
<br>
+1 <br>
<br>
-jay <br>
<br>
_______________________________________________ <br>
OpenStack-dev mailing list <br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
<br>
</div>
</div>
</span></blockquote>
</div>
</div>
</blockquote>
</span>
</body>
</html>