Open Stack

On Mon, Mar 03, 2014 at 02:58:34PM -0700, Pete Zaitcev wrote:
> Dear Solly:
> 
> I cobbled together a working prototype of Guru Meditation for Swift
> just to see how it worked. I did not use Oslo classes, but used the
> code from Dan's prototype and from your Nova review. Here's the

Sigh, pllease don't reinvent the wheel with special code just for swift.
Use the oslo common report module for this, so we have standardized
behaviour and code across all projects - that's the whole point of oslo
after all.

> Gerrit link:
>  https://review.openstack.org/70513
> 
> Looking at the collected tracebacks, most of all they seem singularly
> useless. No matter how loaded the process is, they always show
> something like:
> 
>   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 226, in run
>     self.wait(sleep_time)
>   File "/usr/lib/python2.6/site-packages/eventlet/hubs/poll.py", line 84, in wait
>     presult = self.do_poll(seconds)
>   File "/usr/lib/python2.6/site-packages/eventlet/hubs/poll.py", line 73, in do_poll
>     return self.poll.poll(int(seconds * 1000.0))
>   File "/usr/lib/python2.6/site-packages/swift/common/daemon.py", line 103, in <lambda>
>     *args))
>   File "/usr/lib/python2.6/site-packages/swift/common/guru_meditation.py", line 79, in signal_handler
>     dump_threads(gthr_model, report_fp)
>   File "/usr/lib/python2.6/site-packages/swift/common/guru_meditation.py", line 53, in dump_threads
>     thread.dump(report_fp)
>   File "/usr/lib/python2.6/site-packages/swift/common/guru_meditation.py", line 29, in dump
>     traceback.print_stack(self.stack, file=report_fp)
> 
> The same is true for native and gree threads: they all seem to be
> anchored in he lambda that passes parameters to the signal handler,
> so they show nothing of value.
> 
> So, my question is: did you look at traces in Nova and if yes,
> did you catch anything? If yes, where is the final code that works?

Yes, we've successfully used the oslo common code to diagnose deadlock
in Nova recently, from the greenthread stack traces - eg this log:

  http://paste.openstack.org/show/62463/

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|