Open Stack

Has anyone had any luck with trying out some tracing/coverage with the openstack projects to see where the bottlenecks are (outside of test coverage)?

I was thinking about possible ways to do this (there seems to be a lot of different libraries that might help) but was wondering if anyone else has figured out the best one to use yet.

Ideally it should have the following properties (in my mind):

 1.  Non-intrusive (shouldn't require sprinkling of timing/trace logic all over)
 2.  Works with eventlet/greenlet (eventlet is going to switch things in and out, so that has to be taken account of)
 3.  Probably does this via sampling (?)
 4.  Writes out some standard format (valgrind like?) for analysis…
 5.  Can be turned on and off remotely (nice to have, it'd be cool to have an API/entrypoint/… that says enable tracing which can be used on a live system, that system will become slower but it'd be neat)
    *   This could be some special 'admin' entry point (restricted to certain users of course) that could also do stuff like 'reload-configs' or 'enable-tracing' or 'adjust-log-level' or similar administrative actions that would be useful during those crazy debug sessions (think a simple admin telnet entrypoint to view stats, similar to what memcache/redis provide via there 'stats' commands…)

Anyone have any ideas on this :-)

-Josh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20120904/9d3faa1f/attachment.html>