Open Stack

doh, "instrument" ... not orchestrate. 

-S

________________________________________
From: Sandy Walsh
Sent: Tuesday, September 04, 2012 2:46 PM
To: OpenStack Development Mailing List
Cc: openstack-dev
Subject: RE: OS tracing??

We've been using Tach to orchestrate Openstack services and report to statsd/graphite. https://github.com/ohthree/tach ... works great

I've been trying to land this Inflight Service branch to measure RPC and greenlet overhead
https://review.openstack.org/#/c/11179/
BP: https://blueprints.launchpad.net/nova/+spec/monitoring-service

Hope it helps,
-S




From: Joshua Harlow [harlowja at yahoo-inc.com]
Sent: Tuesday, September 04, 2012 2:35 PM
To: OpenStack Development Mailing List
Cc: openstack-dev
Subject: [openstack-dev] OS tracing??

Has anyone had any luck with trying out some tracing/coverage with the openstack projects to see where the bottlenecks are (outside of test coverage)?

I was thinking about possible ways to do this (there seems to be a lot of different libraries that might help) but was wondering if anyone else has figured out the best one to use yet.

Ideally it should have the following properties (in my mind):

Non-intrusive (shouldn't require sprinkling of timing/trace logic all over)Works with eventlet/greenlet (eventlet is going to switch things in and out, so that has to be taken account of)Probably does this via sampling (?)Writes out some standard format (valgrind like?) for analysis…Can be turned on and off remotely (nice to have, it'd be cool to have an API/entrypoint/… that says enable tracing which can be used on a live system, that system will become slower but it'd be neat)

This could be some special 'admin' entry point (restricted to certain users of course) that could also do stuff like 'reload-configs' or 'enable-tracing' or 'adjust-log-level' or similar administrative actions that would be useful during those crazy debug
 sessions (think a simple admin telnet entrypoint to view stats, similar to what memcache/redis provide via there 'stats' commands…)


Anyone have any ideas on this :-)





-Josh