[Openstack-operators] [openstack-dev][tc][all][osprofiler] OSprofiler is dead, long live OSprofiler
Daniel P. Berrange
berrange at redhat.com
Mon Nov 9 12:11:17 UTC 2015
On Mon, Nov 09, 2015 at 02:57:37AM -0800, Boris Pavlovic wrote:
> Hi stackers,
> It's not a big secret that OpenStack is huge and complicated ecosystem of
> services that are working together to implement OpenStack API.
> For example booting VM is going through many projects and services:
> nova-api, nova-scheduler, nova-compute, glance-api, glance-registry,
> keystone, cinder-api, neutron-api... and many others.
> The question is how to understand what part of the request takes the most
> of the time and should be improved. It's especially interested to get such
> data under the load.
> To make it simple, I wrote OSProfiler which is tiny library that should be
> added to all OpenStack
> projects to create cross project/service tracer/profiler.
> Demo (trace of CLI command: nova boot) can be found here:
> This library is very simple. For those who wants to know how it works and
> how it's integrated with OpenStack take a look here:
> What is the current status?
> Good news:
> - OSprofiler is mostly done
> - OSprofiler is integrated with Cinder, Glance, Trove & Ceilometer
> Bad news:
> - OSprofiler is not integrated in a lot of important projects: Keystone,
> Nova, Neutron
> - OSprofiler can use only Ceilometer + oslo.messaging as a backend
> - OSprofiler stores part of arguments in api-paste.ini part in project.conf
> which is terrible thing
> - There is no DSVM job that check that changes in OSprofiler don't break
> the projects that are using it
> - It's hard to enable OSprofiler in DevStack
> Good news:
> I spend some time and made 4 specs that should address most of issues:
> Let's make it happen in Mitaka!
> By the way somebody would like to join this effort?)
I'm very interested in seeing this kind of capability integrated across
openstack. I've really needed it in working in Nova for many times.
6 months back or so (when I didn't know osprofiler existed), I hacked up
a roughly equivalent library for openstack:
I never had time to persue this further and then found out about osprofiler
so it dropped in priority for me.
Some notable things I think I did differently
- Used oslo.versionedobjects for recording the data to provide a
structured data model with easy support for future extension,
and well defined serialization format
- Structured data types for all the various common types of operation
to instrument (database request, RPC call, RPC dispatch, REST call
REST dispatch, native library call, thread spawn, thread main,
external command spawn). This is to ensure all apps using the library
provide the same set of data for each type of operation.
- Ability to capture stack traces against each profile point to
allow creation of control flow graphs showing which code paths
consumed significant time.
- Abstract "consumer" API for different types of persistence backend.
Rather than ceilometer, my initial consumer just serialized to
plain files in well known directories, using oslo.versionedobjects
serialization format. I can see ceilometer might be nice for
production deployments, but plain files was simpler for developer
environments which might not even be running ceilometer
- Explicit tracking of nesting of instrunmented operation had a parent
operation. At the top level was things like thread main, RPC dispatch
and REST dispatch. IIUC, with osprofiler you could potentially infer
the nesting by sorting based on start/end timestamps, but I felt an
explicit representation was nicer to work with from the object
My approach would require oslo.devsupport to be integrated into the
oslo.concurrency, oslo.db, oslo.messaging components, as well as the
various apps. I did quick hacks to enable this for Nova & pieces it
uses you can see from these (non-upstreamed) commits:
I see you already intend to make ceilometer optional and allow other
backends, so that's nice. I would love to see Osprofiler take on some
of the other ideas I had in my alternative, particularly the data model
based around oslo.versionedobjects to provide standard format for various
core types of operation, and the ability to record stack traces.
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
More information about the OpenStack-operators