Open Stack

On Mon, Nov 09, 2015 at 02:57:37AM -0800, Boris Pavlovic wrote:
> Hi stackers,
> 
> Intro
> -------
> 
> It's not a big secret that OpenStack is huge and complicated ecosystem of
> different
> services that are working together to implement OpenStack API.
> 
> For example booting VM is going through many projects and services:
> nova-api, nova-scheduler, nova-compute, glance-api, glance-registry,
> keystone, cinder-api, neutron-api... and many others.
> 
> The question is how to understand what part of the request takes the most
> of the time and should be improved. It's especially interested to get such
> data under the load.
> 
> To make it simple, I wrote OSProfiler which is tiny library that should be
> added to all OpenStack
> projects to create cross project/service tracer/profiler.
> 
> Demo (trace of CLI command: nova boot) can be found here:
> http://boris-42.github.io/ngk.html
> 
> This library is very simple. For those who wants to know how it works and
> how it's integrated with OpenStack take a look here:
> https://github.com/openstack/osprofiler/blob/master/README.rst
> 
> What is the current status?
> -----------------------------------
> 
> Good news:
> - OSprofiler is mostly done
> - OSprofiler is integrated with Cinder, Glance, Trove & Ceilometer
> 
> Bad news:
> - OSprofiler is not integrated in a lot of important projects: Keystone,
> Nova, Neutron
> - OSprofiler can use only Ceilometer + oslo.messaging as a backend
> - OSprofiler stores part of arguments in api-paste.ini part in project.conf
> which is terrible thing
> - There is no DSVM job that check that changes in OSprofiler don't break
> the projects that are using it
> - It's hard to enable OSprofiler in DevStack
> 
> Good news:
> I spend some time and made 4 specs that should address most of issues:
> https://github.com/openstack/osprofiler/tree/master/doc/specs
> 
> Let's make it happen in Mitaka!
> 
> Thoughts?
> By the way somebody would like to join this effort?)

I'm very interested in seeing this kind of capability integrated across
openstack. I've really needed it in working in Nova for many times.
6 months back or so (when I didn't know osprofiler existed), I hacked up
a roughly equivalent library for openstack:

   https://github.com/berrange/oslo.devsupport

I never had time to persue this further and then found out about osprofiler
so it dropped in priority for me.

Some notable things I think I did differently

 - Used oslo.versionedobjects for recording the data to provide a
   structured data model with easy support for future extension,
   and well defined serialization format

 - Structured data types for all the various common types of operation
   to instrument (database request, RPC call, RPC dispatch, REST call
   REST dispatch, native library call, thread spawn, thread main,
   external command spawn). This is to ensure all apps using the library
   provide the same set of data for each type of operation.

 - Ability to capture stack traces against each profile point to
   allow creation of control flow graphs showing which code paths
   consumed significant time.

 - Abstract "consumer" API for different types of persistence backend.
   Rather than ceilometer, my initial consumer just serialized to
   plain files in well known directories, using oslo.versionedobjects
   serialization format. I can see ceilometer might be nice for
   production deployments, but plain files was simpler for developer
   environments which might not even be running ceilometer

 - Explicit tracking of nesting of instrunmented operation had a parent
   operation. At the top level was things like thread main, RPC dispatch
   and REST dispatch. IIUC, with osprofiler you could potentially infer
   the nesting by sorting based on start/end timestamps, but I felt an
   explicit representation was nicer to work with from the object
   model POV.

My approach would require oslo.devsupport to be integrated into the
oslo.concurrency, oslo.db, oslo.messaging components, as well as the
various apps. I did quick hacks to enable this for Nova & pieces it
uses you can see from these (non-upstreamed) commits:

  https://github.com/berrange/oslo.concurrency/commit/2fbbc9cf4f23c5c2b30ff21e9e06235a79edbc20
  https://github.com/berrange/oslo.messaging/commit/5a1fd87650e56a01ae9d8cc773e4d030d84cc6d8
  https://github.com/berrange/nova/commit/3320b8957728a1acf786296eadf0bb40cb4df165

I see you already intend to make ceilometer optional and allow other
backends, so that's nice. I would love to see Osprofiler take on some
of the other ideas I had in my alternative, particularly the data model
based around oslo.versionedobjects to provide standard format for various
core types of operation, and the ability to record stack traces.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|