Open Stack

On Mon, Dec 02, 2013 at 09:47:54AM -0500, Jay Pipes wrote:
> On 12/01/2013 10:04 PM, John Dickinson wrote:
> >Just to add to the story, Swift uses "X-Trans-Id" and generates it in
> >the outer-most "catch_errors" middleware.
> >
> >Swift's catch errors middleware is responsible for ensuring that the
> >transaction id exists on each request, and that all errors previously
> >uncaught, anywhere in the pipeline, are caught and logged. If there
> >is not a common way to do this, yet, I submit it as a great template
> >for solving this problem. It's simple, scalable, and well-tested (ie
> >tests and running in prod for years).
> >
> >https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py
> 
> ++
> 
> If there's prior art here, might as well use it. I'm not a huge fan
> of using the term "transaction" within things that do not have a
> transactional safety context... but that's just because of my
> background in RDBMS stuff. If X-Trans-Id is already in use by
> another OpenStack project, it should probably take precedence over
> something new unless there is a really good reason otherwise (and my
> personal opinion about the semantics of transactions ain't a good
> reason!).
> 
> > Leaving aside error handling and only focusing on the transaction id
> >(or request id) generation, since OpenStack services are exposed to
> >untrusted clients, how would you propose communicating the
> >appropriate transaction id to a different service? I can see great
> >benefit to having a glance transaction ID carry through to Swift
> >requests (and so on), but how should the transaction id be
> >communicated? It's not sensitive info, but I can imagine a pretty big
> >problem when trying to track down errors if a client application
> >decides to set eg the X-Set-Transaction-Id header on every request to
> >the same thing.
> 
> I suppose if this were really a problem (and I'm not sold on the
> idea that it is a problem), one solution might be to store a
> checksum somewhere for the transaction ID and some other piece of
> data. But I don't really see that as super useful, and it would slow
> things down. Glance already stores a checksum for important things
> like the data in an image. If a service needs to be absolutely sure
> that a piece of data hasn't been messed with, this cross-service
> request ID probably isn't the thing to use...

There was actually a summit session on this in the nova track. The direction
decided there was not to have request-id be set by the clients but instead to
just log the mappings in the service logs. For example, if nova makes a call
to glance when it gets the glance api response it will log both the nova
request-id and the glance request-id. There was also talk of making a
ceilometer notification for this mapping event. The session etherpad is here:

https://etherpad.openstack.org/p/icehouse-summit-nova-cross-project-request-ids

> 
> >Thanks for bringing this up, and I'd welcome a patch in Swift that
> >would use a common library to generate the transaction id, if it were
> >installed. I can see that there would be huge advantage to operators
> >to trace requests through multiple systems.
> 
> Hmm, so does that mean that you'd be open to (gradually) moving to
> an x-openstack-request-id header to replace x-trans-id?
> 
> >Another option would be for each system that calls an another
> >OpenStack system to expect and log the transaction ID for the request
> >that was given. This would be looser coupling and be more forgiving
> >for a heterogeneous cluster. Eg when Glance makes a call to Swift,
> >Glance cloud log the transaction id that Swift used (from the Swift
> >response). Likewise, when Swift makes a call to Keystone, Swift could
> >log the Keystone transaction id. This wouldn't result in a single
> >transaction id across all systems, but it would provide markers so an
> >admin could trace the request.
> 
> Sure, this is a perfectly fine option, but doesn't really provide
> the single traceable ID value that I think we're looking for here.

Yes, but if you're looking for that it wouldn't be very hard to write a simple
tool to change the IDs after the fact. I think having the mapping messages are
fine though, because the idea is you would be tracing a request in the logs.
Although, I guess it would make just grepping for a request id and getting the
full request log a bit more difficult.

-Matt Treinish

> >On Dec 1, 2013, at 5:48 PM, Maru Newby <marun at redhat.com> wrote:
> >
> >>
> >>On Nov 30, 2013, at 1:00 AM, Sean Dague <sean at dague.net> wrote:
> >>
> >>>On 11/29/2013 10:33 AM, Jay Pipes wrote:
> >>>>On 11/28/2013 07:45 AM, Akihiro Motoki wrote:
> >>>>>Hi,
> >>>>>
> >>>>>I am working on adding request-id to API response in
> >>>>>Neutron. After I checked what header is used in other
> >>>>>projects header name varies project by project. It seems
> >>>>>there is no consensus what header is recommended and it is
> >>>>>better to have some consensus.
> >>>>>
> >>>>>nova:     x-compute-request-id cinder:
> >>>>>x-compute-request-id glance:   x-openstack-request-id
> >>>>>neutron:  x-network-request-id  (under review)
> >>>>>
> >>>>>request-id is assigned and used inside of each project now,
> >>>>>so x-<service>-request-id looks good. On the other hand, if
> >>>>>we have a plan to enhance request-id across projects,
> >>>>>x-openstack-request-id looks better.
> >>>>
> >>>>My vote is for:
> >>>>
> >>>>x-openstack-request-id
> >>>>
> >>>>With an implementation of "create a request UUID if none exists
> >>>>yet" in some standardized WSGI middleware...
> >>>
> >>>Agreed. I don't think I see any value in having these have
> >>>different service names, having just x-openstack-request-id
> >>>across all the services seems a far better idea, and come back
> >>>through and fix nova and cinder to be that as well.
> >>
> >>+1
> >>
> >>An openstack request id should be service agnostic to allow
> >>tracking of a request across many services (e.g. a call to nova to
> >>boot a VM should generate a request id that is provided to other
> >>services in requests to provision said VM).  All services would
> >>ideally share a facility for generating new request ids and for
> >>securely accepting request ids from other services.
> >>
> >>
> >>m.
> >>
> >>>
> >>>-Sean
> >>>
> >>>-- Sean Dague http://dague.net
> >>>
> >>>_______________________________________________ OpenStack-dev
> >>>mailing list OpenStack-dev at lists.openstack.org
> >>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>
> >>
> >>>
> _______________________________________________
> >>OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
> >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> >_______________________________________________ OpenStack-dev mailing
> >list OpenStack-dev at lists.openstack.org
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev