Open Stack

(2013/12/06 17:57), Joe Gordon wrote:
>
> On Dec 6, 2013 9:57 AM, "Maru Newby" <marun at redhat.com <mailto:marun at redhat.com>> wrote:
>  >
>  >
>  > On Dec 6, 2013, at 1:09 AM, John Dickinson <me at not.mn <mailto:me at not.mn>> wrote:
>  >
>  > >
>  > > On Dec 5, 2013, at 1:36 AM, Maru Newby <marun at redhat.com <mailto:marun at redhat.com>> wrote:
>  > >
>  > >>
>  > >> On Dec 3, 2013, at 12:18 AM, Joe Gordon <joe.gordon0 at gmail.com <mailto:joe.gordon0 at gmail.com>> wrote:
>  > >>
>  > >>>
>  > >>>
>  > >>>
>  > >>> On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson <me at not.mn <mailto:me at not.mn>> wrote:
>  > >>> Just to add to the story, Swift uses "X-Trans-Id" and generates it in the outer-most "catch_errors" middleware.
>  > >>>
>  > >>> Swift's catch errors middleware is responsible for ensuring that the transaction id exists on each request, and that all errors previously uncaught, anywhere in the pipeline, are caught and logged. If there is not a common way to do this, yet, I submit it as a great template for solving this
> problem. It's simple, scalable, and well-tested (ie tests and running in prod for years).
>  > >>>
>  > >>> https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py
>  > >>>
>  > >>> Leaving aside error handling and only focusing on the transaction id (or request id) generation, since OpenStack services are exposed to untrusted clients, how would you propose communicating the appropriate transaction id to a different service? I can see great benefit to having a glance
> transaction ID carry through to Swift requests (and so on), but how should the transaction id be communicated? It's not sensitive info, but I can imagine a pretty big problem when trying to track down errors if a client application decides to set eg the X-Set-Transaction-Id header on every request
> to the same thing.
>  > >>>
>  > >>> -1 to cross service request IDs, for the reasons John mentions above.
>  > >>>
>  > >>>
>  > >>> Thanks for bringing this up, and I'd welcome a patch in Swift that would use a common library to generate the transaction id, if it were installed. I can see that there would be huge advantage to operators to trace requests through multiple systems.
>  > >>>
>  > >>> Another option would be for each system that calls an another OpenStack system to expect and log the transaction ID for the request that was given. This would be looser coupling and be more forgiving for a heterogeneous cluster. Eg when Glance makes a call to Swift, Glance cloud log the
> transaction id that Swift used (from the Swift response). Likewise, when Swift makes a call to Keystone, Swift could log the Keystone transaction id. This wouldn't result in a single transaction id across all systems, but it would provide markers so an admin could trace the request.
>  > >>>
>  > >>> There was a session on this at the summit, and although the notes are a little scarce this was the conclusion we came up with.  Every time a cross service call is made, we will log and send a notification for ceilometer to consume, with the request-ids of both request ids.  One of the
> benefits of this approach is that we can easily generate a tree of all the API calls that are made (and clearly show when multiple calls are made to the same service), something that just a cross service request id would have trouble with.
>  > >>
>  > >> Is wise to trust anything a client provides to ensure traceability?  If a user receives a request id back from Nova, then submits that request id in an unrelated request to Neutron, the traceability would be effectively corrupted.  If the consensus is that we don't want to securely deliver
> request ids for inter-service calls, how about requiring a service to log its request id along with the request id returned from a call to another service to achieve the a similar result?
>  > >
>  > > Yes, this is what I was proposing. I think this is the best path forward.

I think Logging returned request-id at client side is the best way.
We can track each API call even when multiple API calls are invoked in one API request.
 >
 > Nova does this for glance client today.  Glance client logs the out glances request Id and that message is wrapped with novas request id.

When I ran devstack, glance client log (with glance request-id) is not wrapped
with nova request-id: http://paste.openstack.org/show/54590/
It is because glanceclient uses standard logging instead of openstack.common.log.
It is common to all client libraries and I am not sure why client libraries
do not use openstack.common.log.
>
> Ceilometer wanted notifications of these events as well so it could track things better.

Should this feature a part of client libraries or server side which calls client libraries?
At now client libraries do not return information about response headers including request-id.
If we support it in server side, a standard way to return request-id in a response to a caller.
I don't have good idea on it at the moment and it is just a question.

Thanks,
Akihiro

>
>  >
>  > Ok, great.  And as per your suggestion, a middleware-based error handler will soon be proposed for oslo that will secondarily ensure that a request id is added to the request.
>  >
>  > >
>  > >
>  > >> The catch is that every call point (or client instantiation?) would have to be modified to pass the request id instead of just logging at one place in each service.  Is that a cost worth paying?
>  > >
>  > > Perhaps this is my ignorance of how other projects work today, but does this not already happen? Is it possible to get a response from an API call to an OpenStack project that doesn't include a request id?
>  >
>  > We'll have it in Neutron real-soon-now, and then I think the answer will be 'yes'.
>  >
>  > On reflection, it should be easy enough for a given service to ensure that calls to other services are automatically instrumented to log request id pairs.  Again, probably something for oslo.
>  >
>  >
>  > m.
>  > _______________________________________________
>  > OpenStack-dev mailing list
>  > OpenStack-dev at lists.openstack.org <mailto:OpenStack-dev at lists.openstack.org>
>  > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>