Open Stack

On Dec 6, 2013, at 1:09 AM, John Dickinson <me at not.mn> wrote:

> 
> On Dec 5, 2013, at 1:36 AM, Maru Newby <marun at redhat.com> wrote:
> 
>> 
>> On Dec 3, 2013, at 12:18 AM, Joe Gordon <joe.gordon0 at gmail.com> wrote:
>> 
>>> 
>>> 
>>> 
>>> On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson <me at not.mn> wrote:
>>> Just to add to the story, Swift uses "X-Trans-Id" and generates it in the outer-most "catch_errors" middleware.
>>> 
>>> Swift's catch errors middleware is responsible for ensuring that the transaction id exists on each request, and that all errors previously uncaught, anywhere in the pipeline, are caught and logged. If there is not a common way to do this, yet, I submit it as a great template for solving this problem. It's simple, scalable, and well-tested (ie tests and running in prod for years).
>>> 
>>> https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py
>>> 
>>> Leaving aside error handling and only focusing on the transaction id (or request id) generation, since OpenStack services are exposed to untrusted clients, how would you propose communicating the appropriate transaction id to a different service? I can see great benefit to having a glance transaction ID carry through to Swift requests (and so on), but how should the transaction id be communicated? It's not sensitive info, but I can imagine a pretty big problem when trying to track down errors if a client application decides to set eg the X-Set-Transaction-Id header on every request to the same thing.
>>> 
>>> -1 to cross service request IDs, for the reasons John mentions above.
>>> 
>>> 
>>> Thanks for bringing this up, and I'd welcome a patch in Swift that would use a common library to generate the transaction id, if it were installed. I can see that there would be huge advantage to operators to trace requests through multiple systems.
>>> 
>>> Another option would be for each system that calls an another OpenStack system to expect and log the transaction ID for the request that was given. This would be looser coupling and be more forgiving for a heterogeneous cluster. Eg when Glance makes a call to Swift, Glance cloud log the transaction id that Swift used (from the Swift response). Likewise, when Swift makes a call to Keystone, Swift could log the Keystone transaction id. This wouldn't result in a single transaction id across all systems, but it would provide markers so an admin could trace the request.
>>> 
>>> There was a session on this at the summit, and although the notes are a little scarce this was the conclusion we came up with.  Every time a cross service call is made, we will log and send a notification for ceilometer to consume, with the request-ids of both request ids.  One of the benefits of this approach is that we can easily generate a tree of all the API calls that are made (and clearly show when multiple calls are made to the same service), something that just a cross service request id would have trouble with.
>> 
>> Is wise to trust anything a client provides to ensure traceability?  If a user receives a request id back from Nova, then submits that request id in an unrelated request to Neutron, the traceability would be effectively corrupted.  If the consensus is that we don't want to securely deliver request ids for inter-service calls, how about requiring a service to log its request id along with the request id returned from a call to another service to achieve the a similar result?
> 
> Yes, this is what I was proposing. I think this is the best path forward.

Ok, great.  And as per your suggestion, a middleware-based error handler will soon be proposed for oslo that will secondarily ensure that a request id is added to the request.  

> 
> 
>> The catch is that every call point (or client instantiation?) would have to be modified to pass the request id instead of just logging at one place in each service.  Is that a cost worth paying?
> 
> Perhaps this is my ignorance of how other projects work today, but does this not already happen? Is it possible to get a response from an API call to an OpenStack project that doesn't include a request id?

We'll have it in Neutron real-soon-now, and then I think the answer will be 'yes'.

On reflection, it should be easy enough for a given service to ensure that calls to other services are automatically instrumented to log request id pairs.  Again, probably something for oslo.


m.