Open Stack

Hi,

According to the thread of request-id [1], there seems a need for
a wsgi middleware which generates request-id for each REST
request and ensures it in a corresponding response.
To do this, the middlware is located outer-most of a wsgi pipeline
and needs to catch all kind of exceptions.
It is the concept swift already has as catch_error middleware.

How about adding such middlware to oslo?
Ensuring request-id in log and API response is a common requirement
across OpenStack projects but each project implements in different ways now.
I was suggested from Maru on IRC too.

The feature of the proposed middleware is simple:
  * Catch all exceptions in wsgi pipeline
  * Generate request-id for each REST request
  * Set the generated request-id to request environment
  * Add X-OpenStack-Request-Id header to API response to return request-id
  * Expected to place outer-most (i.e., first) in wsgi pipeline

The initial implemetation proposed to neutron is available at
https://review.openstack.org/#/c/58270/
(The name of middleware is not good....)

Regarding implementation,
a question is whether webob can be used or not.


Now request-id is generated in Context class and Context is instantiated
in auth middlware in most projects like nova, cinder, neutron.
We need to change auth middlware so that request-id will be retreived from request env.
Context class and auth middlware are also similar but different acorss projects.
It may time to update/add these classes in oslo.

Regards,
Akihiro

[1] http://lists.openstack.org/pipermail/openstack-dev/2013-November/thread.html#20683

(2013/12/02 23:47), Jay Pipes wrote:
> On 12/01/2013 10:04 PM, John Dickinson wrote:
>> Just to add to the story, Swift uses "X-Trans-Id" and generates it in
>> the outer-most "catch_errors" middleware.
>>
>> Swift's catch errors middleware is responsible for ensuring that the
>> transaction id exists on each request, and that all errors previously
>> uncaught, anywhere in the pipeline, are caught and logged. If there
>> is not a common way to do this, yet, I submit it as a great template
>> for solving this problem. It's simple, scalable, and well-tested (ie
>> tests and running in prod for years).
>>
>>
> https://github.com/openstack/swift/blob/master/swift/common/middleware/cat
> ch_errors.py
>
> ++
>
> If there's prior art here, might as well use it. I'm not a huge fan of
> using the term "transaction" within things that do not have a
> transactional safety context... but that's just because of my background
> in RDBMS stuff. If X-Trans-Id is already in use by another OpenStack
> project, it should probably take precedence over something new unless
> there is a really good reason otherwise (and my personal opinion about
> the semantics of transactions ain't a good reason!).
>
>>   Leaving aside error handling and only focusing on the transaction id
>> (or request id) generation, since OpenStack services are exposed to
>> untrusted clients, how would you propose communicating the
>> appropriate transaction id to a different service? I can see great
>> benefit to having a glance transaction ID carry through to Swift
>> requests (and so on), but how should the transaction id be
>> communicated? It's not sensitive info, but I can imagine a pretty big
>> problem when trying to track down errors if a client application
>> decides to set eg the X-Set-Transaction-Id header on every request to
>> the same thing.
>
> I suppose if this were really a problem (and I'm not sold on the idea
> that it is a problem), one solution might be to store a checksum
> somewhere for the transaction ID and some other piece of data. But I
> don't really see that as super useful, and it would slow things down.
> Glance already stores a checksum for important things like the data in
> an image. If a service needs to be absolutely sure that a piece of data
> hasn't been messed with, this cross-service request ID probably isn't
> the thing to use...
>
>> Thanks for bringing this up, and I'd welcome a patch in Swift that
>> would use a common library to generate the transaction id, if it were
>> installed. I can see that there would be huge advantage to operators
>> to trace requests through multiple systems.
>
> Hmm, so does that mean that you'd be open to (gradually) moving to an
> x-openstack-request-id header to replace x-trans-id?
>
>> Another option would be for each system that calls an another
>> OpenStack system to expect and log the transaction ID for the request
>> that was given. This would be looser coupling and be more forgiving
>> for a heterogeneous cluster. Eg when Glance makes a call to Swift,
>> Glance cloud log the transaction id that Swift used (from the Swift
>> response). Likewise, when Swift makes a call to Keystone, Swift could
>> log the Keystone transaction id. This wouldn't result in a single
>> transaction id across all systems, but it would provide markers so an
>> admin could trace the request.
>
> Sure, this is a perfectly fine option, but doesn't really provide the
> single traceable ID value that I think we're looking for here.
>
> Best,
> -jay
>
>> On Dec 1, 2013, at 5:48 PM, Maru Newby <marun at redhat.com> wrote:
>>
>>>
>>> On Nov 30, 2013, at 1:00 AM, Sean Dague <sean at dague.net> wrote:
>>>
>>>> On 11/29/2013 10:33 AM, Jay Pipes wrote:
>>>>> On 11/28/2013 07:45 AM, Akihiro Motoki wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I am working on adding request-id to API response in
>>>>>> Neutron. After I checked what header is used in other
>>>>>> projects header name varies project by project. It seems
>>>>>> there is no consensus what header is recommended and it is
>>>>>> better to have some consensus.
>>>>>>
>>>>>> nova:     x-compute-request-id cinder:
>>>>>> x-compute-request-id glance:   x-openstack-request-id
>>>>>> neutron:  x-network-request-id  (under review)
>>>>>>
>>>>>> request-id is assigned and used inside of each project now,
>>>>>> so x-<service>-request-id looks good. On the other hand, if
>>>>>> we have a plan to enhance request-id across projects,
>>>>>> x-openstack-request-id looks better.
>>>>>
>>>>> My vote is for:
>>>>>
>>>>> x-openstack-request-id
>>>>>
>>>>> With an implementation of "create a request UUID if none exists
>>>>> yet" in some standardized WSGI middleware...
>>>>
>>>> Agreed. I don't think I see any value in having these have
>>>> different service names, having just x-openstack-request-id
>>>> across all the services seems a far better idea, and come back
>>>> through and fix nova and cinder to be that as well.
>>>
>>> +1
>>>
>>> An openstack request id should be service agnostic to allow
>>> tracking of a request across many services (e.g. a call to nova to
>>> boot a VM should generate a request id that is provided to other
>>> services in requests to provision said VM).  All services would
>>> ideally share a facility for generating new request ids and for
>>> securely accepting request ids from other services.
>>>
>>>
>>> m.
>>>
>>>>
>>>> -Sean
>>>>
>>>> -- Sean Dague http://dague.net
>>>>
>>>> _______________________________________________ OpenStack-dev
>>>> mailing list OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>>
> _______________________________________________
>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> _______________________________________________ OpenStack-dev mailing
>> list OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>