Open Stack

On 5/27/15, 10:15, "Doug Hellmann" <doug at doughellmann.com> wrote:

>Excerpts from Kekane, Abhishek's message of 2015-05-27 07:06:56 +0000:
>> Hi Devs,
>> 
>> Each OpenStack service sends a request ID header with HTTP responses.
>>This request ID can be useful for tracking down problems in the logs.
>>However, when operation crosses service boundaries, this tracking can
>>become difficult, as each service has its own request ID. Request ID is
>>not returned to the caller, so it is not easy to track the request. This
>>becomes especially problematic when requests are coming in parallel. For
>>example, glance will call cinder for creating image, but that cinder
>>instance may be handling several other requests at the same time. By
>>using same request ID in the log, user can easily find the cinder
>>request ID that is same as glance request ID in the g-api log. It will
>>help operators/developers to analyse logs effectively.
>> 
>> To address this issue we have come up with following solutions:
>> 
>> Solution 1: Return tuple containing headers and body from respective
>>clients (also favoured by Joe Gordon)
>> Reference: 
>>https://review.openstack.org/#/c/156508/6/specs/log-request-id-mappings.r
>>st
>> 
>> Pros:
>> 1. Maintains backward compatibility
>> 2. Effective debugging/analysing of the problem as both calling service
>>request-id and called service request-id are logged in same log message
>> 3. Build a full call graph
>> 4. End user will able to know the request-id of the request and can
>>approach service provider to know the cause of failure of particular
>>request.
>> 
>> Cons:
>> 1. The changes need to be done first in cross-projects before making
>>changes in clients
>> 2. Applications which are using python-*clients needs to do required
>>changes (check return type of  response)
>> 
>> 
>> Solution 2:  Use thread local storage to store 'x-openstack-request-id'
>>returned from headers (suggested by Doug Hellmann)
>> Reference: 
>>https://review.openstack.org/#/c/156508/9/specs/log-request-id-mappings.r
>>st
>> 
>> Add new method 'get_openstack_request_id' to return this request-id to
>>the caller.
>> 
>> Pros:
>> 1. Doesn't break compatibility
>> 2. Minimal changes are required in client
>> 3. Build a full call graph
>> 
>> Cons:
>> 1. Malicious user can send long request-id to fill up the disk-space,
>>resulting in potential DoS
>> 2. Changes need to be done in all python-*clients
>> 3. Last request id should be flushed out in a subsequent call otherwise
>>it will return wrong request id to the caller
>> 
>> 
>> Solution 3: Unique request-id across OpenStack Services (suggested by
>>Jamie Lennox)
>> Reference: 
>>https://review.openstack.org/#/c/156508/10/specs/log-request-id-mappings.
>>rst
>> 
>> Get 'x-openstack-request-id' from auth plugin and add it to the request
>>headers. If 'x-openstack-request-id' key is present in the request
>>header, then it will use the same one further or else it will generate a
>>new one.
>> 
>> Dependencies:
>> https://review.openstack.org/#/c/164582/ - Include request-id in auth
>>plugin and add it to request headers
>> https://review.openstack.org/#/c/166063/ - Add session-object for
>>glance client
>> Add 'UserAuthPlugin' and '_ContextAuthPlugin' same as nova in cinder
>>and neutron
>> 
>> 
>> Pros:
>> 1. Using same request id for the request crossing multiple service
>>boundaries will help operators/developers identify the problem quickly
>> 2. Required changes only in keystonemiddleware and oslo_middleware
>>libraries. No changes are required in the python client bindings or
>>OpenStack core services
>> 
>> Cons:
>> 1. As 'x-openstack-request-id' in the request header will be visible to
>>the user, it is possible to send same request id for multiple requests
>>which in turn could create more problems in case of troubleshooting
>>cause of the failure as request_id middleware will not check for its
>>uniqueness in the scope of the running OpenStack service.
>> 2. Having the same request ID for all services for a single user API
>>call means you cannot generate a full call graph. For example if a
>>single user's nova API call produces 2 calls to glance you want to be
>>able to differentiate the two different calls.
>> 
>> 
>> During the Liberty design summit, I had a chance of discussing these
>>designs with some of the core members like Doug, Joe Gordon, Jamie
>>Lennox etc. But not able to came to any conclusion on the final design
>>and know the communities direction by which way they want to use this
>>request-id effectively.
>> 
>> However IMO, solution 1 sounds more useful as the debugger can able to
>>build the full call graph which can be helpful for analysing gate
>>failures effectively as well as end user will be able to know his
>>request-id and can track his request.
>> 
>> I request all community members to go through these solutions and let
>>us know which is the appropriate way to improve the logs by logging
>>request-id.
>
>Robert Collins pointed out that os-profiler is already tracking
>requests across REST calls. Does it use one of the proposed methods?
>Since os-profiler is seeing more adoption, could we combine efforts
>here?
>
>Doug

Speaking with my stackforge/os-ansible-deployment hat on, I doubt people
are running osprofiler in production and those same people would likely
still want to be able to trace things via request-id. Further, osprofiler
currently only reports to ceilometer. A shared request-id would ideally
appear in the logs and assuming the deployment uses something like the ELK
stack, it could leverage that without needing osprofiler (or ceilometer)
installed, configured, or enabled.

While there's room for collaboration, I don't think we should tie this to
osprofiler since it's something people could want to use without
osprofiler installed or running as well.

- Ian