Open Stack

On Wed, May 27, 2015 at 12:06 AM, Kekane, Abhishek <
Abhishek.Kekane at nttdata.com> wrote:

>  Hi Devs,
>
>
>
> Each OpenStack service sends a request ID header with HTTP responses. This
> request ID can be useful for tracking down problems in the logs. However,
> when operation crosses service boundaries, this tracking can become
> difficult, as each service has its own request ID. Request ID is not
> returned to the caller, so it is not easy to track the request. This
> becomes especially problematic when requests are coming in parallel. For
> example, glance will call cinder for creating image, but that cinder
> instance may be handling several other requests at the same time. By using
> same request ID in the log, user can easily find the cinder request ID that
> is same as glance request ID in the g-api log. It will help
> operators/developers to analyse logs effectively.
>

Thank you for writing this up.


>
>
> To address this issue we have come up with following solutions:
>
>
>
> Solution 1: Return tuple containing headers and body from respective
> clients (also favoured by Joe Gordon)
>
> Reference:
> https://review.openstack.org/#/c/156508/6/specs/log-request-id-mappings.rst
>
>
>
> Pros:
>
> 1. Maintains backward compatibility
>
> 2. Effective debugging/analysing of the problem as both calling service
> request-id and called service request-id are logged in same log message
>
> 3. Build a full call graph
>
> 4. End user will able to know the request-id of the request and can
> approach service provider to know the cause of failure of particular
> request.
>
>
>
> Cons:
>
> 1. The changes need to be done first in cross-projects before making
> changes in clients
>
> 2. Applications which are using python-*clients needs to do required
> changes (check return type of  response)
>

Additional cons:

3. Cannot simply search all logs (ala logstash) using the request-id
returned to the user without any post processing of the logs.


>
>
>
>
> Solution 2:  Use thread local storage to store 'x-openstack-request-id'
> returned from headers (suggested by Doug Hellmann)
>
> Reference:
> https://review.openstack.org/#/c/156508/9/specs/log-request-id-mappings.rst
>
>
>
> Add new method ‘get_openstack_request_id’ to return this request-id to the
> caller.
>
>
>
> Pros:
>
> 1. Doesn’t break compatibility
>
> 2. Minimal changes are required in client
>
> 3. Build a full call graph
>
>
>
> Cons:
>
> 1. Malicious user can send long request-id to fill up the disk-space,
> resulting in potential DoS
>
> 2. Changes need to be done in all python-*clients
>
> 3. Last request id should be flushed out in a subsequent call otherwise it
> will return wrong request id to the caller
>
>
>
>
>
> Solution 3: Unique request-id across OpenStack Services (suggested by
> Jamie Lennox)
>
> Reference:
> https://review.openstack.org/#/c/156508/10/specs/log-request-id-mappings.rst
>
>
>
> Get 'x-openstack-request-id' from auth plugin and add it to the request
> headers. If 'x-openstack-request-id' key is present in the request header,
> then it will use the same one further or else it will generate a new one.
>
>
>
> Dependencies:
>
> https://review.openstack.org/#/c/164582/ - Include request-id in auth
> plugin and add it to request headers
>
> https://review.openstack.org/#/c/166063/ - Add session-object for glance
> client
>
> Add 'UserAuthPlugin' and '_ContextAuthPlugin' same as nova in cinder and
> neutron
>
>
>
>
>
> Pros:
>
> 1. Using same request id for the request crossing multiple service
> boundaries will help operators/developers identify the problem quickly
>
> 2. Required changes only in keystonemiddleware and oslo_middleware
> libraries. No changes are required in the python client bindings or
> OpenStack core services
>
>
>
> Cons:
>
> 1. As 'x-openstack-request-id' in the request header will be visible to
> the user, it is possible to send same request id for multiple requests
> which in turn could create more problems in case of troubleshooting cause
> of the failure as request_id middleware will not check for its uniqueness
> in the scope of the running OpenStack service.
>
> 2. Having the same request ID for all services for a single user API call
> means you cannot generate a full call graph. For example if a single user's
> nova API call produces 2 calls to glance you want to be able to
> differentiate the two different calls.
>
>
>
>
>
> During the Liberty design summit, I had a chance of discussing these
> designs with some of the core members like Doug, Joe Gordon, Jamie Lennox
> etc. But not able to came to any conclusion on the final design and know
> the communities direction by which way they want to use this request-id
> effectively.
>
>
>
> However IMO, solution 1 sounds more useful as the debugger can able to
> build the full call graph which can be helpful for analysing gate failures
> effectively as well as end user will be able to know his request-id and can
> track his request.
>
>
>
> I request all community members to go through these solutions and let us
> know which is the appropriate way to improve the logs by logging request-id.
>
>
>
>
>
> Thanks & Regards,
>
>
>
> Abhishek Kekane
>
> ______________________________________________________________________
> Disclaimer: This email and any attachments are sent in strictest confidence
> for the sole use of the addressee and may contain legally privileged,
> confidential, and proprietary data. If you are not the intended recipient,
> please advise the sender by replying promptly to this email and then delete
> and destroy this email and any attachments without any further use, copying
> or forwarding.
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150527/462baf56/attachment.html>