Open Stack

Hi Nikhil,

Thanks for the opinion, actually I was in different time-zone and was not aware about this LogWG. I will certainly approach them in next meeting.

Thank you,

Abhishek Kekane

-----Original Message-----
From: Nikhil Komawar [mailto:nik.komawar at gmail.com] 
Sent: 28 May 2015 12:34
To: openstack-dev at lists.openstack.org
Subject: Re: [openstack-dev] [all] cross project communication: Return request-id to caller

Did you get to talk with anyone in the LogWG ( https://wiki.openstack.org/wiki/LogWorkingGroup )? In wonder what kind of recommendations, standards we can come up with while adopting a cross project solution. If our logs follow certain prefix and or suffix style across projects, that would help a long way.

Personally: +1 on Solution 1

On 5/28/15 2:14 AM, Kekane, Abhishek wrote:
>
> Hi Devs,
>
>  
>
> Thank you for your opinions/thoughts.
>
> However I would like to suggest that please give +1 against the 
> solution which you will like to propose so that at the end it will be 
> helpful for us to consolidate the voting against each solution and 
> make some decision.
>
>  
>
> Thanks in advance.
>
>  
>
> Abhishek Kekane
>
>  
>
>  
>
> *From:*Joe Gordon [mailto:joe.gordon0 at gmail.com]
> *Sent:* 28 May 2015 00:31
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [all] cross project communication:
> Return request-id to caller
>
>  
>
>  
>
>  
>
> On Wed, May 27, 2015 at 12:06 AM, Kekane, Abhishek 
> <Abhishek.Kekane at nttdata.com <mailto:Abhishek.Kekane at nttdata.com>> wrote:
>
> Hi Devs,
>
>  
>
> Each OpenStack service sends a request ID header with HTTP responses.
> This request ID can be useful for tracking down problems in the logs.
> However, when operation crosses service boundaries, this tracking can 
> become difficult, as each service has its own request ID. Request ID 
> is not returned to the caller, so it is not easy to track the request.
> This becomes especially problematic when requests are coming in 
> parallel. For example, glance will call cinder for creating image, but 
> that cinder instance may be handling several other requests at the 
> same time. By using same request ID in the log, user can easily find 
> the cinder request ID that is same as glance request ID in the g-api 
> log. It will help operators/developers to analyse logs effectively.
>
>  
>
> Thank you for writing this up.
>
>  
>
>      
>
>     To address this issue we have come up with following solutions:
>
>      
>
>     Solution 1: Return tuple containing headers and body from
>     respective clients (also favoured by Joe Gordon)
>
>     Reference:
>     
> https://review.openstack.org/#/c/156508/6/specs/log-request-id-mapping
> s.rst
>
>      
>
>     Pros:
>
>     1. Maintains backward compatibility
>
>     2. Effective debugging/analysing of the problem as both calling
>     service request-id and called service request-id are logged in
>     same log message
>
>     3. Build a full call graph
>
>     4. End user will able to know the request-id of the request and
>     can approach service provider to know the cause of failure of
>     particular request.
>
>      
>
>     Cons:
>
>     1. The changes need to be done first in cross-projects before
>     making changes in clients
>
>     2. Applications which are using python-*clients needs to do
>     required changes (check return type of  response)
>
>  
>
> Additional cons:
>
>  
>
> 3. Cannot simply search all logs (ala logstash) using the request-id 
> returned to the user without any post processing of the logs.
>
>  
>
>      
>
>      
>
>     Solution 2:  Use thread local storage to store
>     'x-openstack-request-id' returned from headers (suggested by Doug
>     Hellmann)
>
>     Reference:
>     
> https://review.openstack.org/#/c/156508/9/specs/log-request-id-mapping
> s.rst
>
>      
>
>     Add new method 'get_openstack_request_id' to return this
>     request-id to the caller.
>
>      
>
>     Pros:
>
>     1. Doesn't break compatibility
>
>     2. Minimal changes are required in client
>
>     3. Build a full call graph
>
>      
>
>     Cons:
>
>     1. Malicious user can send long request-id to fill up the
>     disk-space, resulting in potential DoS
>
>     2. Changes need to be done in all python-*clients
>
>     3. Last request id should be flushed out in a subsequent call
>     otherwise it will return wrong request id to the caller
>
>      
>
>      
>
>     Solution 3: Unique request-id across OpenStack Services (suggested
>     by Jamie Lennox)
>
>     Reference:
>     
> https://review.openstack.org/#/c/156508/10/specs/log-request-id-mappin
> gs.rst
>
>      
>
>     Get 'x-openstack-request-id' from auth plugin and add it to the
>     request headers. If 'x-openstack-request-id' key is present in the
>     request header, then it will use the same one further or else it
>     will generate a new one.
>
>      
>
>     Dependencies:
>
>     https://review.openstack.org/#/c/164582/ - Include request-id in
>     auth plugin and add it to request headers
>
>     https://review.openstack.org/#/c/166063/ - Add session-object for
>     glance client
>
>     Add 'UserAuthPlugin' and '_ContextAuthPlugin' same as nova in
>     cinder and neutron
>
>      
>
>      
>
>     Pros:
>
>     1. Using same request id for the request crossing multiple service
>     boundaries will help operators/developers identify the problem 
> quickly
>
>     2. Required changes only in keystonemiddleware and oslo_middleware
>     libraries. No changes are required in the python client bindings
>     or OpenStack core services
>
>      
>
>     Cons:
>
>     1. As 'x-openstack-request-id' in the request header will be
>     visible to the user, it is possible to send same request id for
>     multiple requests which in turn could create more problems in case
>     of troubleshooting cause of the failure as request_id middleware
>     will not check for its uniqueness in the scope of the running
>     OpenStack service.
>
>     2. Having the same request ID for all services for a single user
>     API call means you cannot generate a full call graph. For example
>     if a single user's nova API call produces 2 calls to glance you
>     want to be able to differentiate the two different calls.
>
>      
>
>      
>
>     During the Liberty design summit, I had a chance of discussing
>     these designs with some of the core members like Doug, Joe Gordon,
>     Jamie Lennox etc. But not able to came to any conclusion on the
>     final design and know the communities direction by which way they
>     want to use this request-id effectively.
>
>      
>
>     However IMO, solution 1 sounds more useful as the debugger can
>     able to build the full call graph which can be helpful for
>     analysing gate failures effectively as well as end user will be
>     able to know his request-id and can track his request.
>
>      
>
>     I request all community members to go through these solutions and
>     let us know which is the appropriate way to improve the logs by
>     logging request-id.
>
>      
>
>      
>
>     Thanks & Regards,
>
>      
>
>     Abhishek Kekane
>
>
>     ______________________________________________________________________
>     Disclaimer: This email and any attachments are sent in strictest
>     confidence
>     for the sole use of the addressee and may contain legally privileged,
>     confidential, and proprietary data. If you are not the intended
>     recipient,
>     please advise the sender by replying promptly to this email and
>     then delete
>     and destroy this email and any attachments without any further
>     use, copying
>     or forwarding.
>
>
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>  
>
>
> ______________________________________________________________________
> Disclaimer: This email and any attachments are sent in strictest 
> confidence for the sole use of the addressee and may contain legally 
> privileged, confidential, and proprietary data. If you are not the 
> intended recipient, please advise the sender by replying promptly to 
> this email and then delete and destroy this email and any attachments 
> without any further use, copying or forwarding.
>
>
> ______________________________________________________________________
> ____ OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 

Thanks,
Nikhil


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.