<div><span style="font-size: 15px;">Doesn’t this overlap with the work done for the OSProfiler ? </span></div><div><br></div><div><br></div><div><span style="font-size: 15px;">More comments inline.</span></div>
<div><div><br></div><div><span style="font-size: 10pt;">Miguel Ángel Ajo</span></div><div><br></div></div>
<p style="color: #A0A0A8;">On Wednesday, 3 de June de 2015 at 11:43, Kekane, Abhishek wrote:</p>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;">
<span><div><div><div>Hi Devs,</div><div><br></div><div>So for I have got following responses on the proposed solutions:</div><div><br></div><div>Solution 1: Return tuple containing headers and body from - 3 +1</div><div>Solution 2: Use thread local storage to store 'x-openstack-request-id' returned from headers - 0 +1</div><div>Solution 3: Unique request-id across OpenStack Services - 1 +1</div></div></div></span></blockquote><div><br></div><div><span style="font-size: 15px;">I’d vote for Solution 3, without involving keystone (first caller with no req-id generates one randomly),</span></div><div><span style="font-size: 15px;">the req-id contains a call/hop count, which is incremented on every new call...</span> </div><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div><div><div> </div></div></div></span></blockquote><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div><br></div><div>Requesting community people, cross-project members and PTL's to go through this mailing thread [1] and give your suggestions/opinions about the solutions proposed so that It will be easy to finalize the solution.</div><div><br></div><div>[1] <a href="http://lists.openstack.org/pipermail/openstack-dev/2015-May/064842.html">http://lists.openstack.org/pipermail/openstack-dev/2015-May/064842.html</a></div><div><br></div><div>Thanks & Regards,</div><div><br></div><div>Abhishek Kekane</div><div><br></div><div>-----Original Message-----</div><div>From: Nikhil Komawar [<a href="mailto:nik.komawar@gmail.com">mailto:nik.komawar@gmail.com</a>] </div><div>Sent: 28 May 2015 12:34</div><div>To: <a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a></div><div>Subject: Re: [openstack-dev] [all] cross project communication: Return request-id to caller</div><div><br></div><div>Did you get to talk with anyone in the LogWG ( <a href="https://wiki.openstack.org/wiki/LogWorkingGroup">https://wiki.openstack.org/wiki/LogWorkingGroup</a> )? In wonder what kind of recommendations, standards we can come up with while adopting a cross project solution. If our logs follow certain prefix and or suffix style across projects, that would help a long way.</div><div><br></div><div>Personally: +1 on Solution 1</div><div><br></div><div>On 5/28/15 2:14 AM, Kekane, Abhishek wrote:</div><blockquote type="cite"><div><br></div><div>Hi Devs,</div><div><br></div><div> </div><div><br></div><div>Thank you for your opinions/thoughts.</div><div><br></div><div>However I would like to suggest that please give +1 against the </div><div>solution which you will like to propose so that at the end it will be </div><div>helpful for us to consolidate the voting against each solution and </div><div>make some decision.</div><div><br></div><div> </div><div><br></div><div>Thanks in advance.</div><div><br></div><div> </div><div><br></div><div>Abhishek Kekane</div><div><br></div><div> </div><div><br></div><div> </div><div><br></div><div>*From:*Joe Gordon [<a href="mailto:joe.gordon0@gmail.com">mailto:joe.gordon0@gmail.com</a>]</div><div>*Sent:* 28 May 2015 00:31</div><div>*To:* OpenStack Development Mailing List (not for usage questions)</div><div>*Subject:* Re: [openstack-dev] [all] cross project communication:</div><div>Return request-id to caller</div><div><br></div><div> </div><div><br></div><div> </div><div><br></div><div> </div><div><br></div><div>On Wed, May 27, 2015 at 12:06 AM, Kekane, Abhishek </div><div><Abhishek.Kekane@nttdata.com <<a href="mailto:Abhishek.Kekane@nttdata.com">mailto:Abhishek.Kekane@nttdata.com</a>>> wrote:</div><div><br></div><div>Hi Devs,</div><div><br></div><div> </div><div><br></div><div>Each OpenStack service sends a request ID header with HTTP responses.</div><div>This request ID can be useful for tracking down problems in the logs.</div><div>However, when operation crosses service boundaries, this tracking can </div><div>become difficult, as each service has its own request ID. Request ID </div><div>is not returned to the caller, so it is not easy to track the request.</div><div>This becomes especially problematic when requests are coming in </div><div>parallel. For example, glance will call cinder for creating image, but </div><div>that cinder instance may be handling several other requests at the </div><div>same time. By using same request ID in the log, user can easily find </div><div>the cinder request ID that is same as glance request ID in the g-api </div><div>log. It will help operators/developers to analyse logs effectively.</div><div><br></div><div> </div><div><br></div><div>Thank you for writing this up.</div><div><br></div><div> </div><div><br></div><div> </div><div><br></div><div> To address this issue we have come up with following solutions:</div><div><br></div><div> </div><div><br></div><div> Solution 1: Return tuple containing headers and body from</div><div> respective clients (also favoured by Joe Gordon)</div><div><br></div><div> Reference:</div><div> </div><div><a href="https://review.openstack.org/#/c/156508/6/specs/log-request-id-mapping">https://review.openstack.org/#/c/156508/6/specs/log-request-id-mapping</a></div><div>s.rst</div><div><br></div><div> </div><div><br></div><div> Pros:</div><div><br></div><div> 1. Maintains backward compatibility</div><div><br></div><div> 2. Effective debugging/analysing of the problem as both calling</div><div> service request-id and called service request-id are logged in</div><div> same log message</div><div><br></div><div> 3. Build a full call graph</div><div><br></div><div> 4. End user will able to know the request-id of the request and</div><div> can approach service provider to know the cause of failure of</div><div> particular request.</div><div><br></div><div> </div><div><br></div><div> Cons:</div><div><br></div><div> 1. The changes need to be done first in cross-projects before</div><div> making changes in clients</div><div><br></div><div> 2. Applications which are using python-*clients needs to do</div><div> required changes (check return type of response)</div><div><br></div><div> </div><div><br></div><div>Additional cons:</div><div><br></div><div> </div><div><br></div><div>3. Cannot simply search all logs (ala logstash) using the request-id </div><div>returned to the user without any post processing of the logs.</div></blockquote></span></blockquote><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div><div><blockquote type="cite"><div><div></div><div> </div><div><br></div><div> </div><div><br></div><div> </div><div><br></div><div> Solution 2: Use thread local storage to store</div><div> 'x-openstack-request-id' returned from headers (suggested by Doug</div><div> Hellmann)</div><div><br></div><div> Reference:</div><div> </div><div><a href="https://review.openstack.org/#/c/156508/9/specs/log-request-id-mapping">https://review.openstack.org/#/c/156508/9/specs/log-request-id-mapping</a></div><div>s.rst</div><div><br></div><div> </div><div><br></div><div> Add new method 'get_openstack_request_id' to return this</div><div> request-id to the caller.</div><div><br></div></div></blockquote></div></div></span></blockquote><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div><div><blockquote type="cite"><div><div></div><div> </div><div><br></div><div> Pros:</div><div><br></div><div> 1. Doesn't break compatibility</div><div><br></div><div> 2. Minimal changes are required in client</div><div><br></div><div> 3. Build a full call graph</div><div><br></div><div> </div><div><br></div><div> Cons:</div><div><br></div><div> 1. Malicious user can send long request-id to fill up the</div><div> disk-space, resulting in potential DoS</div></div></blockquote></div></div></span></blockquote><div><br></div><div><span style="font-size: 15px;">Can’t this be easily mitigated by limiting the request-id size,</span></div><div><span style="font-size: 15px;">and denying any wrong request-id?</span></div><div> </div><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div><div><blockquote type="cite"><div><div><br></div><div> 2. Changes need to be done in all python-*clients</div><div><br></div><div> 3. Last request id should be flushed out in a subsequent call</div><div> otherwise it will return wrong request id to the caller</div><div><br></div><div> </div><div><br></div><div> </div><div><br></div><div> Solution 3: Unique request-id across OpenStack Services (suggested</div><div> by Jamie Lennox)</div><div><br></div><div> Reference:</div><div> </div><div><a href="https://review.openstack.org/#/c/156508/10/specs/log-request-id-mappin">https://review.openstack.org/#/c/156508/10/specs/log-request-id-mappin</a></div><div>gs.rst</div><div><br></div><div> </div><div><br></div><div> Get 'x-openstack-request-id' from auth plugin and add it to the</div><div> request headers. If 'x-openstack-request-id' key is present in the</div><div> request header, then it will use the same one further or else it</div><div> will generate a new one.</div><div><br></div><div> </div><div><br></div><div> Dependencies:</div><div><br></div><div> <a href="https://review.openstack.org/#/c/164582/">https://review.openstack.org/#/c/164582/</a> - Include request-id in</div><div> auth plugin and add it to request headers</div><div><br></div><div> <a href="https://review.openstack.org/#/c/166063/">https://review.openstack.org/#/c/166063/</a> - Add session-object for</div><div> glance client</div><div><br></div><div> Add 'UserAuthPlugin' and '_ContextAuthPlugin' same as nova in</div><div> cinder and neutron</div><div><br></div><div> </div><div><br></div><div> </div><div><br></div><div> Pros:</div><div><br></div><div> 1. Using same request id for the request crossing multiple service</div><div> boundaries will help operators/developers identify the problem </div><div>quickly</div><div><br></div><div> 2. Required changes only in keystonemiddleware and oslo_middleware</div><div> libraries. No changes are required in the python client bindings</div><div> or OpenStack core services</div></div></blockquote></div></div></span></blockquote><div><br></div><div><span style="font-size: 15px;">Why do we need to call keystone to get a request ID?, can’t we just</span></div><div><span style="font-size: 15px;">simply generate a random one if no req-id is provided in headers?</span></div><div><span style="font-size: 15px;"><br></span></div><div><span style="font-size: 15px;">I believe calling keystone to start a new request is a bottleneck, and that’s</span></div><div><span style="font-size: 15px;">not generally necessary when you have an auth token already...</span></div><div> </div><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div><div><blockquote type="cite"><div><div><br></div><div> </div><div><br></div><div> Cons:</div><div><br></div><div> 1. As 'x-openstack-request-id' in the request header will be</div><div> visible to the user, it is possible to send same request id for</div><div> multiple requests which in turn could create more problems in case</div><div> of troubleshooting cause of the failure as request_id middleware</div><div> will not check for its uniqueness in the scope of the running</div><div> OpenStack service.</div><div><br></div><div> 2. Having the same request ID for all services for a single user</div><div> API call means you cannot generate a full call graph. For example</div><div> if a single user's nova API call produces 2 calls to glance you</div><div> want to be able to differentiate the two different calls.</div><div><br></div><div> </div><div><br></div></div></blockquote></div></div></span></blockquote><div><span style="font-size: 15px;">Why not tagging the request ID with the call number?</span></div><div><span style="font-size: 15px;"><br></span></div><div><span style="font-size: 15px;"><req-id>/1 , <req-id>/2, <req-id>/3 … then it’s very easy</span></div><div><span style="font-size: 15px;">to trace order.</span></div><div><br></div><div> </div><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div><div><blockquote type="cite"><div><div></div><div> </div><div><br></div><div> During the Liberty design summit, I had a chance of discussing</div><div> these designs with some of the core members like Doug, Joe Gordon,</div><div> Jamie Lennox etc. But not able to came to any conclusion on the</div><div> final design and know the communities direction by which way they</div><div> want to use this request-id effectively.</div><div><br></div><div> </div><div><br></div><div> However IMO, solution 1 sounds more useful as the debugger can</div><div> able to build the full call graph which can be helpful for</div><div> analysing gate failures effectively as well as end user will be</div><div> able to know his request-id and can track his request.</div><div><br></div><div> </div><div><br></div><div> I request all community members to go through these solutions and</div><div> let us know which is the appropriate way to improve the logs by</div><div> logging request-id.</div><div><br></div><div> </div><div><br></div><div> </div><div><br></div><div> Thanks & Regards,</div><div><br></div><div> </div><div><br></div><div> Abhishek Kekane</div><div><br></div><div><br></div><div> ______________________________________________________________________</div><div> Disclaimer: This email and any attachments are sent in strictest</div><div> confidence</div><div> for the sole use of the addressee and may contain legally privileged,</div><div> confidential, and proprietary data. If you are not the intended</div><div> recipient,</div><div> please advise the sender by replying promptly to this email and</div><div> then delete</div><div> and destroy this email and any attachments without any further</div><div> use, copying</div><div> or forwarding.</div><div><br></div><div><br></div><div> __________________________________________________________________________</div><div> OpenStack Development Mailing List (not for usage questions)</div><div> Unsubscribe:</div><div> <a href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a></div><div> <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>></div><div> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div><div><br></div><div> </div><div><br></div><div><br></div><div>______________________________________________________________________</div><div>Disclaimer: This email and any attachments are sent in strictest </div><div>confidence for the sole use of the addressee and may contain legally </div><div>privileged, confidential, and proprietary data. If you are not the </div><div>intended recipient, please advise the sender by replying promptly to </div><div>this email and then delete and destroy this email and any attachments </div><div>without any further use, copying or forwarding.</div><div><br></div><div><br></div><div>______________________________________________________________________</div><div>____ OpenStack Development Mailing List (not for usage questions)</div><div>Unsubscribe: </div><div><a href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a></div><div><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div></div></blockquote><div><br></div><div>-- </div><div><br></div><div>Thanks,</div><div>Nikhil</div><div><br></div><div><br></div><div>__________________________________________________________________________</div><div>OpenStack Development Mailing List (not for usage questions)</div><div>Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a></div><div><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div><div><br></div><div>______________________________________________________________________</div><div>Disclaimer: This email and any attachments are sent in strictest confidence</div><div>for the sole use of the addressee and may contain legally privileged,</div><div>confidential, and proprietary data. If you are not the intended recipient,</div><div>please advise the sender by replying promptly to this email and then delete</div><div>and destroy this email and any attachments without any further use, copying</div><div>or forwarding.</div><div><br></div><div>__________________________________________________________________________</div><div>OpenStack Development Mailing List (not for usage questions)</div><div>Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a></div><div><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div></div></div></span>
</blockquote>
<div>
<br>
</div>