Hi Abhishek, thanks for having a look! I've filed corresponding bug reports: Glance client: https://bugs.launchpad.net/python-glanceclient/+bug/1886650 Glance API: https://bugs.launchpad.net/glance/+bug/1886657 Best regards, Markus Abhishek Kekane wrote:
Hi Markus,
Thank you for detailed analysis. Both cases you pointed out are valid bugs. Could you please report this to launchpad?
Thanks & Best Regards,
Abhishek Kekane
On Fri, Jun 26, 2020 at 6:33 PM Markus Hentsch <markus.hentsch@secustack.com <mailto:markus.hentsch@secustack.com>> wrote:
Hello everyone,
while I was experimenting with the Global Request ID functionality of OpenStack [1], I identified two issues in Glance related to this topic. I have written my findings below and would appreciate it if you could take a look and confirm whether those are intended behaviors or indeed issues with the implementation.
In case of the latter please advice me which bug tracker to report them to.
1. The Glance client does not correctly forward the global ID
When the SessionClient class is used, the global_request_id is removed from kwargs in the constructor using pop() [2]. Directly after this, the parent constructor is called using super(), which in this case is Adapter from the keystoneauth1 library. Therein the global_request_id is set again [3] but since it has been removed from the kwargs, it defaults to None as specified in the Adapter's __init__() header. Thus, the global_request_id passed to the SessionClient constructor never actually makes it to the Glance API. This is in contrast to the HTTPClient class, where get() is used instead of pop() [4].
This can be reproduced simply by creating a server in Nova from an image in Glance, which will attempt to create the Glance client instance using the global_request_id [5]. Passing the "X-Openstack-Request-Id" header during the initial API call for the server creation, makes it visible in Nova (using a suitable "logging_context_format_string" setting) but it's not visible in Glance. Using a Python debugger shows Glance generating a new local ID instead.
2. Glance interprets global ID as local one for Oslo Context objects
While observing the Glance log file, I observed Glance always logging the global_request_id instead of a local one if it is available.
Using "%(global_request_id)s" within "logging_context_format_string"[6] in the glance-api.conf will always print "None" in the logs whereas "%(request_id)s" will either be an ID generated by Glance if no global ID is available or the received global ID.
Culprit seems to be the context middleware of Glance where the global ID in form of the "X-Openstack-Request-Id" header is parsed from the request and passed as "request_id" instead of "global_request_id" to the "glance.context.RequestContext.from_environ()" call [7].
This is in contrast to other services such as Nova or Neutron where the two variables actually print the values according to their name (request_id always being the local one, whereas global_request_id is the global one or None).
[1] https://specs.openstack.org/openstack/oslo-specs/specs/pike/global-req-id.ht... [2] https://github.com/openstack/python-glanceclient/blob/de178ac4382716cc93022b... [3] https://github.com/openstack/keystoneauth/blob/dab8e1057ae8bb9a0e778fb8d3141... [4] https://github.com/openstack/python-glanceclient/blob/de178ac4382716cc93022b... [5] https://github.com/openstack/nova/blob/1cae0cd7229207478b70275509aecd778ca69... [6] https://docs.openstack.org/oslo.context/2.17.0/user/usage.html#context-varia... [7] https://github.com/openstack/glance/blob/e6db0b10a703037f754007bef6f56451086...
Thanks!
Markus
-- Markus Hentsch Team Leader
secustack GmbH - Digital Sovereignty in the Cloud https://www.secustack.com Königsbrücker Straße 96 (Gebäude 30) | 01099 Dresden District Court Dresden, Register Number: HRB 38890
-- Markus Hentsch Team Leader secustack GmbH - Digital Sovereignty in the Cloud https://www.secustack.com Königsbrücker Straße 96 (Gebäude 30) | 01099 Dresden District Court Dresden, Register Number: HRB 38890