On 10/6/24 13:55, Takashi Kajinami wrote:
Hello,
(I'm adding tags from teams which may be directly involved with this discussion but also add all tag because this topic is actually a global topic)
I recently spent some time to try neutron deployments with api run by httpd and rcp-server, and I noticed that I no longer see request details logged in neutron logs. It used to be logged by WSGI server implementation in oslo.service (which heavily depends on eventlet and being removed).
I noticed I have to add some more contexts to make my point clear. So actually we see request logs generated by wsgi server such as apache or uwsgi. (eg. access logs generated by apache) The problem with these logs from wsgi server is that these logs do not contain items from contexts, such as request id or user id, which is essential to analyze logs for debugging. What I'm proposing here is to log requests in WSGI applications additionally, although some of the information may be dumped twice, because the current one does not provide us enough information.
Based on my past experience, the information is quite important when we have to debug any problems triggered by API request (API returned 500, Tried to create a resource but it failed, etc...). WSGI server (uwsgi or apache) may have their own request loggings but the problem with these is that these do not log context details (especially request id) so analyzing logs is still difficult.
I initially created https://bugs.launchpad.net/neutron/+bug/2083760 to discuss this problem in neutron. However, when I looked at a few other repos, I noticed that current implementation is quite inconsistent across projects and now I think this needs to be discussed as a more global topic. IMO the current inconsistency makes debugging quite difficult and we need more consistent logging to help operators get more consistent and detailed information.
1. Some projects (eg. aodh, designate, octavia) does not log requests
2. Other projects (eg. cinder, glance, heat) log requests in their logic but does not log some items such as remote addr and response time.
3. nova[1] and placement[2] have own request_log middleware with more complete information (but one in nova does not have debug log dumped when api starts processing the request)
I'm wondering if it makes sense to import the existing request_log middleware implementation from nova/placement and use it globally (and drop the existing logging with less incomplete information to avoid redundant logs).
Thank you, Takashi
[1] https://github.com/openstack/nova/blob/master/nova/api/openstack/requestlog.... [2] https://github.com/openstack/placement/blob/master/placement/requestlog.py