We may capture similar information in auditing and request logging I'm talking about in this thread, but the core differences between these two are purpose and the way to store data. In auditing we may need all auditing data in central place while for logging I expect the information is stored into log file (and then it might be gathered to a central logging system) because the information really needs to be checked together with the other application logs to analyze the application behavior. One possible option is to sue audit middleware with notification disabled (so that all events are recorded in logs) but the information dumped in the log looks huge and some fields like typeURI and addresses may not be required for the debugging purpose. So I think it's better to create a separate one for logging instead of extending scope of the existing audit middleware. (I noticed that audit event does not include request id which we may be able to add later) On 10/8/24 00:31, Michael Johnson wrote:

This sounds a lot like API auditing. Is that maybe what you are looking for?

https://docs.openstack.org/octavia/latest/admin/api-audit.html

Michael

On Mon, Oct 7, 2024 at 3:22 AM Takashi Kajinami <kajinamit@oss.nttdata.com> wrote:

...

On 10/6/24 13:55, Takashi Kajinami wrote:

...

Hello,

(I'm adding tags from teams which may be directly involved with this discussion but also add all tag because this topic is actually a global topic)

I recently spent some time to try neutron deployments with api run by httpd and rcp-server, and I noticed that I no longer see request details logged in neutron logs. It used to be logged by WSGI server implementation in oslo.service (which heavily depends on eventlet and being removed).

I noticed I have to add some more contexts to make my point clear.

So actually we see request logs generated by wsgi server such as apache or uwsgi. (eg. access logs generated by apache)

The problem with these logs from wsgi server is that these logs do not contain items from contexts, such as request id or user id, which is essential to analyze logs for debugging. What I'm proposing here is to log requests in WSGI applications additionally, although some of the information may be dumped twice, because the current one does not provide us enough information.

...

Based on my past experience, the information is quite important when we have to debug any problems triggered by API request (API returned 500, Tried to create a resource but it failed, etc...). WSGI server (uwsgi or apache) may have their own request loggings but the problem with these is that these do not log context details (especially request id) so analyzing logs is still difficult.

I initially created https://bugs.launchpad.net/neutron/+bug/2083760 to discuss this problem in neutron. However, when I looked at a few other repos, I noticed that current implementation is quite inconsistent across projects and now I think this needs to be discussed as a more global topic. IMO the current inconsistency makes debugging quite difficult and we need more consistent logging to help operators get more consistent and detailed information.

1. Some projects (eg. aodh, designate, octavia) does not log requests

2. Other projects (eg. cinder, glance, heat) log requests in their logic but does not log some items such as remote addr and response time.

3. nova[1] and placement[2] have own request_log middleware with more complete information (but one in nova does not have debug log dumped when api starts processing the request)

I'm wondering if it makes sense to import the existing request_log middleware implementation from nova/placement and use it globally (and drop the existing logging with less incomplete information to avoid redundant logs).

Thank you, Takashi

[1] https://github.com/openstack/nova/blob/master/nova/api/openstack/requestlog.... [2] https://github.com/openstack/placement/blob/master/placement/requestlog.py