Hi, I come across a questionable behavior of nova while I tried to use the notify_on_api_faults configuration option [0] while testing the related versioned notification transformation patch [1]. Based on the description of the config option and the code that uses it [2] nova sends and api.fault notification if the nova-api service encounters an unhandle exception. There is a FaultWrapper class [3] added to the pipeline of the REST request which catches every exception and calls the notification sending. Based on some debugging in devstack this FaultWrapper never catches any exception. I injected a ValueError to the beginning of nova.objects.aggregate.Aggregate.create method. This resulted in an HTTPInternalServerError exception and HTTP 500 error code but the exception handling part of the FaultWrapper [4] was never reached. So I dig a bit deeper and I think I found the reason. Every REST API method is decorated with expected_errors decorator [5] which as a last resort translate the unexpected exception to HTTPInternalServerError. In the wsgi stack the actual REST api call is guarded with ResourceExceptionHandler context manager [7] which translates HTTPException to a Fault [8]. Then Fault is catched and translated to the REST response [7]. This way the exception never propagates back to the FaultWrapper in [6] and therefore the api.fault notification is never emitted. You can see the api logs here [9] and the patch that I used to add the extra traces here [10]. Please note that there is a compute.exception notification visible in the log but that is a different notification emitted from wrap_exception decorator [11] used in compute.manager [12] and compute.api [13] only. So my questions are: 1) Is it a bug in the nova wsgi or it is expected that the wsgi code catches everything? 2) Do we need FaultWrapper at all if the wsgi stack catches every exception? 3) Do we need api.fault notification at all? It seems nobody missed it so far. 4) If we want to have api.fault notification then what would be the good place to emit it? Maybe ResourceExceptionHandler at [8]? I filed a bug for tracking purposes [14]. Cheers, gibi [0] https://github.com/openstack/nova/blob/e611116e5822abf0e9f933cf6bd1b4c63007b170/nova/conf/notifications.py#L49 [1] https://review.openstack.org/#/c/469038 [2] https://github.com/openstack/nova/blob/d68626595ed54698c7eb013a788ee3b98e068cdd/nova/notifications/base.py#L83 [3] https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/__init__.py#L79 [4] https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/__init__.py#L87 [5] https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/extensions.py#L325 [6] https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/extensions.py#L368 [7] https://github.com/openstack/nova/blob/4a0fb6ae79acedabf134086d4dce6aae0e4f6209/nova/api/openstack/wsgi.py#L637 [8] https://github.com/openstack/nova/blob/4a0fb6ae79acedabf134086d4dce6aae0e4f6209/nova/api/openstack/wsgi.py#L418 [9] https://pastebin.com/Eu6rBjNN [10] https://pastebin.com/en4aFutc [11] https://github.com/openstack/nova/blob/master/nova/exception_wrapper.py#L57 [12] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L105 [13] https://github.com/openstack/nova/blob/master/nova/compute/api.py#L92 [14] https://bugs.launchpad.net/nova/+bug/1699115