[openstack-dev] [nova] api.fault notification is never emitted

Balazs Gibizer balazs.gibizer at ericsson.com
Tue Jun 20 12:22:20 UTC 2017


I come across a questionable behavior of nova while I tried to use the 
notify_on_api_faults configuration option [0] while testing the related 
versioned notification transformation patch [1]. Based on the 
description of the config option and the code that uses it [2] nova 
sends and api.fault notification if the nova-api service encounters an 
unhandle exception. There is a FaultWrapper class [3] added to the 
pipeline of the REST request which catches every exception and calls 
the notification sending.
Based on some debugging in devstack this FaultWrapper never catches any 
exception. I injected a ValueError to the beginning of 
nova.objects.aggregate.Aggregate.create method. This resulted in an 
HTTPInternalServerError exception and HTTP 500 error code but the 
exception handling part of the FaultWrapper [4] was never reached. So I 
dig a bit deeper and I think I found the reason. Every REST API method 
is decorated with expected_errors decorator [5] which as a last resort 
translate the unexpected exception to HTTPInternalServerError. In the 
wsgi stack the actual REST api call is guarded with 
ResourceExceptionHandler context manager [7] which translates 
HTTPException to a Fault [8]. Then Fault is catched and translated to 
the REST response [7]. This way the exception never propagates back to 
the FaultWrapper in [6] and therefore the api.fault notification is 
never emitted.

You can see the api logs here [9] and the patch that I used to add the 
extra traces here [10]. Please note that there is a compute.exception 
notification visible in the log but that is a different notification 
emitted from wrap_exception decorator [11] used in compute.manager [12] 
and compute.api [13] only.

So my questions are:
1) Is it a bug in the nova wsgi or it is expected that the wsgi code 
catches everything?
2) Do we need FaultWrapper at all if the wsgi stack catches every 
3) Do we need api.fault notification at all? It seems nobody missed it 
so far.
4) If we want to have api.fault notification then what would be the 
good place to emit it? Maybe ResourceExceptionHandler at [8]?

I filed a bug for tracking purposes [14].


[1] https://review.openstack.org/#/c/469038
[9] https://pastebin.com/Eu6rBjNN
[10] https://pastebin.com/en4aFutc
[14] https://bugs.launchpad.net/nova/+bug/1699115

More information about the OpenStack-dev mailing list