[openstack-dev] [nova] api.fault notification is never emitted

Matt Riedemann mriedemos at gmail.com
Tue Sep 26 17:15:47 UTC 2017


Cross-posting to the operators list since they are the ones that would 
care about this.

Basically, the "notify_on_api_faults" config option hasn't worked since 
probably Kilo when the 2.1 microversion wsgi stack code was added.

Rackspace added it back in 2012:

https://review.openstack.org/#/c/13288/

Gibi has a patch proposed to remove it since it's dead code:

https://review.openstack.org/#/c/505164/

Given how long this has been regressed, and no one has noticed, it seems 
fair to just remove this.

Is anyone relying on this and for some reason disagrees with the bug and 
that we should try to fix this?

On 6/20/2017 7:22 AM, Balazs Gibizer wrote:
> Hi,
> 
> I come across a questionable behavior of nova while I tried to use the 
> notify_on_api_faults configuration option [0] while testing the related 
> versioned notification transformation patch [1]. Based on the 
> description of the config option and the code that uses it [2] nova 
> sends and api.fault notification if the nova-api service encounters an 
> unhandle exception. There is a FaultWrapper class [3] added to the 
> pipeline of the REST request which catches every exception and calls the 
> notification sending.
> Based on some debugging in devstack this FaultWrapper never catches any 
> exception. I injected a ValueError to the beginning of 
> nova.objects.aggregate.Aggregate.create method. This resulted in an 
> HTTPInternalServerError exception and HTTP 500 error code but the 
> exception handling part of the FaultWrapper [4] was never reached. So I 
> dig a bit deeper and I think I found the reason. Every REST API method 
> is decorated with expected_errors decorator [5] which as a last resort 
> translate the unexpected exception to HTTPInternalServerError. In the 
> wsgi stack the actual REST api call is guarded with 
> ResourceExceptionHandler context manager [7] which translates 
> HTTPException to a Fault [8]. Then Fault is catched and translated to 
> the REST response [7]. This way the exception never propagates back to 
> the FaultWrapper in [6] and therefore the api.fault notification is 
> never emitted.
> 
> You can see the api logs here [9] and the patch that I used to add the 
> extra traces here [10]. Please note that there is a compute.exception 
> notification visible in the log but that is a different notification 
> emitted from wrap_exception decorator [11] used in compute.manager [12] 
> and compute.api [13] only.
> 
> So my questions are:
> 1) Is it a bug in the nova wsgi or it is expected that the wsgi code 
> catches everything?
> 2) Do we need FaultWrapper at all if the wsgi stack catches every 
> exception?
> 3) Do we need api.fault notification at all? It seems nobody missed it 
> so far.
> 4) If we want to have api.fault notification then what would be the good 
> place to emit it? Maybe ResourceExceptionHandler at [8]?
> 
> I filed a bug for tracking purposes [14].
> 
> Cheers,
> gibi
> 
> 
> [0] 
> https://github.com/openstack/nova/blob/e611116e5822abf0e9f933cf6bd1b4c63007b170/nova/conf/notifications.py#L49 
> 
> [1] https://review.openstack.org/#/c/469038
> [2] 
> https://github.com/openstack/nova/blob/d68626595ed54698c7eb013a788ee3b98e068cdd/nova/notifications/base.py#L83 
> 
> [3] 
> https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/__init__.py#L79 
> 
> [4] 
> https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/__init__.py#L87 
> 
> [5] 
> https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/extensions.py#L325 
> 
> [6] 
> https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/extensions.py#L368 
> 
> [7] 
> https://github.com/openstack/nova/blob/4a0fb6ae79acedabf134086d4dce6aae0e4f6209/nova/api/openstack/wsgi.py#L637 
> 
> [8] 
> https://github.com/openstack/nova/blob/4a0fb6ae79acedabf134086d4dce6aae0e4f6209/nova/api/openstack/wsgi.py#L418 
> 
> [9] https://pastebin.com/Eu6rBjNN
> [10] https://pastebin.com/en4aFutc
> [11] 
> https://github.com/openstack/nova/blob/master/nova/exception_wrapper.py#L57
> [12] 
> https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L105
> [13] https://github.com/openstack/nova/blob/master/nova/compute/api.py#L92
> [14] https://bugs.launchpad.net/nova/+bug/1699115
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-- 

Thanks,

Matt



More information about the OpenStack-dev mailing list