Cross-posting to the operators list since they are the ones that would care about this. Basically, the "notify_on_api_faults" config option hasn't worked since probably Kilo when the 2.1 microversion wsgi stack code was added. Rackspace added it back in 2012: https://review.openstack.org/#/c/13288/ Gibi has a patch proposed to remove it since it's dead code: https://review.openstack.org/#/c/505164/ Given how long this has been regressed, and no one has noticed, it seems fair to just remove this. Is anyone relying on this and for some reason disagrees with the bug and that we should try to fix this? On 6/20/2017 7:22 AM, Balazs Gibizer wrote: > Hi, > > I come across a questionable behavior of nova while I tried to use the > notify_on_api_faults configuration option [0] while testing the related > versioned notification transformation patch [1]. Based on the > description of the config option and the code that uses it [2] nova > sends and api.fault notification if the nova-api service encounters an > unhandle exception. There is a FaultWrapper class [3] added to the > pipeline of the REST request which catches every exception and calls the > notification sending. > Based on some debugging in devstack this FaultWrapper never catches any > exception. I injected a ValueError to the beginning of > nova.objects.aggregate.Aggregate.create method. This resulted in an > HTTPInternalServerError exception and HTTP 500 error code but the > exception handling part of the FaultWrapper [4] was never reached. So I > dig a bit deeper and I think I found the reason. Every REST API method > is decorated with expected_errors decorator [5] which as a last resort > translate the unexpected exception to HTTPInternalServerError. In the > wsgi stack the actual REST api call is guarded with > ResourceExceptionHandler context manager [7] which translates > HTTPException to a Fault [8]. Then Fault is catched and translated to > the REST response [7]. This way the exception never propagates back to > the FaultWrapper in [6] and therefore the api.fault notification is > never emitted. > > You can see the api logs here [9] and the patch that I used to add the > extra traces here [10]. Please note that there is a compute.exception > notification visible in the log but that is a different notification > emitted from wrap_exception decorator [11] used in compute.manager [12] > and compute.api [13] only. > > So my questions are: > 1) Is it a bug in the nova wsgi or it is expected that the wsgi code > catches everything? > 2) Do we need FaultWrapper at all if the wsgi stack catches every > exception? > 3) Do we need api.fault notification at all? It seems nobody missed it > so far. > 4) If we want to have api.fault notification then what would be the good > place to emit it? Maybe ResourceExceptionHandler at [8]? > > I filed a bug for tracking purposes [14]. > > Cheers, > gibi > > > [0] > https://github.com/openstack/nova/blob/e611116e5822abf0e9f933cf6bd1b4c63007b170/nova/conf/notifications.py#L49 > > [1] https://review.openstack.org/#/c/469038 > [2] > https://github.com/openstack/nova/blob/d68626595ed54698c7eb013a788ee3b98e068cdd/nova/notifications/base.py#L83 > > [3] > https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/__init__.py#L79 > > [4] > https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/__init__.py#L87 > > [5] > https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/extensions.py#L325 > > [6] > https://github.com/openstack/nova/blob/efde7a5dfad24cca361989eadf482899a5cab5db/nova/api/openstack/extensions.py#L368 > > [7] > https://github.com/openstack/nova/blob/4a0fb6ae79acedabf134086d4dce6aae0e4f6209/nova/api/openstack/wsgi.py#L637 > > [8] > https://github.com/openstack/nova/blob/4a0fb6ae79acedabf134086d4dce6aae0e4f6209/nova/api/openstack/wsgi.py#L418 > > [9] https://pastebin.com/Eu6rBjNN > [10] https://pastebin.com/en4aFutc > [11] > https://github.com/openstack/nova/blob/master/nova/exception_wrapper.py#L57 > [12] > https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L105 > [13] https://github.com/openstack/nova/blob/master/nova/compute/api.py#L92 > [14] https://bugs.launchpad.net/nova/+bug/1699115 > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Matt