Open Stack

We recently merged support for placement traits in openstacksdk. Since then,
we've seen an uptick in failures of various functional jobs [1]. The failure is
always the same test:

   openstack.tests.functional.placement.v1.test_trait.TestTrait.test_resource_pr
   ovider_inventory

That test simply creates a new, custom trait and then attempts to list all
traits, show an individual trait, and finally delete the trait. The failure
occurs during the first step, creation of the custom trait:

   openstack.exceptions.HttpException: HttpException: 502: Server Error for url:
   https://10.209.100.9/placement/traits/CUSTOM_A982E0BA1C2B4D08BFD6D2594C678313
   , Bad Gateway: response from an upstream server.: The proxy server received
   an invalid: Apache/2.4.52 (Ubuntu) Server at 10.209.100.9 Port 80:
   Additionally, a 201 Created: 502 Bad Gateway: error was encountered while
   trying to use an ErrorDocument to handle the request.
   
I've looked through the various job artefacts and haven't found any smoking
guns. I can see placement receive and reply to the request so it would seem
something is happening in between.

*Fortunately*, this is also reproducible locally against a standard devstack
deployment by running the following in the openstacksdk repo:

   OS_TEST_TIMEOUT=60 tox -e functional-py310 -- \
     -n openstack/tests/functional/placement/v1/test_trait.py \
     --until-failure

Does anyone have any insight into what could be causing this issue and have
suggestions for how we might go about debugging it? As things I haven't a clue
😔

Cheers,
Stephen

[1] https://zuul.opendev.org/t/openstack/builds?job_name=openstacksdk-functional-devstack&project=openstack%2Fdevstack&branch=master&skip=0