We recently merged support for placement traits in openstacksdk. Since then, we've seen an uptick in failures of various functional jobs [1]. The failure is always the same test:
openstack.tests.functional.placement.v1.test_trait.TestTrait.test_resource_pr ovider_inventory
That test simply creates a new, custom trait and then attempts to list all traits, show an individual trait, and finally delete the trait. The failure occurs during the first step, creation of the custom trait:
openstack.exceptions.HttpException: HttpException: 502: Server Error for url: https://10.209.100.9/placement/traits/CUSTOM_A982E0BA1C2B4D08BFD6D2594C67831... , Bad Gateway: response from an upstream server.: The proxy server received an invalid: Apache/2.4.52 (Ubuntu) Server at 10.209.100.9 Port 80: Additionally, a 201 Created: 502 Bad Gateway: error was encountered while trying to use an ErrorDocument to handle the request.
I've looked through the various job artefacts and haven't found any smoking guns. I can see placement receive and reply to the request so it would seem something is happening in between.
*Fortunately*, this is also reproducible locally against a standard devstack deployment by running the following in the openstacksdk repo:
OS_TEST_TIMEOUT=60 tox -e functional-py310 -- \ -n openstack/tests/functional/placement/v1/test_trait.py \ --until-failure
Does anyone have any insight into what could be causing this issue and have suggestions for how we might go about debugging it? As things I haven't a clue 😔
Cheers, Stephen
[1] https://zuul.opendev.org/t/openstack/builds?job_name=openstacksdk-functional...