On 9/27/21 15:20, Ghanshyam Mann wrote:
Hello Everyone,
All the devstack based jobs are failing with placement URL issue in devstack. The root cause is not known yet.
You will see the below error in devstack/summary log-
+ ./stack.sh:main:1455 : /usr/local/bin/nova-status --config-file /etc/nova/nova.conf upgrade check
Placement logs: INFO placement.requestlog [None req-3ad7ab8b-7c85-477a-b156-f5ca652a38cc service placement] 10.208.226.48 "GET /placemen//" status: 404 len: 162 microversion: 1.0
I looked at this for awhile and found that placement is returning a 404 because it fails to match [1] any of the declared routes [2]. This issue reproduces on a local devstack. I added some debug print statements to determine this [3]. (The "ENVIRON" and "NO MATCH FOUND" are the print statements I added in PlacementHandler.__call__() and dispatch() in placement/handler.py). Example: def dispatch(environ, start_response, mapper): """Find a matching route for the current request. If no match is found, raise a 404 response. If there is a matching route, but no matching handler for the given method, raise a 405. """ result = mapper.match(environ=environ) if result is None: print('NO MATCH FOUND') raise webob.exc.HTTPNotFound( json_formatter=util.json_error_formatter) In short, the WSGI environment dict is giving a bad/not matching url in the PATH_INFO variable if the call path is not simply '/'. Example of a good PATH_INFO: ENVIRON = {'proxy-sendchunked': '1', 'PATH_INFO': '/', [...] 'REQUEST_URI': '/placement', 'SCRIPT_NAME': '/placement', [...]} INFO placement.requestlog [None req-bde0b07f-e6a4-4b3d-bb78-51bee95524e1 None None] 127.0.0.1 "GET /placement/" status: 200 len: 136 mi croversion: 1.0 Example of a bad PATH_INFO and SCRIPT_NAME: ENVIRON = {'proxy-sendchunked': '1', 'PATH_INFO': '//', [...] 'REQUEST_URI': '/placement/', 'SCRIPT_NAME': '/placemen', [...]} NO MATCH FOUND INFO placement.requestlog [None req-317708d5-8c6f-48f5-abce-225a242bb31a admin admin] 127.0.0.1 "GET /placemen//" status: 404 len: 198 microversion: 1.29 Another example of bad PATH_INFO and SCRIPT_NAME: ENVIRON = {'proxy-sendchunked': '1', 'PATH_INFO': '//resource_providers', [...] 'REQUEST_URI': '/placement/resource_providers', 'SCRIPT_NAME': '/placemen', [...]} Note that the REQUEST_URI in all cases looks correct. This supports what I observed that the clients are calling the correct urls. The same 404s happen using curl -X GET statements as well. So it's definitely something wrong in the server. I'm not sure how to debug further. I don't know a great deal about WSGI and where the contents of the environment dict come from. If anyone can help, it would be appreciated. Cheers, -melanie [1] https://github.com/openstack/placement/blob/bd5b19c00e1ab293fc157f4699bc4f47... [2] https://github.com/openstack/placement/blob/bd5b19c00e1ab293fc157f4699bc4f47... [3] https://paste.opendev.org/show/809636