Open Stack

On 9/27/21 15:20, Ghanshyam Mann wrote:
> Hello Everyone,
> 
> All the devstack based jobs are failing with placement URL issue in devstack. The root cause is not known yet.
> 
> You will see the below error in devstack/summary log-
> 
> + ./stack.sh:main:1455                     :   /usr/local/bin/nova-status --config-file /etc/nova/nova.conf upgrade check
> 
> Placement logs:
> INFO placement.requestlog [None req-3ad7ab8b-7c85-477a-b156-f5ca652a38cc service placement] 10.208.226.48 "GET /placemen//" status: 404 len: 162 microversion: 1.0

I looked at this for awhile and found that placement is returning a 404 
because it fails to match [1] any of the declared routes [2]. This issue 
reproduces on a local devstack.

I added some debug print statements to determine this [3]. (The 
"ENVIRON" and "NO MATCH FOUND" are the print statements I added in 
PlacementHandler.__call__() and dispatch() in placement/handler.py).

Example:

def dispatch(environ, start_response, mapper):
     """Find a matching route for the current request.

     If no match is found, raise a 404 response.
     If there is a matching route, but no matching handler
     for the given method, raise a 405.
     """
     result = mapper.match(environ=environ)
     if result is None:
         print('NO MATCH FOUND')
         raise webob.exc.HTTPNotFound(
             json_formatter=util.json_error_formatter)

In short, the WSGI environment dict is giving a bad/not matching url in 
the PATH_INFO variable if the call path is not simply '/'.

Example of a good PATH_INFO:

ENVIRON = {'proxy-sendchunked': '1', 'PATH_INFO': '/', [...] 
'REQUEST_URI': '/placement', 'SCRIPT_NAME': '/placement', [...]}
INFO placement.requestlog [None req-bde0b07f-e6a4-4b3d-bb78-51bee95524e1 
None None] 127.0.0.1 "GET /placement/" status: 200 len: 136 mi
croversion: 1.0

Example of a bad PATH_INFO and SCRIPT_NAME:

ENVIRON = {'proxy-sendchunked': '1', 'PATH_INFO': '//', [...] 
'REQUEST_URI': '/placement/', 'SCRIPT_NAME': '/placemen', [...]}
NO MATCH FOUND
INFO placement.requestlog [None req-317708d5-8c6f-48f5-abce-225a242bb31a 
admin admin] 127.0.0.1 "GET /placemen//" status: 404 len: 198 
microversion: 1.29

Another example of bad PATH_INFO and SCRIPT_NAME:

ENVIRON = {'proxy-sendchunked': '1', 'PATH_INFO': 
'//resource_providers', [...] 'REQUEST_URI': 
'/placement/resource_providers', 'SCRIPT_NAME': '/placemen', [...]}

Note that the REQUEST_URI in all cases looks correct. This supports what 
I observed that the clients are calling the correct urls. The same 404s 
happen using curl -X GET statements as well. So it's definitely 
something wrong in the server.

I'm not sure how to debug further. I don't know a great deal about WSGI 
and where the contents of the environment dict come from. If anyone can 
help, it would be appreciated.

Cheers,
-melanie

[1] 
https://github.com/openstack/placement/blob/bd5b19c00e1ab293fc157f4699bc4f4719731c25/placement/handler.py#L142
[2] 
https://github.com/openstack/placement/blob/bd5b19c00e1ab293fc157f4699bc4f4719731c25/placement/handler.py#L51
[3] https://paste.opendev.org/show/809636