Open Stack

 ---- On Mon, 27 Sep 2021 20:24:18 -0500 melanie witt <melwittt at gmail.com> wrote ----
 > On 9/27/21 15:20, Ghanshyam Mann wrote:
 > > Hello Everyone,
 > > 
 > > All the devstack based jobs are failing with placement URL issue in devstack. The root cause is not known yet.
 > > 
 > > You will see the below error in devstack/summary log-
 > > 
 > > + ./stack.sh:main:1455                     :   /usr/local/bin/nova-status --config-file /etc/nova/nova.conf upgrade check
 > > 
 > > Placement logs:
 > > INFO placement.requestlog [None req-3ad7ab8b-7c85-477a-b156-f5ca652a38cc service placement] 10.208.226.48 "GET /placemen//" status: 404 len: 162 microversion: 1.0
 > 
 > I looked at this for awhile and found that placement is returning a 404 
 > because it fails to match [1] any of the declared routes [2]. This issue 
 > reproduces on a local devstack.
 > 
 > I added some debug print statements to determine this [3]. (The 
 > "ENVIRON" and "NO MATCH FOUND" are the print statements I added in 
 > PlacementHandler.__call__() and dispatch() in placement/handler.py).
 > 
 > Example:
 > 
 > def dispatch(environ, start_response, mapper):
 >      """Find a matching route for the current request.
 > 
 >      If no match is found, raise a 404 response.
 >      If there is a matching route, but no matching handler
 >      for the given method, raise a 405.
 >      """
 >      result = mapper.match(environ=environ)
 >      if result is None:
 >          print('NO MATCH FOUND')
 >          raise webob.exc.HTTPNotFound(
 >              json_formatter=util.json_error_formatter)
 > 
 > In short, the WSGI environment dict is giving a bad/not matching url in 
 > the PATH_INFO variable if the call path is not simply '/'.
 > 
 > Example of a good PATH_INFO:
 > 
 > ENVIRON = {'proxy-sendchunked': '1', 'PATH_INFO': '/', [...] 
 > 'REQUEST_URI': '/placement', 'SCRIPT_NAME': '/placement', [...]}
 > INFO placement.requestlog [None req-bde0b07f-e6a4-4b3d-bb78-51bee95524e1 
 > None None] 127.0.0.1 "GET /placement/" status: 200 len: 136 mi
 > croversion: 1.0
 > 
 > Example of a bad PATH_INFO and SCRIPT_NAME:
 > 
 > ENVIRON = {'proxy-sendchunked': '1', 'PATH_INFO': '//', [...] 
 > 'REQUEST_URI': '/placement/', 'SCRIPT_NAME': '/placemen', [...]}
 > NO MATCH FOUND
 > INFO placement.requestlog [None req-317708d5-8c6f-48f5-abce-225a242bb31a 
 > admin admin] 127.0.0.1 "GET /placemen//" status: 404 len: 198 
 > microversion: 1.29
 > 
 > Another example of bad PATH_INFO and SCRIPT_NAME:
 > 
 > ENVIRON = {'proxy-sendchunked': '1', 'PATH_INFO': 
 > '//resource_providers', [...] 'REQUEST_URI': 
 > '/placement/resource_providers', 'SCRIPT_NAME': '/placemen', [...]}
 > 
 > Note that the REQUEST_URI in all cases looks correct. This supports what 
 > I observed that the clients are calling the correct urls. The same 404s 
 > happen using curl -X GET statements as well. So it's definitely 
 > something wrong in the server.

Yeah.  To find out who is corrupting the PATH_INFO and SCRIPT_NAME,
I tried to trace the environ (could not finish it) in webob layer and while
building the Request in webob, environ variables are already corrupted.

'PATH_INFO': '//resource_providers'
'REQUEST_URI':
 '/placement/resource_providers?in_tree=baba07a8-3c17-474c-82e6-5d63308a1f9d',
 'SCRIPT_NAME': '/placemen', 

Below log is from webob.des.py->__call__method
- https://paste.opendev.org/show/809639/

There is no change in webob recently and as it is only failing in placement (nova, 
keystone request works fine) so webob or wsgi is less doubtful and it might be only
in placement server-side in our Request/Response objects ?

-gmann

 > 
 > I'm not sure how to debug further. I don't know a great deal about WSGI 
 > and where the contents of the environment dict come from. If anyone can 
 > help, it would be appreciated.
 > 
 > Cheers,
 > -melanie
 > 
 > [1] 
 > https://github.com/openstack/placement/blob/bd5b19c00e1ab293fc157f4699bc4f4719731c25/placement/handler.py#L142
 > [2] 
 > https://github.com/openstack/placement/blob/bd5b19c00e1ab293fc157f4699bc4f4719731c25/placement/handler.py#L51
 > [3] https://paste.opendev.org/show/809636
 > 
 > 
 >