Open Stack

On 16/02/19 3:42 AM, Florian Engelmann wrote:
> Hi all,
> 
> - Version: heat-base-archive-stable-rocky
> - Commit: Ica99cec6765d22d7ee2262e2d402b2e98cb5bd5e
> 
> 
> I have a fresh openstack deployment (kolla-Ansible). Everything but Heat 
> is working fine.
> 
> 
> When I do a webrequest (either horizon or curl)  on the openstack heat 
> endpoint (internal or public), I just get nothing and after a while, it 
> times out with a 500 http error.

Hold up, 504 is a gateway timeout (presumably from HAProxy). But this 
isn't a 504, it's a 500. And it's not clear if it's heat-api or HAProxy 
that's generating the 500 response.

FWIW, it always pays to set your HAProxy timeout longer than the RPC 
timeout in heat-api, so that if a message gets dropped you'll see that 
reported by heat-api rather than HAProxy.

> root at xxxx-kolla-xxxx:~# curl -vvv 
> http://10.10.10.10:8004/v1/e7f405fb2b7b4b029dfc48e06920eb92
> *   Trying 10.x.y.z
> * Connected to heat.xxxxxxxx (10.x.x.x.) port 8004 (#0)
>  > GET /v1/e7f405fb2b7b4b029dfc48e06920eb92 HTTP/1.1
>  > Host: heat.xxxxxxxxx:8004
>  > User-Agent: curl/7.47.0
>  > Accept: */*
>  >
> 
> < HTTP/1.1 500 Internal Server Error
> < Content-Type: application/json
> < Content-Length: 4338
> < X-Openstack-Request-Id: req-46afc474-682b-4938-8777-b3b4b6fcb973
> < Date: Fri, 15 Feb 2019 13:08:04 GMT
> <
> {"explanation": "The server has either erred or is incapable of 
> performing the requested operation.", "code": 500,
> 
> 
> In the heat_api log I see the request coming through, but it seems that 
> there is just no reply. >> 2019-02-15 14:16:40.047 25 DEBUG 
heat.api.middleware.version_negotiation
> [-] Processing request: GET / Accept: <no header in request> 
> process_request 
> /var/lib/kolla/venv/lib/python2.7/site-packages/heat/api/middleware/version_negotiation.py:50 
> 
> 2019-02-15 14:16:40.048 25 INFO eventlet.wsgi.server [-] 10.x.y.z - - 
> [15/Feb/2019 14:16:40] "GET / HTTP/1.0" 300 327 0.001106

This *is* logging a response though - 300 Multiple Choices, 327 bytes in 
0.001106s. Which is the correct response for "GET /" (it should return 
the version negotiation doc).

> Should the api give some output when I do a http request?

Generally speaking, yes.

> 
> Any hints?
> 
> 
> Thanks a lot, its quite urgent..
> 
> Built 11.0.0 and current rocky-stable (11.0.0.1dev), same on both versions.
> 
> 
> 
> ##
> 
> with a horizon request (just click on Project -> Compute -> 
> Orchestration -> Stacks
> 
> 2019-02-15 14:22:22.250 22 DEBUG heat.api.middleware.version_negotiation 
> [-] Processing request: GET /v1/beb568af3781471d94c3623805946ca3/stacks 
> Accept: application/json process_request 
> /var/lib/kolla/venv/lib/python2.7/site-packages/heat/api/middleware/version_negotiation.py:50 
> 
> 2019-02-15 14:22:22.250 22 DEBUG heat.api.middleware.version_negotiation 
> [-] Matched versioned URI. Version: 1.0 process_request 
> /var/lib/kolla/venv/lib/python2.7/site-packages/heat/api/middleware/version_negotiation.py:65 
> 
> 2019-02-15 14:22:23.003 22 DEBUG eventlet.wsgi.server 
> [req-663d17e3-7e41-4c9e-a30a-a43ef18cf056 - - - - -] (22) accepted 
> ('10.xxx.xxx.xxx', 51084) server 
> /var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/wsgi.py:883
> 2019-02-15 14:22:23.004 22 DEBUG heat.api.middleware.version_negotiation 
> [-] Processing request: GET / Accept: <no header in request> 
> process_request 
> /var/lib/kolla/venv/lib/python2.7/site-packages/heat/api/middleware/version_negotiation.py:50 

There are indeed no responses logged here.

Have you checked the heat-engine log to see if the cause of the delay 
appears there? Is heat-engine alive? Is RabbitMQ working?

- ZB