On 16/02/19 3:42 AM, Florian Engelmann wrote:

Hi all,

- Version: heat-base-archive-stable-rocky - Commit: Ica99cec6765d22d7ee2262e2d402b2e98cb5bd5e

I have a fresh openstack deployment (kolla-Ansible). Everything but Heat is working fine.

When I do a webrequest (either horizon or curl)  on the openstack heat endpoint (internal or public), I just get nothing and after a while, it times out with a 500 http error.

Hold up, 504 is a gateway timeout (presumably from HAProxy). But this isn't a 504, it's a 500. And it's not clear if it's heat-api or HAProxy that's generating the 500 response. FWIW, it always pays to set your HAProxy timeout longer than the RPC timeout in heat-api, so that if a message gets dropped you'll see that reported by heat-api rather than HAProxy.

root@xxxx-kolla-xxxx:~# curl -vvv http://10.10.10.10:8004/v1/e7f405fb2b7b4b029dfc48e06920eb92 *   Trying 10.x.y.z * Connected to heat.xxxxxxxx (10.x.x.x.) port 8004 (#0)

...

GET /v1/e7f405fb2b7b4b029dfc48e06920eb92 HTTP/1.1 Host: heat.xxxxxxxxx:8004 User-Agent: curl/7.47.0 Accept: */*

< HTTP/1.1 500 Internal Server Error < Content-Type: application/json < Content-Length: 4338 < X-Openstack-Request-Id: req-46afc474-682b-4938-8777-b3b4b6fcb973 < Date: Fri, 15 Feb 2019 13:08:04 GMT < {"explanation": "The server has either erred or is incapable of performing the requested operation.", "code": 500,

In the heat_api log I see the request coming through, but it seems that there is just no reply. >> 2019-02-15 14:16:40.047 25 DEBUG heat.api.middleware.version_negotiation [-] Processing request: GET / Accept: <no header in request> process_request /var/lib/kolla/venv/lib/python2.7/site-packages/heat/api/middleware/version_negotiation.py:50

2019-02-15 14:16:40.048 25 INFO eventlet.wsgi.server [-] 10.x.y.z - - [15/Feb/2019 14:16:40] "GET / HTTP/1.0" 300 327 0.001106

This *is* logging a response though - 300 Multiple Choices, 327 bytes in 0.001106s. Which is the correct response for "GET /" (it should return the version negotiation doc).

Should the api give some output when I do a http request?

Generally speaking, yes.

Any hints?

Thanks a lot, its quite urgent..

Built 11.0.0 and current rocky-stable (11.0.0.1dev), same on both versions.

##

with a horizon request (just click on Project -> Compute -> Orchestration -> Stacks

2019-02-15 14:22:22.250 22 DEBUG heat.api.middleware.version_negotiation [-] Processing request: GET /v1/beb568af3781471d94c3623805946ca3/stacks Accept: application/json process_request /var/lib/kolla/venv/lib/python2.7/site-packages/heat/api/middleware/version_negotiation.py:50

2019-02-15 14:22:22.250 22 DEBUG heat.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 process_request /var/lib/kolla/venv/lib/python2.7/site-packages/heat/api/middleware/version_negotiation.py:65

2019-02-15 14:22:23.003 22 DEBUG eventlet.wsgi.server [req-663d17e3-7e41-4c9e-a30a-a43ef18cf056 - - - - -] (22) accepted ('10.xxx.xxx.xxx', 51084) server /var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/wsgi.py:883 2019-02-15 14:22:23.004 22 DEBUG heat.api.middleware.version_negotiation [-] Processing request: GET / Accept: <no header in request> process_request /var/lib/kolla/venv/lib/python2.7/site-packages/heat/api/middleware/version_negotiation.py:50

There are indeed no responses logged here. Have you checked the heat-engine log to see if the cause of the delay appears there? Is heat-engine alive? Is RabbitMQ working? - ZB