Hello everyone, My colleagues and I are maintaining a newton-based openstack installation. We use linux containers on ubuntu 16.04 nodes for service separation and HA. After switiching to HTTPS + Domain for all our Endpoints (was http and IP-based before) we face the problem of MessagingTimeout for os-getVNCConsole-Requests. In (1) you can find the Stack Trace we get on a request for VNCConsole Access. We have traced this down to the point were we actually tcpdumped the communication and we can see the response message arriving back(2). The timeline looks like: - 13:37:22 GMT - Sending request via nova get-vnc-console a29f9b95-055e-4f30-8a30-ac0bcde32dd1 novnc - xx:xx:xx - Arrival at RabbitMQ - 14:37:22.806 CET - Compute Hosts gets the request - 14:37:22.810 CET - Compute Hosts sends response to RabbitMQ - xx:xx:xx - Response enters RabbitMQ - 14:37:22.812 CET - tcpdumped response at the interface of the control node where the nova cloud controller resides on, who send the request - 13:38:22.826 UTC - Timeout /var/log/nova/nova-api-os-compute.log It could be expected, that the response messages actually arrives at the requesting process. However, I can not narrow it down to exactly this point, because - I can't tcpdump in an lxd container - The system is highly productive and possibilities for debugging are limited - We have no other problems with this installation in any matter, so a global problem can be excluded We are actually not sure if the mismatch in timezones acutally leads to a problem. The fact, that our problem is isolated and all other communication runs fine makes us believe it's not. Taking the returned "access_url" from the sniffed package(2) and e.g. pasting it to a browser, we get the Standard VNC Console Screen. This shows the general communication via port 6080 is also possible. Nonetheless, even doing the as soon as the package was sniffed I never got an actual novnc-session to the VM. We are stuck at this point, because we simply can't completely understand the python code :( I hope you can ask some helping questions to get us a step further. You can find a list of our current dpkg and pip packages in (3). Questions we are currently able formulate are: - Is there any communication taking place besides the regular ports (8774, 6080, 5672) in the process? - are the generated access_urls verified in any way after receiving the response message from RabbitMQ, which could lead to an exception being thrown? - are we looking at the right communication flow (https://docs.openstack.org/nova/pike/_images/SCH_5009_V00_NUAC-VNC_OpenStack..., there is no one for newton!?) at all? Best regards Ralf Teckelmann (1) 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 338, in wrapped 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/remote_consoles.py", line 56, in get_vnc_console 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions console_type) 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 157, in wrapped 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions return function(self, context, instance, *args, **kwargs) 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3283, in get_vnc_console 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions access_url=connect_info['access_url']) 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/consoleauth/rpcapi.py", line 89, in authorize_console 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions return cctxt.call(ctxt, 'authorize_console', **msg_args) 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions retry=self.retry) 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry) 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions retry=retry) 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 453, in _send 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions result = self._waiter.wait(msg_id, timeout) 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in wait 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions message = self.waiters.get(msg_id, timeout=timeout) 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 238, in get 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions 'to message ID %s' % msg_id) 2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID bd66ed7b4fe44f5c8527573081379639 (2) IP addresses are obfuscate in order to meet company regulation 2019-03-29 14:37:22.812563 IP (tos 0x0, ttl 64, id 36768, offset 0, flags [DF], proto TCP (6), length 662) x.31.129.217.5672 > x.31.129.206.45604: Flags [P.], cksum 0x83a3 (correct), seq 959640600:959641210, ack 3516429040, win 243, options [nop,nop,TS val 3070303066 ecr 3070033633], length 610 E.....@.@.L..........(.$92....r............ ...Z..........].<.<.1.......N.&reply_d5cb71955a3a4a39b1e8f6539a72be4f&reply_d5cb71955a3a4a39b1e8f6539a72be4f.......+.<.............application/json.utf-8..............{"oslo.message": "{\"_msg_id\": \"ae4ea417e79c43af9d1cd2d603b32076\", \"failure\": null, \"_unique_id\": \"e357642aaef34d1fbaa4d240c405d572\", \"result\": {\"internal_access_path\": null, \"host\": \"x.31.129.164\", \"access_url\": \"https://api.secret.cloud:6080/vnc_auto.html?token=44b7d9be-2aab-4239-914c-971d615fa9c0\", \"token\": \"44b7d9be-2aab-4239-914c-971d615fa9c0\", \"port\": \"5912\"}, \"ending\": true}", "oslo.version": "2.0"}. (3) :~# dpkg -l | grep amqp ii python-amqp 1.4.9-1 all Low-level AMQP client :~# dpkg -l | grep oslo.messaging ii python-oslo.messaging 5.10.0-0ubuntu2~cloud0 all oslo messaging library - Python 2.x :~# dpkg -l | grep nova ii nova-api-os-compute 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute - OpenStack Compute API frontend ii nova-cert 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute - certificate management ii nova-common 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute - common files ii nova-conductor 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute - conductor service ii nova-consoleauth 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute - Console Authenticator ii nova-novncproxy 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute - NoVNC proxy ii nova-scheduler 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute - virtual machine scheduler ii python-nova 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute Python libraries ii python-novaclient 2:6.0.0-0ubuntu1~cloud0 all client library for OpenStack Compute API - Python 2.7 :~# pip list | grep amqp amqp (1.4.9) :~# pip list | grep oslo oslo.cache (1.14.0) oslo.concurrency (3.14.0) oslo.config (3.17.0) oslo.context (2.9.0) oslo.db (4.13.3) oslo.i18n (3.9.0) oslo.log (3.16.0) oslo.messaging (5.10.0) oslo.middleware (3.19.0) oslo.policy (1.14.0) oslo.privsep (1.13.0) oslo.reports (1.14.0) oslo.rootwrap (5.1.0) oslo.serialization (2.13.0) oslo.service (1.16.0) oslo.utils (3.16.0) oslo.versionedobjects (1.17.0)