<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<div style="color: rgb(0, 0, 0);">
<div dir="ltr">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top:0; margin-bottom:0"></p>
Hello everyone,<br>
<div><br>
My colleagues and I are maintaining a newton-based openstack installation.<br>
We use linux containers on ubuntu 16.04 nodes for service separation and HA.<br>
After switiching to HTTPS + Domain for all our Endpoints (was http and IP-based before) we face the problem of MessagingTimeout for os-getVNCConsole-Requests.<br>
<br>
In (1) you can find the Stack Trace we get on a request for VNCConsole Access.<br>
We have traced this down to the point were we actually tcpdumped the communication and we can see the response message arriving back(2).
<br>
</div>
<div><br>
</div>
<div>The timeline looks like:<br>
<br>
- 13:37:22 GMT - Sending request via nova get-vnc-console a29f9b95-055e-4f30-8a30-ac0bcde32dd1 novnc<br>
- xx:xx:xx - Arrival at RabbitMQ<br>
- 14:37:22.806 CET - Compute Hosts gets the request<br>
- 14:37:22.810 CET - Compute Hosts sends response to RabbitMQ<br>
- xx:xx:xx - Response enters RabbitMQ<br>
- 14:37:22.812 CET - tcpdumped response at the interface of the control node where the nova cloud controller resides on, who send the request<br>
- 13:38:22.826 UTC - Timeout /var/log/nova/nova-api-os-compute.log<br>
<br>
It could be expected, that the response messages actually arrives at the requesting process.
<br>
However, I can not narrow it down to exactly this point, because<br>
- I can't tcpdump in an lxd container<br>
- The system is highly productive and possibilities for debugging are limited<br>
- We have no other problems with this installation in any matter, so a global problem can be excluded<br>
<br>
We are actually not sure if the mismatch in timezones acutally leads to a problem.
<br>
The fact, that our problem is isolated and all other communication runs fine makes us believe it's not.<br>
<br>
Taking the returned "access_url" from the sniffed package(2) and e.g. pasting it to a browser, we get the Standard VNC Console Screen.<br>
This shows the general communication via port 6080 is also possible.<br>
Nonetheless, even doing the as soon as the package was sniffed I never got an actual novnc-session to the VM.<br>
<br>
We are stuck at this point, because we simply can't completely understand the python code :(<br>
I hope you can ask some helping questions to get us a step further.<br>
<br>
You can find a list of our current dpkg and pip packages in (3).<br>
<br>
Questions we are currently able formulate are:<br>
- Is there any communication taking place besides the regular ports (8774, 6080, 5672) in the process?<br>
- are the generated access_urls verified in any way after receiving the response message from RabbitMQ, which could lead to an exception being thrown?<br>
- are we looking at the right communication flow (https://docs.openstack.org/nova/pike/_images/SCH_5009_V00_NUAC-VNC_OpenStack.png, there is no one for newton!?) at all?<br>
<br>
Best regards<br>
<br>
Ralf Teckelmann<br>
<br>
(1)<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions Traceback (most recent call last):<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 338, in wrapped<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions return f(*args, **kwargs)<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions return func(*args, **kwargs)<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/remote_consoles.py", line 56, in get_vnc_console<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions console_type)<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 157, in wrapped<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions return function(self, context, instance, *args, **kwargs)<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3283, in get_vnc_console<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions access_url=connect_info['access_url'])<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/consoleauth/rpcapi.py", line 89, in authorize_console<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions return cctxt.call(ctxt, 'authorize_console', **msg_args)<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions retry=self.retry)<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry)<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions retry=retry)<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 453, in _send<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions result = self._waiter.wait(msg_id, timeout)<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in wait<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions message = self.waiters.get(msg_id, timeout=timeout)<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 238, in get<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions 'to message ID %s' % msg_id)<br>
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID bd66ed7b4fe44f5c8527573081379639<br>
<br>
(2) IP addresses are obfuscate in order to meet company regulation<br>
2019-03-29 14:37:22.812563 IP (tos 0x0, ttl 64, id 36768, offset 0, flags [DF], proto TCP (6), length 662)<br>
x.31.129.217.5672 > x.31.129.206.45604: Flags [P.], cksum 0x83a3 (correct), seq 959640600:959641210, ack 3516429040, win 243, options [nop,nop,TS val 3070303066 ecr 3070033633], length 610<br>
E.....@.@.L..........(.$92....r............<br>
...Z..........].<.<.1.......N.&reply_d5cb71955a3a4a39b1e8f6539a72be4f&reply_d5cb71955a3a4a39b1e8f6539a72be4f.......+.<.............application/json.utf-8..............{"oslo.message": "{\"_msg_id\": \"ae4ea417e79c43af9d1cd2d603b32076\", \"failure\": null, \"_unique_id\":
\"e357642aaef34d1fbaa4d240c405d572\", \"result\": {\"internal_access_path\": null, \"host\": \"x.31.129.164\", \"access_url\": \"https://api.secret.cloud:6080/vnc_auto.html?token=44b7d9be-2aab-4239-914c-971d615fa9c0\", \"token\": \"44b7d9be-2aab-4239-914c-971d615fa9c0\",
\"port\": \"5912\"}, \"ending\": true}", "oslo.version": "2.0"}.<br>
<br>
(3)<br>
:~# dpkg -l | grep amqp<br>
ii python-amqp 1.4.9-1 all Low-level AMQP client<br>
<br>
:~# dpkg -l | grep oslo.messaging<br>
ii python-oslo.messaging 5.10.0-0ubuntu2~cloud0 all oslo messaging library - Python 2.x<br>
<br>
:~# dpkg -l | grep nova<br>
ii nova-api-os-compute 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute - OpenStack Compute API frontend<br>
ii nova-cert 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute - certificate management<br>
ii nova-common 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute - common files<br>
ii nova-conductor 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute - conductor service<br>
ii nova-consoleauth 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute - Console Authenticator<br>
ii nova-novncproxy 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute - NoVNC proxy<br>
ii nova-scheduler 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute - virtual machine scheduler<br>
ii python-nova 2:14.1.0-0ubuntu1~cloud0 all OpenStack Compute Python libraries<br>
ii python-novaclient 2:6.0.0-0ubuntu1~cloud0 all client library for OpenStack Compute API - Python 2.7<br>
<br>
:~# pip list | grep amqp<br>
amqp (1.4.9)<br>
<br>
:~# pip list | grep oslo<br>
oslo.cache (1.14.0)<br>
oslo.concurrency (3.14.0)<br>
oslo.config (3.17.0)<br>
oslo.context (2.9.0)<br>
oslo.db (4.13.3)<br>
oslo.i18n (3.9.0)<br>
oslo.log (3.16.0)<br>
oslo.messaging (5.10.0)<br>
oslo.middleware (3.19.0)<br>
oslo.policy (1.14.0)<br>
oslo.privsep (1.13.0)<br>
oslo.reports (1.14.0)<br>
oslo.rootwrap (5.1.0)<br>
oslo.serialization (2.13.0)<br>
oslo.service (1.16.0)<br>
oslo.utils (3.16.0)<br>
oslo.versionedobjects (1.17.0)</div>
<div id="x_Signature">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span style="font-size:10pt; font-family:"Arial",sans-serif,serif,"EmojiFont""><br>
</span>
<p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>