Open Stack

Hello everyone,

My colleagues and I are maintaining a newton-based openstack installation.
We use linux containers on ubuntu 16.04 nodes for service separation and HA.
After switiching to HTTPS + Domain for all our Endpoints (was http and IP-based before) we face the problem of MessagingTimeout for os-getVNCConsole-Requests.

In (1) you can find the Stack Trace we get on a request for VNCConsole Access.
We have traced this down to the point were we actually tcpdumped the communication and we can see the response message arriving back(2).

The timeline looks like:

- 13:37:22 GMT        - Sending request via nova get-vnc-console a29f9b95-055e-4f30-8a30-ac0bcde32dd1 novnc
- xx:xx:xx                  - Arrival at RabbitMQ
- 14:37:22.806 CET  - Compute Hosts gets the request
- 14:37:22.810 CET  - Compute Hosts sends response to RabbitMQ
- xx:xx:xx                  - Response enters RabbitMQ
- 14:37:22.812 CET  - tcpdumped response at the interface of the control node where the nova cloud controller resides on, who send the request
- 13:38:22.826 UTC  - Timeout /var/log/nova/nova-api-os-compute.log

It could be expected, that the response messages actually arrives at the requesting process.
However, I can not narrow it down to exactly this point, because
- I can't tcpdump in an lxd container
- The system is highly productive and possibilities for debugging are limited
- We have no other problems with this installation in any matter, so a global problem can be excluded

We are actually not sure if the mismatch in timezones acutally leads to a problem.
The fact, that our problem is isolated and all other communication runs fine makes us believe it's not.

Taking the returned "access_url" from the sniffed package(2) and e.g. pasting it to a browser, we get the Standard VNC Console Screen.
This shows the general communication via port 6080 is also possible.
Nonetheless, even doing the as soon as the package was sniffed I never got an actual novnc-session to the VM.

We are stuck at this point, because we simply can't completely understand the python code :(
I hope you can ask some helping questions to get us a step further.

You can find a list of our current dpkg and pip packages in (3).

Questions we are currently able formulate are:
- Is there any communication taking place besides the regular ports (8774, 6080, 5672) in the process?
- are the generated access_urls verified in any way after receiving the response message from RabbitMQ, which could lead to an exception being thrown?
- are we looking at the right communication flow (https://docs.openstack.org/nova/pike/_images/SCH_5009_V00_NUAC-VNC_OpenStack.png, there is no one for newton!?) at all?

Best regards

Ralf Teckelmann

(1)
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/remote_consoles.py", line 56, in get_vnc_console
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions console_type)
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 157, in wrapped
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions return function(self, context, instance, *args, **kwargs)
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3283, in get_vnc_console
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions access_url=connect_info['access_url'])
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/consoleauth/rpcapi.py", line 89, in authorize_console
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions return cctxt.call(ctxt, 'authorize_console', **msg_args)
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions retry=self.retry)
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry)
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions retry=retry)
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 453, in _send
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions result = self._waiter.wait(msg_id, timeout)
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in wait
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions message = self.waiters.get(msg_id, timeout=timeout)
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 238, in get
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions 'to message ID %s' % msg_id)
2019-03-29 13:38:22.826 18711 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID bd66ed7b4fe44f5c8527573081379639

(2) IP addresses are obfuscate in order to meet company regulation
2019-03-29 14:37:22.812563 IP (tos 0x0, ttl 64, id 36768, offset 0, flags [DF], proto TCP (6), length 662)
 x.31.129.217.5672 > x.31.129.206.45604: Flags [P.], cksum 0x83a3 (correct), seq 959640600:959641210, ack 3516429040, win 243, options [nop,nop,TS val 3070303066 ecr 3070033633], length 610
E..... at .@.L..........(.$92....r............
...Z..........].<.<.1.......N.&reply_d5cb71955a3a4a39b1e8f6539a72be4f&reply_d5cb71955a3a4a39b1e8f6539a72be4f.......+.<.............application/json.utf-8..............{"oslo.message": "{\"_msg_id\": \"ae4ea417e79c43af9d1cd2d603b32076\", \"failure\": null, \"_unique_id\": \"e357642aaef34d1fbaa4d240c405d572\", \"result\": {\"internal_access_path\": null, \"host\": \"x.31.129.164\", \"access_url\": \"https://api.secret.cloud:6080/vnc_auto.html?token=44b7d9be-2aab-4239-914c-971d615fa9c0\", \"token\": \"44b7d9be-2aab-4239-914c-971d615fa9c0\", \"port\": \"5912\"}, \"ending\": true}", "oslo.version": "2.0"}.

(3)
:~# dpkg -l | grep amqp
ii  python-amqp                      1.4.9-1                                    all          Low-level AMQP client

:~# dpkg -l | grep oslo.messaging
ii  python-oslo.messaging            5.10.0-0ubuntu2~cloud0                     all          oslo messaging library - Python 2.x

:~# dpkg -l | grep nova
ii  nova-api-os-compute              2:14.1.0-0ubuntu1~cloud0                   all          OpenStack Compute - OpenStack Compute API frontend
ii  nova-cert                        2:14.1.0-0ubuntu1~cloud0                   all          OpenStack Compute - certificate management
ii  nova-common                      2:14.1.0-0ubuntu1~cloud0                   all          OpenStack Compute - common files
ii  nova-conductor                   2:14.1.0-0ubuntu1~cloud0                   all          OpenStack Compute - conductor service
ii  nova-consoleauth                 2:14.1.0-0ubuntu1~cloud0                   all          OpenStack Compute - Console Authenticator
ii  nova-novncproxy                  2:14.1.0-0ubuntu1~cloud0                   all          OpenStack Compute - NoVNC proxy
ii  nova-scheduler                   2:14.1.0-0ubuntu1~cloud0                   all          OpenStack Compute - virtual machine scheduler
ii  python-nova                      2:14.1.0-0ubuntu1~cloud0                   all          OpenStack Compute Python libraries
ii  python-novaclient                2:6.0.0-0ubuntu1~cloud0                    all          client library for OpenStack Compute API - Python 2.7

:~# pip list | grep amqp
amqp (1.4.9)

:~# pip list | grep oslo
oslo.cache (1.14.0)
oslo.concurrency (3.14.0)
oslo.config (3.17.0)
oslo.context (2.9.0)
oslo.db (4.13.3)
oslo.i18n (3.9.0)
oslo.log (3.16.0)
oslo.messaging (5.10.0)
oslo.middleware (3.19.0)
oslo.policy (1.14.0)
oslo.privsep (1.13.0)
oslo.reports (1.14.0)
oslo.rootwrap (5.1.0)
oslo.serialization (2.13.0)
oslo.service (1.16.0)
oslo.utils (3.16.0)
oslo.versionedobjects (1.17.0)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190404/04961b15/attachment-0001.html>