Folks, Life was good till I restarted nova-novncproxy services on all controller nodes after that my vnc console stopped working. I can directly connect to the vm console using vnc viewer but not from Horizon. This is not a new setup or something. What could be wrong here and how do I debug this protocol issue? This is what I have on controller nodes [vnc] novncproxy_host = 192.168.8.14 novncproxy_port = 6080 server_listen = 192.168.8.14 server_proxyclient_address = 192.168.8.14 This is what I have on all computes nodes [vnc] novncproxy_host = 192.168.8.18 novncproxy_port = 6080 server_listen = 192.168.8.18 server_proxyclient_address = 192.168.8.18 novncproxy_base_url = https://os2.example.com:6080/vnc_lite.html 2024-02-13 20:08:16.315 7 INFO nova.console.websocketproxy [-] In exit 2024-02-13 20:08:30.402 7 INFO nova.console.websocketproxy [-] WebSocket server settings: 2024-02-13 20:08:30.403 7 INFO nova.console.websocketproxy [-] - Listen on 192.168.8.14:6080 2024-02-13 20:08:30.403 7 INFO nova.console.websocketproxy [-] - Web server (no directory listings). Web root: /usr/share/novnc 2024-02-13 20:08:30.404 7 INFO nova.console.websocketproxy [-] - No SSL/TLS support (no cert file) 2024-02-13 20:08:30.404 7 INFO nova.console.websocketproxy [-] - proxying from 192.168.8.14:6080 to None:None 2024-02-13 20:08:42.800 7 DEBUG nova.console.websocketproxy [-] 192.168.8.15: new handler Process vmsg /var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py:605 2024-02-13 20:08:42.810 71 INFO nova.console.websocketproxy [-] 192.168.8.15 - - [13/Feb/2024 20:08:42] 192.168.8.15: Plain non-SSL (ws://) WebSocket connection 2024-02-13 20:08:42.810 71 INFO nova.console.websocketproxy [-] 192.168.8.15 - - [13/Feb/2024 20:08:42] 192.168.8.15: Path: '/?token=03ad44a0-b876-4725-a71f-d383c15efb63' 2024-02-13 20:08:42.832 71 DEBUG oslo_db.sqlalchemy.engines [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:335 2024-02-13 20:08:42.923 71 DEBUG nova.objects.console_auth_token [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] Validated token - console connection is ConsoleAuthToken(access_url_base='https://os2.example.com:6080/vnc_lite.html',console_type='novnc',created_at=2024-02-13T20:08:41Z,host='192.168.8.18',id=3993,instance_uuid=7b6e0c71-ea3c-4314-9a4c-1fc86d5680f2,internal_access_path=None,port=5920,token='***',updated_at=None) validate /var/lib/kolla/venv/lib/python3.10/site-packages/nova/objects/console_auth_token.py:153 2024-02-13 20:08:42.961 71 DEBUG oslo_concurrency.lockutils [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] Acquiring lock "compute-rpcapi-router" lock /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:312 2024-02-13 20:08:42.961 71 DEBUG oslo_concurrency.lockutils [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] Acquired lock "compute-rpcapi-router" lock /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:315 2024-02-13 20:08:42.967 71 INFO nova.compute.rpcapi [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] Automatically selected compute RPC version 6.2 from minimum service version 66 2024-02-13 20:08:42.968 71 DEBUG oslo_concurrency.lockutils [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] Releasing lock "compute-rpcapi-router" lock /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:333 2024-02-13 20:08:43.008 71 INFO nova.console.websocketproxy [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] handler exception: Origin header protocol does not match this host. 2024-02-13 20:08:43.008 71 DEBUG nova.console.websocketproxy [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] exception vmsg /var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py:605 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy Traceback (most recent call last): 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 662, in top_new_client 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy client = self.do_handshake(startsock, address) 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 590, in do_handshake 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy self.RequestHandlerClass(retsock, address, self) 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/console/websocketproxy.py", line 93, in __init__ 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy websockify.ProxyRequestHandler.__init__(self, *args, **kwargs) 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 87, in __init__ 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy super().__init__(req, addr, server) 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/usr/lib/python3.10/http/server.py", line 668, in __init__ 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy super().__init__(*args, **kwargs) 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/usr/lib/python3.10/socketserver.py", line 747, in __init__ 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy self.handle() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 297, in handle 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy super().handle() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/usr/lib/python3.10/http/server.py", line 433, in handle 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy self.handle_one_request() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websocketserver.py", line 40, in handle_one_request 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy super().handle_one_request() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/usr/lib/python3.10/http/server.py", line 421, in handle_one_request 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy method() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websocketserver.py", line 50, in _websocket_do_GET 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy self.handle_upgrade() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 203, in handle_upgrade 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy super().handle_upgrade() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websocketserver.py", line 77, in handle_upgrade 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy self.handle_websocket() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 241, in handle_websocket 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy self.new_websocket_client() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/console/websocketproxy.py", line 215, in new_websocket_client 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy raise exception.ValidationError(detail=detail) 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy nova.exception.ValidationError: Origin header protocol does not match this host. 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy