Folks,
Life was good till I restarted nova-novncproxy services on all controller nodes after that my vnc console stopped working. I can directly connect to the vm console using vnc viewer but not from Horizon. This is not a new setup or something. What could be wrong here and how do I debug this protocol issue?
This is what I have on controller nodes
[vnc]
novncproxy_host = 192.168.8.14
novncproxy_port = 6080
server_listen = 192.168.8.14
server_proxyclient_address = 192.168.8.14
This is what I have on all computes nodes
[vnc]
novncproxy_host = 192.168.8.18
novncproxy_port = 6080
server_listen = 192.168.8.18
server_proxyclient_address = 192.168.8.18
novncproxy_base_url = https://os2.example.com:6080/vnc_lite.html
2024-02-13 20:08:16.315 7 INFO nova.console.websocketproxy [-] In exit
2024-02-13 20:08:30.402 7 INFO nova.console.websocketproxy [-] WebSocket server settings:
2024-02-13 20:08:30.403 7 INFO nova.console.websocketproxy [-]   - Listen on 192.168.8.14:6080
2024-02-13 20:08:30.403 7 INFO nova.console.websocketproxy [-]   - Web server (no directory listings). Web root: /usr/share/novnc
2024-02-13 20:08:30.404 7 INFO nova.console.websocketproxy [-]   - No SSL/TLS support (no cert file)
2024-02-13 20:08:30.404 7 INFO nova.console.websocketproxy [-]   - proxying from 192.168.8.14:6080 to None:None
2024-02-13 20:08:42.800 7 DEBUG nova.console.websocketproxy [-] 192.168.8.15: new handler Process vmsg /var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py:605
2024-02-13 20:08:42.810 71 INFO nova.console.websocketproxy [-] 192.168.8.15 - - [13/Feb/2024 20:08:42] 192.168.8.15: Plain non-SSL (ws://) WebSocket connection
2024-02-13 20:08:42.810 71 INFO nova.console.websocketproxy [-] 192.168.8.15 - - [13/Feb/2024 20:08:42] 192.168.8.15: Path: '/?token=03ad44a0-b876-4725-a71f-d383c15efb63'
2024-02-13 20:08:42.832 71 DEBUG oslo_db.sqlalchemy.engines [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:335
2024-02-13 20:08:42.923 71 DEBUG nova.objects.console_auth_token [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] Validated token - console connection is ConsoleAuthToken(access_url_base='https://os2.example.com:6080/vnc_lite.html',console_type='novnc',created_at=2024-02-13T20:08:41Z,host='192.168.8.18',id=3993,instance_uuid=7b6e0c71-ea3c-4314-9a4c-1fc86d5680f2,internal_access_path=None,port=5920,token='***',updated_at=None) validate /var/lib/kolla/venv/lib/python3.10/site-packages/nova/objects/console_auth_token.py:153
2024-02-13 20:08:42.961 71 DEBUG oslo_concurrency.lockutils [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] Acquiring lock "compute-rpcapi-router" lock /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:312
2024-02-13 20:08:42.961 71 DEBUG oslo_concurrency.lockutils [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] Acquired lock "compute-rpcapi-router" lock /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:315
2024-02-13 20:08:42.967 71 INFO nova.compute.rpcapi [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] Automatically selected compute RPC version 6.2 from minimum service version 66
2024-02-13 20:08:42.968 71 DEBUG oslo_concurrency.lockutils [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] Releasing lock "compute-rpcapi-router" lock /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:333
2024-02-13 20:08:43.008 71 INFO nova.console.websocketproxy [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] handler exception: Origin header protocol does not match this host.
2024-02-13 20:08:43.008 71 DEBUG nova.console.websocketproxy [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] exception vmsg /var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py:605
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy Traceback (most recent call last):
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy   File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 662, in top_new_client
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy     client = self.do_handshake(startsock, address)
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy   File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 590, in do_handshake
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy     self.RequestHandlerClass(retsock, address, self)
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/console/websocketproxy.py", line 93, in __init__
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy     websockify.ProxyRequestHandler.__init__(self, *args, **kwargs)
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy   File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 87, in __init__
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy     super().__init__(req, addr, server)
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy   File "/usr/lib/python3.10/http/server.py", line 668, in __init__
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy     super().__init__(*args, **kwargs)
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy   File "/usr/lib/python3.10/socketserver.py", line 747, in __init__
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy     self.handle()
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy   File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 297, in handle
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy     super().handle()
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy   File "/usr/lib/python3.10/http/server.py", line 433, in handle
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy     self.handle_one_request()
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy   File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websocketserver.py", line 40, in handle_one_request
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy     super().handle_one_request()
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy   File "/usr/lib/python3.10/http/server.py", line 421, in handle_one_request
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy     method()
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy   File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websocketserver.py", line 50, in _websocket_do_GET
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy     self.handle_upgrade()
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy   File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 203, in handle_upgrade
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy     super().handle_upgrade()
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy   File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websocketserver.py", line 77, in handle_upgrade
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy     self.handle_websocket()
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy   File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 241, in handle_websocket
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy     self.new_websocket_client()
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/console/websocketproxy.py", line 215, in new_websocket_client
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy     raise exception.ValidationError(detail=detail)
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy nova.exception.ValidationError: Origin header protocol does not match this host.
2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy