February 2024 - openstack-discuss

[nova][nova-novncproxy] handler exception: Origin header protocol does not match this host.
by Satish Patel 13 Feb '24

13 Feb '24

Folks, Life was good till I restarted nova-novncproxy services on all controller nodes after that my vnc console stopped working. I can directly connect to the vm console using vnc viewer but not from Horizon. This is not a new setup or something. What could be wrong here and how do I debug this protocol issue? This is what I have on controller nodes [vnc] novncproxy_host = 192.168.8.14 novncproxy_port = 6080 server_listen = 192.168.8.14 server_proxyclient_address = 192.168.8.14 This is what I have on all computes nodes [vnc] novncproxy_host = 192.168.8.18 novncproxy_port = 6080 server_listen = 192.168.8.18 server_proxyclient_address = 192.168.8.18 novncproxy_base_url = https://os2.example.com:6080/vnc_lite.html 2024-02-13 20:08:16.315 7 INFO nova.console.websocketproxy [-] In exit 2024-02-13 20:08:30.402 7 INFO nova.console.websocketproxy [-] WebSocket server settings: 2024-02-13 20:08:30.403 7 INFO nova.console.websocketproxy [-] - Listen on 192.168.8.14:6080 2024-02-13 20:08:30.403 7 INFO nova.console.websocketproxy [-] - Web server (no directory listings). Web root: /usr/share/novnc 2024-02-13 20:08:30.404 7 INFO nova.console.websocketproxy [-] - No SSL/TLS support (no cert file) 2024-02-13 20:08:30.404 7 INFO nova.console.websocketproxy [-] - proxying from 192.168.8.14:6080 to None:None 2024-02-13 20:08:42.800 7 DEBUG nova.console.websocketproxy [-] 192.168.8.15: new handler Process vmsg /var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py:605 2024-02-13 20:08:42.810 71 INFO nova.console.websocketproxy [-] 192.168.8.15 - - [13/Feb/2024 20:08:42] 192.168.8.15: Plain non-SSL (ws://) WebSocket connection 2024-02-13 20:08:42.810 71 INFO nova.console.websocketproxy [-] 192.168.8.15 - - [13/Feb/2024 20:08:42] 192.168.8.15: Path: '/?token=03ad44a0-b876-4725-a71f-d383c15efb63' 2024-02-13 20:08:42.832 71 DEBUG oslo_db.sqlalchemy.engines [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:335 2024-02-13 20:08:42.923 71 DEBUG nova.objects.console_auth_token [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] Validated token - console connection is ConsoleAuthToken(access_url_base='https://os2.example.com:6080/vnc_lite.html',console_type='novnc',created_at=2024-02-13T20:08:41Z,host='192.168.8.18',id=3993,instance_uuid=7b6e0c71-ea3c-4314-9a4c-1fc86d5680f2,internal_access_path=None,port=5920,token='***',updated_at=None) validate /var/lib/kolla/venv/lib/python3.10/site-packages/nova/objects/console_auth_token.py:153 2024-02-13 20:08:42.961 71 DEBUG oslo_concurrency.lockutils [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] Acquiring lock "compute-rpcapi-router" lock /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:312 2024-02-13 20:08:42.961 71 DEBUG oslo_concurrency.lockutils [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] Acquired lock "compute-rpcapi-router" lock /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:315 2024-02-13 20:08:42.967 71 INFO nova.compute.rpcapi [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] Automatically selected compute RPC version 6.2 from minimum service version 66 2024-02-13 20:08:42.968 71 DEBUG oslo_concurrency.lockutils [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] Releasing lock "compute-rpcapi-router" lock /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:333 2024-02-13 20:08:43.008 71 INFO nova.console.websocketproxy [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] handler exception: Origin header protocol does not match this host. 2024-02-13 20:08:43.008 71 DEBUG nova.console.websocketproxy [None req-dc3e111c-7a79-4d56-8077-688c9e813700 - - - - - -] exception vmsg /var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py:605 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy Traceback (most recent call last): 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 662, in top_new_client 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy client = self.do_handshake(startsock, address) 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 590, in do_handshake 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy self.RequestHandlerClass(retsock, address, self) 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/console/websocketproxy.py", line 93, in __init__ 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy websockify.ProxyRequestHandler.__init__(self, *args, **kwargs) 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 87, in __init__ 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy super().__init__(req, addr, server) 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/usr/lib/python3.10/http/server.py", line 668, in __init__ 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy super().__init__(*args, **kwargs) 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/usr/lib/python3.10/socketserver.py", line 747, in __init__ 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy self.handle() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 297, in handle 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy super().handle() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/usr/lib/python3.10/http/server.py", line 433, in handle 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy self.handle_one_request() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websocketserver.py", line 40, in handle_one_request 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy super().handle_one_request() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/usr/lib/python3.10/http/server.py", line 421, in handle_one_request 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy method() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websocketserver.py", line 50, in _websocket_do_GET 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy self.handle_upgrade() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 203, in handle_upgrade 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy super().handle_upgrade() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websocketserver.py", line 77, in handle_upgrade 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy self.handle_websocket() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/websockify/websockifyserver.py", line 241, in handle_websocket 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy self.new_websocket_client() 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/console/websocketproxy.py", line 215, in new_websocket_client 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy raise exception.ValidationError(detail=detail) 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy nova.exception.ValidationError: Origin header protocol does not match this host. 2024-02-13 20:08:43.008 71 ERROR nova.console.websocketproxy

1 0

[openstack-helm] Ceph-osd + Jammy + loop device is broken now
by Vladimir Kozhukalov 13 Feb '24

13 Feb '24

Hi team, Due to this [1] some of our test jobs which utilize the ceph-osd chart and which run on Ubuntu Jammy are broken now. Recent changes in ioctl are incompatible with parted when used with loop devices. Ceph-osd chart uses parted in the osd init script. So it does not work now with the error like this ``` Error: Partition(s) 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64 on /dev/loop100 have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes. ``` The issue appears only on Ubuntu Jammy. My suggestion is to temporarily disable the 2023.2 TLS job which is broken now. And to use the 2023.1 TLS job on Focal. Please review https://review.opendev.org/c/openstack/openstack-helm/+/908947 [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2049689 -- Best regards, Kozhukalov Vladimir

1 0

[openstack-ansible] Bug fighting day
by Dmitriy Rabotyagov 13 Feb '24

13 Feb '24

Hey everyone. During the previous meeting we decided that it might be a good idea to take a couple hours together to work on the project bug tracker [1]. We are going to focus on closing irrelevant bugs that were issued towards EOLed releases and do not apply to any supported code or were already fixed. Along with that we will try to triage and prioritize ones that are still relevant and need some attention. In order to decide on the date/time when this activity will take place I have created a poll [2]. It will be open till Sunday, Feb 11. Everyone who is willing to help us is very welcome to join the poll. If you have an outstanding bug that somehow slipped our attention - this is really a good chance to bring it up to the team as well. [1] https://bugs.launchpad.net/openstack-ansible [2] https://doodle.com/meeting/organize/id/enGV50Wa

1 2

http://apps.openstack.org/#tab=murano-apps unavailable
by Karera Tony 13 Feb '24

13 Feb '24

I have deployed Murano in my environment, However when I tried to get the application budle following the Murano guide from http://apps.openstack.org/#tab=murano-apps. I realized that the site is down. Is there an alternative ? Regards Regards Tony Karera

2 1

[nova]: openstack-ansible Ussuri - rabbitmq connection distribution
by keshav bareja 13 Feb '24

13 Feb '24

2 1

[magnum] Weekly meeting (14/Feb/2024) cancelled
by Michał Nasiadka 13 Feb '24

13 Feb '24

Hello, Due to absence of key team members - the weekly meeting tomorrow (14/Feb/2024) is cancelled. See you next week! Best regards, Michal

1 0

[all][tc] OpenStack 2024.2 testing runtime
by Ghanshyam Mann 12 Feb '24

12 Feb '24

Hi All, Its time to deifne the testing runtime for next cycle 2024.2. I have proposed it in gerrit, feel free to review/feedback there. - https://review.opendev.org/c/openstack/governance/+/908862 Changes from previous cycle testing runtime for quick ref: * Removing python 3.8 support because it is EOLing in oct 2024[1] which means python 3.9 will be the minimum python version in testing runtime. * Remove Debian 11 from Additional testing support because 2024.2 is non-SLURP release and previous SLURP release 2024.1 supported Debian 12 also which is what can be used as smooth upgrade to future SLURP release 2025.1 * Removing Nodejs from runtime: Nodejs is not a runtime requirement for Horizon instead it is one of the test dependency. It does not fit as a runtime and can be handle by Horizon team based. Added Horizon team to review this change. -gmann

1 0

[neutron][vpnaas][ovn] Pushing VPNaaS support through OVN
by Mohammed Naser 12 Feb '24

12 Feb '24

Hi all! One of the biggest show stoppers for us for OVN is VPNaaS but I think it’s time that we’d like to be involved in getting it over the line, I want to thank Bodo who has shown a lot of patience in getting the code necessary, and it seems that the code works for them in production as well. I’d like to ask if possible for folks to review these two outstanding changes: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/847007 https://review.opendev.org/c/openstack/neutron-vpnaas/+/765353 I know Slawek had some concerns, so hoping that raising via the email list if Bodo can address those and over the next bit of time we can get this landed in, since it even has actual proper testing both in CI and with user feedback. Thanks! Mohammed

6 8

[neutron] Bugs deputy report February 5th to 11th
by Rodolfo Alonso Hernandez 12 Feb '24

12 Feb '24

Hello all: This is the bug list from the past week: Critical ======================== https://bugs.launchpad.net/neutron/+bug/2052509. "[CI][ovs/ovn sqlalchemy master] jobs broken with Exception: Not enough arguments given". Assigned. Patch: https://review.opendev.org/c/openstack/devstack/+/908572 https://bugs.launchpad.net/neutron/+bug/2052787. "SSH timeouts due to problems with metadata server in ML2/OVN backend". Assigned. High ======================== https://bugs.launchpad.net/neutron/+bug/2052628. "[stable-only][ovn-octavia-provider] Multiple listener-pool-member on IPv6 LB getting second pool in ERROR". Assigned. Patch: https://review.opendev.org/q/I37e660d5eaa45deb38ff127cf69195b0a9ef0846 (merged). Medium ======================== https://bugs.launchpad.net/neutron/+bug/2052484. "[DEFAULT] rpc_worker=0 leaves one rpc worker". Assigned. Patch: https://review.opendev.org/c/openstack/neutron/+/908173 https://bugs.launchpad.net/neutron/+bug/2052681. " Many stale neutron-keepalived-state-change processes left after upgrade to native pyroute2 state-change". Not assigned. https://bugs.launchpad.net/neutron/+bug/2052821. "[OVN] Pin a Logical_Router to a chassis when the external network is tunnelled". Assigned. https://bugs.launchpad.net/neutron/+bug/2052843. "[OVN Octavia Provider] subnet with multiple router gateways lb association". Assigned. Patch: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/908679 https://bugs.launchpad.net/neutron/+bug/2052419. "SG list retrieval is slow because we are always loading "synthetic_fields"". Assigned. Patches: https://review.opendev.org/q/I300464472f2348d148f2a3ddac384c883d9d815b Low ======================== https://bugs.launchpad.net/neutron/+bug/2052508. " [doc][troubleshooting] how to enable ovs vswitchd debug logs". Not assigned. https://bugs.launchpad.net/neutron/+bug/2052786. " Add "socket" to NUMA affinity policy". Assigned. Patch: https://review.opendev.org/c/openstack/neutron-lib/+/903531 Regards.

1 0

PTG April 2024 Team Signup Deadline Approaching!
by Kendall Nelson 12 Feb '24

12 Feb '24

Hello Everyone, This is your *last call *to sign up your team up for the next Project Teams Gathering (PTG)[1], which will be held from Monday, April 8 to Friday, April 12 2024! If you haven't already done so and your team is interested in participating, *you need to complete the survey[2] by February 18, 2024 at 7:00 UTC*. Thanks! -Kendall Nelson (diablo_rojo) [1] PTG Site: https://openinfra.dev/ptg/ [2] Team Survey: https://openinfrafoundation.formstack.com/forms/april2024_ptg_survey

1 0