- openstack-discuss - lists.openstack.org

[largescale-sig] Next meeting: May 31, 8utc
by Thierry Carrez 31 May '23

31 May '23

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 8UTC, our APAC+EU-friendly time. I will be chairing. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20230531T08 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[kolla][cinder] cinder containers api, volume, backup unhealthy
by Oliver Weinmann 30 May '23

30 May '23

Hi, after replacing my control nodes with new nodes (all bare-metal) somehow the cinder containers are no longer starting. I checked the logs on one of the control nodes and I see this in the api-eror.log: 2023-05-30 21:31:36.350636 Timeout when reading response headers from daemon process 'cinder-api': /var/www/cgi-bin/cinder/cinder-wsgi 2023-05-30 21:31:37.827101 mod_wsgi (pid=22): Failed to exec Python script file '/var/www/cgi-bin/cinder/cinder-wsgi'. 2023-05-30 21:31:37.827168 mod_wsgi (pid=22): Exception occurred processing WSGI script '/var/www/cgi-bin/cinder/cinder-wsgi'. 2023-05-30 21:31:37.828005 Traceback (most recent call last): 2023-05-30 21:31:37.828046 File "/var/www/cgi-bin/cinder/cinder-wsgi", line 52, in <module> 2023-05-30 21:31:37.828053 application = initialize_application() 2023-05-30 21:31:37.828058 File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/wsgi/wsgi.py", line 44, in initialize_application 2023-05-30 21:31:37.828063 coordination.COORDINATOR.start() 2023-05-30 21:31:37.828068 File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/coordination.py", line 86, in start 2023-05-30 21:31:37.828071 self.coordinator.start(start_heart=True) 2023-05-30 21:31:37.828075 File "/var/lib/kolla/venv/lib/python3.6/site-packages/tooz/coordination.py", line 689, in start 2023-05-30 21:31:37.828078 super(CoordinationDriverWithExecutor, self).start(start_heart) 2023-05-30 21:31:37.828083 File "/var/lib/kolla/venv/lib/python3.6/site-packages/tooz/coordination.py", line 426, in start 2023-05-30 21:31:37.828086 self._start() 2023-05-30 21:31:37.828090 File "/var/lib/kolla/venv/lib/python3.6/site-packages/tooz/drivers/etcd3gw.py", line 224, in _start 2023-05-30 21:31:37.828093 self._membership_lease = self.client.lease(self.membership_timeout) 2023-05-30 21:31:37.828098 File "/var/lib/kolla/venv/lib/python3.6/site-packages/etcd3gw/client.py", line 122, in lease 2023-05-30 21:31:37.828111 json={"TTL": ttl, "ID": 0}) 2023-05-30 21:31:37.828116 File "/var/lib/kolla/venv/lib/python3.6/site-packages/etcd3gw/client.py", line 88, in post 2023-05-30 21:31:37.828123 resp.reason 2023-05-30 21:31:37.828154 etcd3gw.exceptions.ConnectionTimeoutError: Gateway Time-out All other containers are working just fine. Even the cinder_scheduler container works fine. So far I have tried the following: remove the cinder containers including its volume from one control node mariadb_recovery Reboot all control nodes. kolla-ansible reconfigure --tags cinder,nova Any help is highly appreciated. Cheers, Oliver

2 1

Invitation to further discuss the notion of a 'domain manager' role
by Felix Kronlage-Dammers 30 May '23

30 May '23

Hi, We (SCS Project) are looking for CSPs / Operators who would like to join a discussion on the notion of a ‘domain manager role’. To a lot of you this topic is far from new and has been brought up just recently on this list as well[1]. On the upcoming Monday (June 5th) at 15:05 CEST we would like to have a kickstarting discussion of CSPs who currently solve the hurdle of not having such a role in their environments and/or their custom frontends. We will meet here: <url: https://conf.scs.koeln:8443/SCS-Tech> Through the Public Cloud SIG I’ve reached out to Cleura, OTC as well as Catalyst Cloud. From the german scene PlusServer will join the discussion. As part of the discussion on monday we want to see, where we can move towards a joint-effort to maybe see that such a role can be introduced earlier than anticipated. For that it sure is good to understand wether any of the participating CSPs do have “stuff in their drawer” (code, knowledge, experience, …) that they could bring to the table. regards, felix [1] <url: https://lists.openstack.org/pipermail/openstack-discuss/2023-May/033803.html> -- Felix Kronlage-Dammers Product Owner IaaS & Operations Sovereign Cloud Stack Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

2 2

[ironic] OpenInfra Summit planning/attendance
by Jay Faulkner 30 May '23

30 May '23

Hi all, We're doing some light coordination of Ironic-interested operators and contributors at the OpenInfra Summit. If you're a part of the larger Ironic community and you're interested in meeting up with the team, seeing talks about Ironic specifically, or attending the team dinner please go to https://etherpad.opendev.org/p/ironic-openinfra-2023 and comment accordingly. Thanks! See many of you in two short weeks! - Jay Faulkner Ironic PTL

1 0

[ironic] Retirement of ironic-prometheus-exporter bugfix/2.1 branch
by Jay Faulkner 30 May '23

30 May '23

Hi all, The ironic-prometheus-exporter bugfix/2.1 branch is currently broken in CI, and unused based on a poll of known-downstream users of bugfix branches. It's my intention to retire this branch in the same manner for other Ironic bugfix branches -- by tagging it EOL. If you have an objection, please bring it up soon. Thank you, Jay Faulkner Ironic PTL

1 0

[cinder] - Test cases or procedure for active/active support validation
by Roquesalane, Jean-Pierre 30 May '23

30 May '23

Hi community! Here at Dell, we're in the process of certifying our Dell PowerMax storage platform against cinder volume Active/active configuration. I was wondering how vendors who already support this functionality have managed to handle this and if they can share their testing scenario. Thank you. JP Jean-Pierre Roquesalane SW System Sr Principal Engineer - OpenStack Champion Dell Technologies | Security and Ecosystem Software Mobile +33 6 21 86 79 04<tel:1-613-314-8106> Email : jeanpierre.roquesalane(a)dell.com<mailto:jeanpierre.roquesalane@dell.com> [cid:image001.png@01D976C9.D5B91180] Internal Use - Confidential Dell S.A, Si?ge Social 1 rond point Benjamin Franklin 34000 Montpellier. Capital 1,782,769 Euros, 351 528 229 RCS Montpellier -APE 4651Z -TVA Intracommunautaire FR 20 351 528 229, SIRET 351 528 229 00088 Vat Number : FR 20351528229 (France) / IT00001709997 (Italy) / ESN0012622G (Spain)

2 1

[kolla-ansible][yoga] Does glance support cinder-nfs as backend?
by wodel youchi 30 May '23

30 May '23

Hi, The openstak documentation says that glance supports cinder as backend, but it does not exclude any backend used by cinder itself. I'm having trouble configuring glance to use a cinder backend which is backed by an nfs share. Is this configuration supported? First, the rootwrap was missing, after adding it, I faced the lack of privileges, which was corrected by starting the glance-api container in privileged mode and finally I am facing a non existing filter error. Glance is trying to mount the nfs share to use it!!! Which I don't understand , why mount a share that is already mounted by cinder which glance is supposed to use as an intermediary!!!? When I push an image I get this error: Stderr: '/var/lib/kolla/venv/bin/glance-rootwrap: Unauthorized command: mount -t nfs 20.1.0.32:/kolla_nfs /var/lib/glance/mnt/nfs/f6f6b4ee42b4f3522a75f422887010ad2c47f8624 f97bf3623b13014f22186b7 (no filter matched)\n' Regards.

3 3

[kolla-ansible][yoga] Glance backend cinder Privsep daemon failed to start operation not permitted
by wodel youchi 30 May '23

30 May '23

Hi, I am trying to configure glance to use cinder as a backend. This is my glance-api.conf [cinder] cinder_store_auth_address = https://dashint.example.com:5000/v3 cinder_store_user_name = cinder cinder_store_password = cinderpass cinder_store_project_name = service cinder_volume_type = nfstype rootwrap_config = /etc/glance/rootwrap.conf ==> /var/log/kolla/glance/glance-api.log <== > 2023-04-12 18:02:20.842 64 INFO oslo.privsep.daemon > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default default] Spawned new privsep > daemon via rootwrap > 2023-04-12 18:02:20.733 360 INFO oslo.privsep.daemon [-] privsep daemon > starting > 2023-04-12 18:02:20.735 360 INFO oslo.privsep.daemon [-] privsep process > running with uid/gid: 0/0 > > *2023-04-12 18:02:20.737 360 ERROR oslo.privsep.daemon [-] [Errno 1] > Operation not permitted Traceback (most recent call last): * > File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", > line 584, in helper_main > Daemon(channel, context).run() > File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", > line 394, in run > self._drop_privs() > File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", > line 430, in _drop_privs > capabilities.drop_all_caps_except(self.caps, self.caps, []) > File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/capabilities.py", > line 156, in drop_all_caps_except > raise OSError(errno, os.strerror(errno)) > PermissionError: [Errno 1] Operation not permitted > 2023-04-12 18:02:20.844 64 WARNING oslo_privsep.comm > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default > default] Unexpected error: <class 'BrokenPipeError'>: BrokenPipeError: > [Errno 32] Broken pipe > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default > default] Error while sending initial PING to privsep: [Errno 32] Broken > pipe: BrokenPipeError: [Errno 32] Broken pipe > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon Traceback (most > recent call last): > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", > line 192, in exchange_ping > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon reply = > self.send_recv((comm.Message.PING.value,)) > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/comm.py", > line 186, in send_recv > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon > self.writer.send((myid, msg)) > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/comm.py", > line 60, in send > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon > self.writesock.sendall(buf) > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File > "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", > line 407, in sendall > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon tail = > self.send(data, flags) > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File > "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", > line 401, in send > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon return > self._send_loop(self.fd.send, data, flags) > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon File > "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", > line 388, in _send_loop > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon return > send_method(data, *args) > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon BrokenPipeError: > [Errno 32] Broken pipe > 2023-04-12 18:02:20.844 64 ERROR oslo.privsep.daemon > 2023-04-12 18:02:20.846 64 CRITICAL oslo.privsep.daemon > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - defau > lt default] Privsep daemon failed to start > 2023-04-12 18:02:20.847 64 ERROR glance_store._drivers.cinder > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - > default default] Failed to write to volume > 46316c12-6c24-40af-afde-1c16edd616b6.: > oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start > 2023-04-12 18:02:20.890 64 ERROR glance.api.v2.image_data > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - def > ault default] *Failed to upload image data due to internal error: > oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start * > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default d > efault] Caught error: Privsep daemon failed to start: > oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi Traceback (most recent > call last): > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", > line 1332, in __call__ > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi request, > **action_args) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/wsgi.py", > line 1370, in dispatch > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return > method(*args, **kwargs) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/common/utils.py", > line 414, in wrapped > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return func(self, > req, *args, **kwargs) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", > line 303, in upload > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi > self._restore(image_repo, image) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi > self.force_reraise() > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise self.value > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/api/v2/image_data.py", > line 163, in upload > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi > image.set_data(data, size, backend=backend) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line > 497, in set_data > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi > _send_notification(notify_error, 'image.upload', msg) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 227, in __exit__ > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi > self.force_reraise() > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", > line 200, in force_reraise > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise self.value > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/notifier.py", line > 444, in set_data > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi > set_active=set_active) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/quota/__init__.py", > line 323, in set_data > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi > set_active=set_active) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line > 585, in set_data > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi > self._upload_to_store(data, verifier, backend, size) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance/location.py", line > 485, in _upload_to_store > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi verifier=verifier) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", > line 399, in add_with_multihash > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image_id, data, > size, hashing_algo, store, context, verifier) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/multi_backend.py", > line 481, in store_add_to_backe > nd_with_multihash > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi image_id, data, > size, hashing_algo, context=context, verifier=verifier) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/driver.py", > line 279, in add_adapter > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi metadata_dict) = > store_add_fun(*args, **kwargs) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/capabilities.py", > line 176, in op_checker > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return > store_op_fun(store, *args, **kwargs) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", > line 985, in add > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi with > self._open_cinder_volume(client, volume, 'wb') as f: > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return > next(self.gen) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", > line 739, in _open_cinder_vol > ume > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi root_helper, > my_ip, use_multipath, enforce_multipath, host=host) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/utils.py", line > 169, in trace_logging_wrapper > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return f(*args, > **kwargs) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connector.py", > line 240, in get_connector_pr > operties > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi execute=execute)) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", > line 70, in get_connec > tor_properties > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi initiator = > iscsi.get_initiator() > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", > line 963, in get_initi > ator > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi > root_helper=self._root_helper) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/executor.py", > line 53, in _execute > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi result = > self.__execute(*args, **kwargs) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", > line 172, in execute > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi return > execute_root(*cmd, **kwargs) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", > line 269, in _wrap > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi self.start() > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", > line 283, in start > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi channel = > daemon.RootwrapClientChannel(context=self) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", > line 374, in __init__ > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi > super(RootwrapClientChannel, self).__init__(sock, context) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", > line 187, in __init__ > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi > self.exchange_ping() > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi File > "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", > line 201, in exchange_ping > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi raise > FailedToDropPrivileges(msg) > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi *oslo_privsep.daemon.FailedToDropPrivileges: > Privsep daemon failed to start * > 2023-04-12 18:02:20.908 64 ERROR glance.common.wsgi > 2023-04-12 18:02:20.927 64 INFO eventlet.wsgi.server > [req-62883866-1b53-4032-bdc4-d9a294a837c4 0439953e7cfe4a13a1b4bb118b5dc3c4 > b0f76b5c6dcb457fa716762bbf954837 - default > default] 20.3.0.34,127.0.0.1 - - [12/Apr/2023 18:02:20] "PUT > /v2/images/52ed7ed7-330e-4249-abb9-5ec99712846f/file HTTP/1.1" 500 430 > 2.727683 > It seems like a lack of privileges, any ideas? Regards.

4 11

Re: [Port Creation failed] - openstack Wallaby
by Lokendra Rathour 30 May '23

30 May '23

Hi Swogat, Thanks for the inputs, it was showing a similar issue but somehow the issue is not getting resolved. we are trying to explore more around it. getting the error in ovn-metadata-agent.log Cannot find Chassis_Private with name=f80799b9-0cf4-4413-bb4b-e36278c73f6c detailed: 2023-05-17 19:26:31.984 45317 INFO oslo.privsep.daemon [-] privsep daemon running as pid 45317 2023-05-17 19:26:32.712 44735 ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run txn.results.put(txn.do_commit()) File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 93, in do_commit command.run_idl(txn) File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 172, in run_idl record = self.api.lookup(self.table, self.record) File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 208, in lookup return self._lookup(table, record) File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 268, in _lookup row = idlutils.row_by_value(self, rl.table, rl.column, record) File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value raise RowNotFound(table=table, col=column, match=match) ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis_Private with name=f80799b9-0cf4-4413-bb4b-e36278c73f6c 2023-05-17 19:26:32.713 44735 ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command (DbAddCommand): ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis_Private with name=f80799b9-0cf4-4413-bb4b-e36278c73f6c 2023-05-17 19:26:32.713 44735 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): 2023-05-17 19:26:32.713 44735 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 42, in execute waiting for your always helpful inputs. On Tue, May 16, 2023 at 10:47 PM Swogat Pradhan <swogatpradhan22(a)gmail.com> wrote: > Hi > I am not sure if this will help, but i faced something similar. > You might need to check the ovn database entries. > http://www.jimmdenton.com/neutron-ovn-private-chassis/ > > Or maybe try restarting the ovn service from pcs, sometimes issue comes up > when ovn doesn't sync up. > > Again m not sure if this will be of any help to you. > > With regards, > Swogat Pradhan > > On Tue, 16 May 2023, 10:41 pm Lokendra Rathour, <lokendrarathour(a)gmail.com> > wrote: > >> Hi All, >> Was trying to create OpenStack VM in OpenStack wallaby release, not able >> to create VM, it is failing because of Port not getting created. >> >> The error that we are getting: >> nova-compute.log: >> >> 2023-05-16 18:15:35.495 7 INFO nova.compute.provider_config >> [req-faaf38e7-b5ee-43d1-9303-d508285f5ab7 - - - - -] No provider configs >> found in /etc/nova/provider_config. If files are present, ensure the Nova >> process has access. >> 2023-05-16 18:15:35.549 7 ERROR nova.cmd.common >> [req-8842f11c-fe5a-4ad3-92ea-a6898f482bf0 - - - - -] No db access allowed >> in nova-compute: File "/usr/bin/nova-compute", line 10, in <module> >> sys.exit(main()) >> File "/usr/lib/python3.6/site-packages/nova/cmd/compute.py", line 59, >> in main >> topic=compute_rpcapi.RPC_TOPIC) >> File "/usr/lib/python3.6/site-packages/nova/service.py", line 264, in >> create >> utils.raise_if_old_compute() >> File "/usr/lib/python3.6/site-packages/nova/utils.py", line 1068, in >> raise_if_old_compute >> ctxt, ['nova-compute']) >> File "/usr/lib/python3.6/site-packages/nova/objects/service.py", line >> 563, in get_minimum_version_all_cells >> binaries) >> File "/usr/lib/python3.6/site-packages/nova/context.py", line 544, in >> scatter_gather_all_cells >> fn, *args, **kwargs) >> File "/usr/lib/python3.6/site-packages/nova/context.py", line 432, in >> scatter_gather_cells >> with target_cell(context, cell_mapping) as cctxt: >> File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ >> return next(self.gen) >> >> >> neutron/ovn-metadata-agent.log >> >> 2023-05-16 22:33:41.871 45204 ERROR ovsdbapp.backend.ovs_idl.command >> ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis_Private >> with name=b6e6f0d3-40c6-4d4e-8ef6-c935fa027bd2 >> 2023-05-16 22:33:41.871 45204 ERROR ovsdbapp.backend.ovs_idl.command >> 2023-05-16 22:36:41.876 45204 ERROR ovsdbapp.backend.ovs_idl.transaction >> [-] Traceback (most recent call last): >> File >> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", >> line 131, in run >> txn.results.put(txn.do_commit()) >> File >> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", >> line 93, in do_commit >> command.run_idl(txn) >> File >> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", >> line 172, in run_idl >> record = self.api.lookup(self.table, self.record) >> File >> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", >> line 208, in lookup >> return self._lookup(table, record) >> File >> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", >> line 268, in _lookup >> row = idlutils.row_by_value(self, rl.table, rl.column, record) >> File >> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", >> line 114, in row_by_value >> raise RowNotFound(table=table, col=column, match=match) >> ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find >> Chassis_Private with name=b6e6f0d3-40c6-4d4e-8ef6-c935fa027bd2 >> >> any input to help get this issue fixed would be of great help. >> thanks >> -- >> ~ Lokendra >> skype: lokendrarathour >> >> >> -- ~ Lokendra skype: lokendrarathour

2 3

[tc] Technical Committee next weekly meeting on May 30, 2023
by Nikolla, Kristi 29 May '23

29 May '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held on Tuesday, May 30, 2023 at 1800 UTC on #openstack-tc on OFTC IRC. The up-to-date agenda for the meeting can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting Thank you, Kristi Nikolla

1 0

[neutron] Bug Deputy Report May 22 - 28
by Yatin Karel 29 May '23

29 May '23

Hello Neutron Team !! Please find bug report from May 22nd to 28th, Undecided bugs needs further triage:- *Critical:-* - https://bugs.launchpad.net/neutron/+bug/2020363 - [stable/train/] openstacksdk-functional-devstack fails with POST_FAILURE Fixed with https://review.opendev.org/c/openstack/neutron/+/883890 Assigned to ykarel *High:-* - https://bugs.launchpad.net/neutron/+bug/2020698 - neutron-tempest-plugin-bgpvpn-bagpipe job unstable Job made non voting temporary with https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/884270 Assigned to bhaley - https://bugs.launchpad.net/neutron/+bug/2021457 - [fwaas]The firewall group without any port in active status Fix proposed https://review.opendev.org/c/openstack/neutron-fwaas/+/884335 Assigned to ZhouHeng *Medium:-* - https://bugs.launchpad.net/neutron/+bug/2020802 - Make DB migration "Add indexes to RBACs" conditional Unassigned *Low:-* - https://bugs.launchpad.net/neutron/+bug/2020552 - trunk_details missing sub port MAC addresses for LIST Requested more info Unassigned *Wishlist:-* - https://bugs.launchpad.net/neutron/+bug/2020358 - [RFE] Allow to limit conntrack entries per tenant to avoid "nf_conntrack: table full, dropping packet" Unassigned - https://bugs.launchpad.net/neutron/+bug/2020823 - [RFE] Add flavor/service provider support to routers in the L3 OVN plugin Assigned to Miguel Lavalle *Undecided:-* - https://bugs.launchpad.net/neutron/+bug/2020771 - [ml2/ovn] Binding_host_id shouldn't be set when a port is down Requested more info - https://bugs.launchpad.net/neutron/+bug/2020349 - Neutron Dynamic Routing : Connection to peers lost Discussion ongoing, possible misconfiguration - https://bugs.launchpad.net/neutron/+bug/2020328 - Concurrent create VM failed because of vif plug timeout Requested more info Thanks and Regards Yatin Karel

1 0

[nova][ops] EOL'ing stable/train ?
by Sylvain Bauza 26 May '23

26 May '23

Hi folks, in particular operators... We discussed yesterday during the nova meeting [1] about our stable branches and eventually, we were wondering whether we should EOL [2] the stable/train branch for Nova. Why so ? Two points : 1/ The gate is failing at the moment for the branch. 2/ Two CVEs (CVE-2022-47951 [3] and CVE-2023-2088 [4]) aren't fixed in this branch. It would be difficult to fix the CVEs in the upstream branch but hopefully AFAIK all the OpenStack distros already fixed them for their related releases that use Train. So, any concerns ? TBH, I'm not really happy with EOL, but it would be bizarre if we say "oh yeah we support Train backports" but we don't really fix the most important issues... -Sylvain (who will propose the train-eol tag change next week if he doesn't see any concern before) [1] https://meetings.opendev.org/meetings/nova/2023/nova.2023-05-23-16.01.log.h… [2] https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-l… [3] https://security.openstack.org/ossa/OSSA-2023-002.html [4] https://security.openstack.org/ossa/OSSA-2023-003.html

4 3

[nova]vnc and spice used together in openstack
by 啧啧啧 26 May '23

26 May '23

Hi, When using the console, we enabled both nova spicehtml5proxy and nova novancproxy on the controller node, and enabled vnc and spice on compute1 and compute2, respectively. We found that there was a problem with the coexistence of the two, The log is as follows.  ERROR oslo_messaging.rpc.server [req-28674cdd-f06e-466f-8611-516efc8fbe3c 069c1371a95e4ebaa59eea6ee70362bc dd4e9ff713be4311978334ae0d0c0a5e - default default] Exception during message handling: ValueError: Field `port' cannot be None 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message) 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args) 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args) 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 235, in inner 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs) 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb) 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     self.force_reraise() 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb) 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     raise value 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw) 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 219, in decorated_function 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info()) 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     self.force_reraise() 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb) 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     raise value 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 207, in decorated_function 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs) 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 5906, in get_spice_console 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     access_url_base=CONF.spice.html5proxy_base_url, 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 307, in __init__ 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     setattr(self, key, kwargs[key]) 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 72, in setter 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     field_value = field.coerce(self, name, value) 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/fields.py", line 207, in coerce 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     return self._null(obj, attr) 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/fields.py", line 185, in _null 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server     raise ValueError(_("Field `%s' cannot be None") % attr) 2023-05-25 10:58:05.851 7 ERROR oslo_messaging.rpc.server ValueError: Field `port' cannot be None I don't know why there is an appeal issue or what special mechanism OpenStack has for the console. I can only choose one. I hope you can help us. Thank you very much.

2 1

[neutron] Neutron drivers meeting cancelled
by Rodolfo Alonso Hernandez 26 May '23

26 May '23

Hello Neutrinos: Due to the lack of agenda [1], today's meeting is cancelled. Have a nice weekend! [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers

1 0

pep8 error of newest devstack
by Alex Song (宋文平) 26 May '23

26 May '23

Hi, When I check pep8 with the newest devstack env by the cmd ��tox �Ce pep8��, it raise many packages conflict. The log is as follows. root@song-ctrl:/opt/stack/cyborg# tox -e pep8 pep8 create: /opt/stack/cyborg/.tox/shared pep8 installdeps: -chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/cyborg/requirements.txt, -r/opt/stack/cyborg/test-requirements. txt ERROR: invocation failed (exit code 1), logfile: /opt/stack/cyborg/.tox/shared/log/pep8-1.log ============================================================================ ============================================================== log start ============================================================================ ============================================================== Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Collecting pbr!=2.1.0,>=0.11 (from -r /opt/stack/cyborg/requirements.txt (line 5)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/01/06/4ab11bf70db5a60689fc521b636 849c8593eb67a2c6bdf73a16c72d16a12/pbr-5.11.1-py2.py3-none-any.whl (112 kB) Collecting pecan!=1.0.2,!=1.0.3,!=1.0.4,!=1.2,>=1.0.0 (from -r /opt/stack/cyborg/requirements.txt (line 6)) Using cached pecan-1.4.2-py3-none-any.whl Collecting WSME>=0.10.1 (from -r /opt/stack/cyborg/requirements.txt (line 7)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d2/a0/54b71fd2e1ab4f838acf472726a 14dadb6d1c77adc18d7e2c062dd955ff9/WSME-0.11.0-py3-none-any.whl (59 kB) Collecting eventlet>=0.26.0 (from -r /opt/stack/cyborg/requirements.txt (line 8)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/90/97/928b89de2e23cc67136eccccf1c 122adf74ffdb65bbf7d2964b937cedd4f/eventlet-0.33.3-py2.py3-none-any.whl (226 kB) Collecting oslo.i18n>=1.5.0 (from -r /opt/stack/cyborg/requirements.txt (line 9)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/62/5a/ced9667f5a35d712734bf3449af ad763283e31ea9a903137eb42df29d948/oslo.i18n-6.0.0-py3-none-any.whl (46 kB) Collecting oslo.config!=4.3.0,!=4.4.0,>=1.1.0 (from -r /opt/stack/cyborg/requirements.txt (line 10)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ce/15/97fb14b7a1692693610a8e00e2a 08e4186d6cdd875b6ac24c912a429b665/oslo.config-9.1.1-py3-none-any.whl (128 kB) Collecting oslo.log>=5.0.0 (from -r /opt/stack/cyborg/requirements.txt (line 11)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/9c/24/526545a76513c741c65eeb70a85 c93287f10665147c7cff7e0eb24918d43/oslo.log-5.2.0-py3-none-any.whl (71 kB) Collecting oslo.context>=2.9.0 (from -r /opt/stack/cyborg/requirements.txt (line 12)) Using cached https://pypi.tuna.tsinghua.edu.cn/packages/7f/69/3a16785c49890cce2f7f14ee652 5c76e7116f9dad44f122f5e3670e43970/oslo.context-5.1.1-py3-none-any.whl (20 kB) ERROR: Cannot install oslo.messaging>=14.1.0 because these package versions have conflicting dependencies. The conflict is caused by: The user requested oslo.messaging>=14.1.0 The user requested (constraint) oslo-messaging===14.3.0 To fix this you could try to: 1. loosen the range of package versions you've specified 2. remove package versions to allow pip attempt to solve the dependency conflict ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dep endency-conflicts ============================================================================ =============================================================== log end ============================================================================ =============================================================== ERROR: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/cyborg/requirements.txt, -r/opt/stack/cyborg/test-requirements. txt]; v = InvocationError('/opt/stack/cyborg/.tox/shared/bin/python -m pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/cyborg/requirements.txt -r/opt/stack/cyborg/test-requirements.txt', 1) ____________________________________________________________________________ _______________________________________________________________ summary ____________________________________________________________________________ _______________________________________________________________ ERROR: pep8: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/cyborg/requirements.txt, -r/opt/stack/cyborg/test-requirements. txt]; v = InvocationError('/opt/stack/cyborg/.tox/shared/bin/python -m pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/cyborg/requirements.txt -r/opt/stack/cyborg/test-requirements.txt', 1) I hope you can help us. Thank you very much.

3 5

[Glance] [Ceph] [Zed] Glance no longer working after upgrading Ceph cluster to 17.2.6
by Yuri Blas 25 May '23

25 May '23

2 2

[kolla-ansible][yoga] Installation failure while `creating shard root mysql user`
by Rob Jefferson 25 May '23

25 May '23

I'm trying to set up Yoga (as part of a way of upgrading from Xena to Yoga to Zed) using kolla-ansible, and I'm finding that I'm getting the following error at the `kolla-ansible/ansible/roles/mariadb/tasks/register.yml` playbook: TASK [mariadb : Creating shard root mysql user] [...] fatal: [control01]: FAILED! => { "changed": false, "msg": "Can not parse the inner module output: b'[WARNING]: Failure using method (v2_playbook_on_play_start) in callback plugin (<ansible_collections.ansible.posix.plugins.callback.json.CallbackModule object at 0x7fa4e6e18d00>): 'Play' object has no attribute 'get_path' [WARNING]: Failure using method (v2_playbook_on_task_start) in callback plugin (<ansible_collections.ansible.posix.plugins.callback.json.CallbackModule object at 0x7fa4e6e18d00>): list index out of range [WARNING]: Failure using method (v2_runner_on_ok) in callback plugin (<ansible_collections.ansible.posix.plugins.callback.json.CallbackModule object at 0x7fa4e6e18d00>): list index out of range { "custom_stats": {}, "global_custom_stats": {}, "plays": [], "stats\": { "localhost": { "changed\": 0, "failures\": 0, "ignored\": 0, "ok": 1, "rescued\": 0, "skipped\": 0, "unreachable\": 0 } } }' "} I am not sure how to fix this error, or determine what's causing it. I've tried matching up Ansible versions between my installation environment and Yoga, I've tried upgrading Kolla, and so forth. For the installation environment, I'm using: * Debian 11 * Ansible 4.10.0 * ansible-core 2.11.2 * docker-ce 5.20.10 Any help with this would be most appreciated. Thanks, Rob

4 6

Cinder Bug Report 2023-05-25
by Sofia Enriquez 25 May '23

25 May '23

Hello Argonauts, Sorry for the late email. Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> *Low* - Cached images duplicated per host. <https://bugs.launchpad.net/cinder/+bug/2020733> - *Status:* Unassigned. - [rbac] User with Reader role can extend/reserve/retype/unreserve/update_readonly volume. <https://bugs.launchpad.net/cinder/+bug/2020261> - *Status:* Tempest proposed to master. <https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/883182> - [rbac] User with Reader role can create/delete/update/set-bootable volume. <https://bugs.launchpad.net/cinder/+bug/2020113> - *Status:* Tempest proposed to master. <https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/878672> Cheers -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[ironic] Deploy images crashing Dell server BIOS using UEFI boot
by Mike Currin 25 May '23

25 May '23

Hi All, We have a Xena based Openstack deployment, recently we deployed 60+ nodes in our research cluster with Ironic which worked well. All of these were deployed using a standard process I'll describe below. We recently took delivery of a new Dell R6625 server with NVMe devices onlym which only support UEFI boot, so we are trying to get that working. The server PXEs and downloads the RAM disk and then the Deploy image, once running that it immediately crashes (I assume when running linuxefi). We tested UEFI deploy on an existing Dell R640 server, that server works with BIOS but we swapped it over to UEFI and it does the same, so it wasn't due to the much bigger/different architecture (AMD vs Intel) server. We have a few older servers in a test setup (which are Dell R630's) which are working fine and don't do this behaviour. We haven't tried them on our production setup as if even if they worked it wouldn't help us move forward. I made a video showing this: https://www.dropbox.com/s/5jbn1qpylxaevqb/uefiboot2.mov?dl=0 In the iDRAC we just get that the "System BIOS has halted" and somewhere I said to change hardware that you recently added, which feels unlikely as 2 different servers both working elsewhere with totally different hardware,. I've done a iDRAC Serial console debug but it isn't showing me much that is of any use. This is our entire process to deploy a node (some is once off of course, I've not included the network setup): openstack flavor create --ram 256000 --disk 20 --vcpus 32 --public our-baremetal openstack flavor set our-baremetal --property capabilities:boot_mode="uefi" We downloaded the latest (Xena) images from: https://tarballs.opendev.org/openstack/ironic-python-agent/dib/files/ I also tried the latest Centos9 ones just to try them, made no difference. We then extract the image and make a small mod to the service start (we found it didn't bring the NIC immediately up so put a ping delay in the ExecStart), but that's not part of this problem. openstack image create --disk-format aki --container-format aki --public --file ironic-agent.kernel deploy-vmlinuz openstack image create --disk-format ari --container-format ari --public --file ironic-agent.initramfs-ping-patched deploy-initrd Then to do the deploy: export HOSTNAME=<hostname> export MGMTIP=<idracip> openstack baremetal node create --driver ipmi --name $HOSTNAME --driver-info ipmi_port=623 --driver-info ipmi_username=root --driver-info 'ipmi_password=<ourpassword>' --driver-info ipmi_address=$MGMTIP --resource-class baremetal-resource-class --property cpus=32 --property memory_mb=256000 --property local_gb=20 --property cpu_arch=x86_64 --driver-info deploy_ramdisk=$(openstack image show deploy-initrd -f value -c id) --driver-info deploy_kernel=$(openstack image show deploy-vmlinuz -f value -c id) NODE=$(openstack baremetal node show -f value -c uuid $HOSTNAME) openstack baremetal node set $NODE --property capabilities='boot_mode:uefi' openstack baremetal port create <MACADDRESS> --node $NODE --physical-network physnet3 openstack baremetal node manage $NODE --wait && openstack baremetal node list && openstack baremetal node provide $NODE && openstack baremetal node list openstack server create --use-config-drive --image <ourimage> --flavor our-baremetal --security-group worker --network ironic-network --key-name <ourkeyname> servername Does any one have any more info to help or any suggestions as to something more I could try, I'm out of ideas. I know that UEFI itself works on both the servers, we have a setup with Ubuntu MAAS and it can deploy perfectly fine using its process with the UEFI setup so, it's something on the Ironic deploy image that's causing us this problem. Regards, Mike

2 1

upgrade issue with nova/cinder and api version error
by Jim Kilborn 25 May '23

25 May '23

Hello, First time posting here. We have been running a production openstack environment at my office since the kilo release. We are currently on train, and I'm trying to get up to a more recent version. To make it more difficult, we are on centos7, so having to switch to ubuntu as we update versions. The problem that I am having after updaing to victoria, is that when I delete a vm via horizon, the instance disappears but the cinder volume doesn't delete the attachment. It appears this is due to the following error in /var/log/apache2/cinder_error.log ERROR cinder.api.middleware.fault novaclient.exceptions.NotAcceptable: Version 2.89 is not supported by the API. Minimum is 2.1 and maximum is 2.87. (HTTP 406) When I look at the /usr/lib/python3/dist-packages/cinder/compute/nova.py I can see it's using 2.89 in get_server_volume def get_server_volume(context, server_id, volume_id): # Use microversion that includes attachment_id nova = novaclient(context, api_version='2.89') return nova.volumes.get_server_volume(server_id, volume_id) I am not sure why cinder and nova are in disagreement on the api_version. I have verified that they are both upgraded to the victoria release. Anyone have any ideas as to why I would be getting this error or a possible fix? I haven't been able to find any information on this error. Here are the nova package versions: nova-api/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed] nova-common/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed] nova-conductor/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed] nova-novncproxy/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed] nova-scheduler/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed] python3-nova/focal-updates,now 2:22.4.0-0ubuntu1~cloud3 all [installed] python3-novaclient/focal-updates,now 2:17.2.1-0ubuntu1~cloud0 all [installed,automatic] Here are the cinder package versions: cinder-api/focal-updates,now 2:17.4.0-0ubuntu1~cloud3 all [installed] cinder-common/focal-updates,now 2:17.4.0-0ubuntu1~cloud3 all [installed,automatic] cinder-scheduler/focal-updates,now 2:17.4.0-0ubuntu1~cloud3 all [installed] cinder-volume/focal-updates,now 2:17.4.0-0ubuntu1~cloud3 all [installed] python3-cinder/focal-updates,now 2:17.4.0-0ubuntu1~cloud3 all [installed,automatic] python3-cinderclient/focal-updates,now 1:7.2.0-0ubuntu1~cloud0 all [installed] Thanks in advance for any ideas!

3 4

[neutron] Resignation announcement of Nate, Hongbin and Takashi
by Rodolfo Alonso Hernandez 25 May '23

25 May '23

Hello all: Today I would like to announce that Nate Johnston, Takashi Yamamoto and Hongbin Lu are stepping down as Neutron core reviewers and Neutron drivers. Before sending this email, I contacted them and Nate and Takashi agreed on this decision; I tried twice to contact Hongbin with no response. In order to have a participative Neutron community, the Neutron core and Neutron drivers teams should be made up of active members in code reviews, meeting attendance and spec reviewal. Nate, Takashi and Hongbin have been very active and valuable members of the Neutron (and OpenStack) community in the past; for that I would like to thank you for your contributions. Tomorrow I'll update the Launchpad and gerrit groups. Regards.

2 1

Message d'erreur
by BEDDA Fadhel 24 May '23

24 May '23

Good morning who can help me to work around this problem: root@controller certs(keystone)# openstack project create --domain default --description "Service Project" service Failed to discover available identity versions when contacting https://controller.cloud.net:5000/v3. Attempting to parse version from URL. SSL exception connecting to https://controller.cloud.net:5000/v3/auth/tokens: HTTPSConnectionPool(host='controller.cloud.net', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:997)')))

2 1

[kolla] How to patch images during build
by Satish Patel 24 May '23

24 May '23

Folks, I'm using kolla-build to build all images and push them to the local repo. So far all good but let's say If in future for some reason I want to patch some bug and rebuild the image in that case how do i patch kolla images? I am reading at [1] and didn't see any example to patch Image for any bug. Should I be downloading tarball and patch it and then use type=local to build the image from local source or add patch to Dockerfile? What people use for best practice for this kind of thing? [1] https://docs.openstack.org/kolla/latest/admin/image-building.html

5 16

[kolla] [train] [keystone] Number of User/Group entities returned by LDAP exceeded size limit
by Albert Braden 24 May '23

24 May '23

We have 200 groups in our LDAP server. We recently started getting an error when we try to list groups: $ os group list --domain AUTH.OURDOMAIN.COM Number of User/Group entities returned by LDAP exceeded size limit. Contact your LDAP administrator. (HTTP 500) I read the "Additional LDAP integration settings" section in [1] and then tried setting various values of page_size (10, 100, 1000) in the [ldap] section of keystone.conf but that didn't make a difference. What am I missing? [1] https://docs.openstack.org/keystone/train/admin/configuration.html#identity… Here's the stack trace: 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application [req-198741c6-58b2-46b1-8622-bae1fc5c5280 d64c83e1ea954c368e9fe08a5d8450a1 47dc15c280c9436fadac4d41f1d54a64 - default default] Number of User/Group entities returned by LDAP exceeded size limit. Contact your LDAP administrator.: keystone.exception.LDAPSizeLimitExceeded: Number of User/Group entities returned by LDAP exceeded size limit. Contact your LDAP administrator. 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application Traceback (most recent call last): 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/keystone/identity/backends/ldap/common.py", line 996, in search_s 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application attrlist, attrsonly) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/keystone/identity/backends/ldap/common.py", line 689, in wrapper 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application return func(self, conn, *args, **kwargs) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/keystone/identity/backends/ldap/common.py", line 824, in search_s 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application attrsonly) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 870, in search_s 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 1286, in search_ext_s 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application return self._apply_method_s(SimpleLDAPObject.search_ext_s,*args,**kwargs) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 1224, in _apply_method_s 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application return func(self,*args,**kwargs) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 864, in search_ext_s 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application return self.result(msgid,all=1,timeout=timeout)[1] 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 756, in result 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application resp_type, resp_data, resp_msgid = self.result2(msgid,all,timeout) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 760, in result2 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all,timeout) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 767, in result3 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application resp_ctrl_classes=resp_ctrl_classes 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 774, in result4 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 340, in _ldap_call 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application reraise(exc_type, exc_value, exc_traceback) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib64/python3.6/site-packages/ldap/compat.py", line 46, in reraise 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application raise exc_value 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 324, in _ldap_call 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application result = func(*args,**kwargs) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application ldap.SIZELIMIT_EXCEEDED: {'msgtype': 100, 'msgid': 2, 'result': 4, 'desc': 'Size limit exceeded', 'ctrls': []} 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application Traceback (most recent call last): 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application rv = self.dispatch_request() 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application return self.view_functions[rule.endpoint](**req.view_args) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/flask_restful/__init__.py", line 480, in wrapper 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application resp = resource(*args, **kwargs) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/flask/views.py", line 88, in view 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application return self.dispatch_request(*args, **kwargs) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/flask_restful/__init__.py", line 595, in dispatch_request 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application resp = meth(*args, **kwargs) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/keystone/api/groups.py", line 59, in get 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application return self._list_groups() 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/keystone/api/groups.py", line 86, in _list_groups 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application hints=hints) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/keystone/common/manager.py", line 116, in wrapped 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/keystone/identity/core.py", line 414, in wrapper 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application return f(self, *args, **kwargs) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/keystone/identity/core.py", line 424, in wrapper 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application return f(self, *args, **kwargs) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/keystone/identity/core.py", line 1329, in list_groups 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application ref_list = driver.list_groups(hints) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/keystone/identity/backends/ldap/core.py", line 116, in list_groups 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application return self.group.get_all_filtered(hints) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/keystone/identity/backends/ldap/core.py", line 474, in get_all_filtered 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application for group in self.get_all(query, hints)] 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/keystone/identity/backends/ldap/common.py", line 1647, in get_all 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application for x in self._ldap_get_all(hints, ldap_filter)] 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/keystone/common/driver_hints.py", line 42, in wrapper 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application return f(self, hints, *args, **kwargs) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/keystone/identity/backends/ldap/common.py", line 1600, in _ldap_get_all 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application attrs) 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application File "/usr/lib/python3.6/site-packages/keystone/identity/backends/ldap/common.py", line 998, in search_s 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application raise exception.LDAPSizeLimitExceeded() 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application keystone.exception.LDAPSizeLimitExceeded: Number of User/Group entities returned by LDAP exceeded size limit. Contact your LDAP administrator. 2023-05-15 20:18:41.932 36 ERROR keystone.server.flask.application

2 4

[cinder-backup][ceph] cinder-backup support of incremental backup with ceph backend
by Satish Patel 24 May '23

24 May '23

Folks, I have ceph storage for my openstack and configure cinder-volume and cinder-backup service for my disaster solution. I am trying to use the cinder-backup incremental option to save storage space but somehow It doesn't work the way it should work. Whenever I take incremental backup it shows a similar size of original volume. Technically It should be smaller. Question is does ceph support incremental backup with cinder? I am running a Yoga release. $ openstack volume list +--------------------------------------+------------+------------+------+-------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+------------+------------+------+-------------------------------------+ | 285a49a6-0e03-49e5-abf1-1c1efbfeb5f2 | spatel-vol | backing-up | 10 | Attached to spatel-foo on /dev/sdc | +--------------------------------------+------------+------------+------+-------------------------------------+ ### Create full backup $ openstack volume backup create --name spatel-vol-backup spatel-vol --force +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | 4351d9d3-85fa-4cd5-b21d-619b3385aefc | | name | spatel-vol-backup | +-------+--------------------------------------+ ### Create incremental $ openstack volume backup create --name spatel-vol-backup-1 --incremental --force spatel-vol +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | 294b58af-771b-4a9f-bb7b-c37a4f84d678 | | name | spatel-vol-backup-1 | +-------+--------------------------------------+ $ openstack volume backup list +--------------------------------------+---------------------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+---------------------+-------------+-----------+------+ | 294b58af-771b-4a9f-bb7b-c37a4f84d678 | spatel-vol-backup-1 | None | available | 10 | | 4351d9d3-85fa-4cd5-b21d-619b3385aefc | spatel-vol-backup | None | available | 10 | +--------------------------------------+---------------------+-------------+-----------+------+ My incremental backup still shows 10G size which should be lower compared to the first backup.

6 14