- openstack-discuss - lists.openstack.org

the volume cannot be deleted, it is in deleting state
by kjme001＠gmail.com 15 Nov '23

15 Nov '23

hello, I'm having trouble deleting volumes in OpenStackl from a machine that no longer exists. How to execute cinder command reset-state --attach-status detached volume_name then I execute the command: # cinder delete volume_name Delete for volume cifar-10-copy failed: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-b2d3f2e4-7855-4831-a68d-f6710cc97e94) ERROR: Unable to delete any of the specified volumes. then executes the command: cinder reset-state --state available volume_name and cinder delete volume_name Request to delete volume volume_name has been accepted. when I check: openstack volume show volume_name is: | status | deleting The volume is not being deleted, it is in the deleting state. This is a volume from a non-existent node, but the volume is still visible in OpenStack and cannot be deleted because it is in the deleting state. Maybe someone can tell me how to remove such a volume? Regards

1 0

cinder-manage service remove cinder-volume xxxx@lvm-ssd - error
by kjme001＠gmail.com 14 Nov '23

14 Nov '23

cinder-manage service remove cinder-volume xxxx@lvm-ssd /usr/lib/python3/dist-packages/cinder/db/sqlalchemy/models.py:152: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery. last_heartbeat = column_property( /usr/lib/python3/dist-packages/cinder/db/sqlalchemy/models.py:160: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery. num_hosts = column_property( /usr/lib/python3/dist-packages/cinder/db/sqlalchemy/models.py:169: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery. num_down_hosts = column_property( 2023-11-11 14:34:00.928 898920 CRITICAL cinder [req-525973bb-e026-4999-95af-d852aa137066 - - - - -] Unhandled error: oslo_db.exception.DBNonExistentDatabase: (sqlite3.OperationalError) unable to open database file (Background on this error at: https://sqlalche.me/e/14/e3q8) 2023-11-11 14:34:00.928 898920 ERROR cinder Traceback (most recent call last): 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 3250, in _wrap_pool_connect 2023-11-11 14:34:00.928 898920 ERROR cinder return fn() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 310, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return _ConnectionFairy._checkout(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 868, in _checkout 2023-11-11 14:34:00.928 898920 ERROR cinder fairy = _ConnectionRecord.checkout(pool) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 476, in checkout 2023-11-11 14:34:00.928 898920 ERROR cinder rec = pool._do_get() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 256, in _do_get 2023-11-11 14:34:00.928 898920 ERROR cinder return self._create_connection() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 256, in _create_connection 2023-11-11 14:34:00.928 898920 ERROR cinder return _ConnectionRecord(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 371, in __init__ 2023-11-11 14:34:00.928 898920 ERROR cinder self.__connect() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 665, in __connect 2023-11-11 14:34:00.928 898920 ERROR cinder with util.safe_reraise(): 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ 2023-11-11 14:34:00.928 898920 ERROR cinder compat.raise_( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_ 2023-11-11 14:34:00.928 898920 ERROR cinder raise exception 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 661, in __connect 2023-11-11 14:34:00.928 898920 ERROR cinder self.dbapi_connection = connection = pool._invoke_creator(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/create.py", line 590, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return dialect.connect(*cargs, **cparams) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 597, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return self.dbapi.connect(*cargs, **cparams) 2023-11-11 14:34:00.928 898920 ERROR cinder sqlite3.OperationalError: unable to open database file 2023-11-11 14:34:00.928 898920 ERROR cinder 2023-11-11 14:34:00.928 898920 ERROR cinder The above exception was the direct cause of the following exception: 2023-11-11 14:34:00.928 898920 ERROR cinder 2023-11-11 14:34:00.928 898920 ERROR cinder Traceback (most recent call last): 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/bin/cinder-manage", line 10, in <module> 2023-11-11 14:34:00.928 898920 ERROR cinder sys.exit(main()) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/cmd/manage.py", line 1069, in main 2023-11-11 14:34:00.928 898920 ERROR cinder fn(**fn_kwargs) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/cmd/manage.py", line 711, in remove 2023-11-11 14:34:00.928 898920 ERROR cinder svc = objects.Service.get_by_args(ctxt, host_name, binary) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/objects/service.py", line 133, in get_by_args 2023-11-11 14:34:00.928 898920 ERROR cinder db_service = db.service_get(context, host=host, binary=binary_key) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/api.py", line 270, in service_get 2023-11-11 14:34:00.928 898920 ERROR cinder return IMPL.service_get(context, service_id, backend_match_level, 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 168, in wrapper 2023-11-11 14:34:00.928 898920 ERROR cinder return f(*args, **kwargs) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 745, in service_get 2023-11-11 14:34:00.928 898920 ERROR cinder query = _service_query(context, backend_match_level=backend_match_level, 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 686, in _service_query 2023-11-11 14:34:00.928 898920 ERROR cinder query = model_query(context, models.Service, session=session, 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 259, in model_query 2023-11-11 14:34:00.928 898920 ERROR cinder session = kwargs.get('session') or get_session() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 95, in get_session 2023-11-11 14:34:00.928 898920 ERROR cinder return main_context_manager._factory.get_legacy_facade().get_session( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 357, in get_legacy_facade 2023-11-11 14:34:00.928 898920 ERROR cinder self._start() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 504, in _start 2023-11-11 14:34:00.928 898920 ERROR cinder self._setup_for_connection( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 529, in _setup_for_connection 2023-11-11 14:34:00.928 898920 ERROR cinder engine = engines.create_engine( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator 2023-11-11 14:34:00.928 898920 ERROR cinder return wrapped(*args, **kwargs) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 211, in create_engine 2023-11-11 14:34:00.928 898920 ERROR cinder test_conn = _test_connection(engine, max_retries, retry_interval) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 386, in _test_connection 2023-11-11 14:34:00.928 898920 ERROR cinder return engine.connect() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 3204, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return self._connection_cls(self, close_with_result=close_with_result) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 96, in __init__ 2023-11-11 14:34:00.928 898920 ERROR cinder else engine.raw_connection() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 3283, in raw_connection 2023-11-11 14:34:00.928 898920 ERROR cinder return self._wrap_pool_connect(self.pool.connect, _connection) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 3253, in _wrap_pool_connect 2023-11-11 14:34:00.928 898920 ERROR cinder Connection._handle_dbapi_exception_noconnection( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2098, in _handle_dbapi_exception_noconnection 2023-11-11 14:34:00.928 898920 ERROR cinder util.raise_(newraise, with_traceback=exc_info[2], from_=e) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_ 2023-11-11 14:34:00.928 898920 ERROR cinder raise exception 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 3250, in _wrap_pool_connect 2023-11-11 14:34:00.928 898920 ERROR cinder return fn() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 310, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return _ConnectionFairy._checkout(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 868, in _checkout 2023-11-11 14:34:00.928 898920 ERROR cinder fairy = _ConnectionRecord.checkout(pool) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 476, in checkout 2023-11-11 14:34:00.928 898920 ERROR cinder rec = pool._do_get() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 256, in _do_get 2023-11-11 14:34:00.928 898920 ERROR cinder return self._create_connection() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 256, in _create_connection 2023-11-11 14:34:00.928 898920 ERROR cinder return _ConnectionRecord(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 371, in __init__ 2023-11-11 14:34:00.928 898920 ERROR cinder self.__connect() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 665, in __connect 2023-11-11 14:34:00.928 898920 ERROR cinder with util.safe_reraise(): 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ 2023-11-11 14:34:00.928 898920 ERROR cinder compat.raise_( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_ 2023-11-11 14:34:00.928 898920 ERROR cinder raise exception 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 661, in __connect 2023-11-11 14:34:00.928 898920 ERROR cinder self.dbapi_connection = connection = pool._invoke_creator(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/create.py", line 590, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return dialect.connect(*cargs, **cparams) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 597, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return self.dbapi.connect(*cargs, **cparams) 2023-11-11 14:34:00.928 898920 ERROR cinder oslo_db.exception.DBNonExistentDatabase: (sqlite3.OperationalError) unable to open database file 2023-11-11 14:34:00.928 898920 ERROR cinder (Background on this error at: https://sqlalche.me/e/14/e3q8) 2023-11-11 14:34:00.928 898920 ERROR cinder

3 4

(no subject)
by mgutha＠liveperson.com 14 Nov '23

14 Nov '23

I have observed an imbalance in the cluster based on the object.builder output. Is this possibly due to an ongoing rebalance process? Additionally, I'd like to confirm my understanding of 'min_part_hours.' Is it accurate to say that this parameter represents the number of hours an object needs to reside on a disk before it's considered for movement to another device during a rebalance? According to the Swift documentation, the default value is 24 hours. I'm curious if this value is applied to data written specifically by the rebalance process or if it also includes data written by other Swift services. object.builder, build version 775, id 61b2166cb8684b2b80d6fc370cef37a6 65536 partitions, 2.000000 replicas, 2 regions, 2 zones, 95 devices, 38.89 balance, 0.00 dispersion The minimum number of hours before a partition can be reassigned is 0 (0:00:00 remaining) The overload factor is 10.00% (0.100000) Ring file object.ring.gz not found, probably it hasn't been written yet Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta 129 1 1 182.16.128.36:6000 182.16.128.36:6003 d129 1.00 2 38.89 130 1 1 182.16.128.36:6006 182.16.128.36:6003 d130 1.00 2 38.89 131 1 1 182.16.128.36:6007 182.16.128.36:6003 d131 1.00 2 38.89 132 1 1 182.16.128.36:6008 182.16.128.36:6003 d132 1.00 2 38.89 133 1 1 182.16.128.36:6009 182.16.128.36:6003 d133 1.00 2 38.89 134 1 1 182.16.128.36:6010 182.16.128.36:6003 d134 1.00 1 -30.56 135 1 1 182.16.128.36:6011 182.16.128.36:6003 d135 1.00 1 -30.56 136 1 1 182.16.128.36:6012 182.16.128.36:6003 d136 1.00 1 -30.56 137 1 1 182.16.128.36:6013 182.16.128.36:6003 d137 1.00 1 -30.56 138 1 1 182.16.128.36:6014 182.16.128.36:6003 d138 1.00 1 -30.56 139 1 1 182.16.128.36:6015 182.16.128.36:6003 d139 1.00 1 -30.56 140 1 1 182.16.128.38:6000 182.16.128.38:6003 d140 1.00 2 38.89 141 1 1 182.16.128.38:6006 182.16.128.38:6003 d141 1.00 2 38.89 142 1 1 182.16.128.38:6007 182.16.128.38:6003 d142 1.00 2 38.89 143 1 1 182.16.128.38:6008 182.16.128.38:6003 d143 1.00 2 38.89 144 1 1 182.16.128.38:6009 182.16.128.38:6003 d144 1.00 2 38.89 145 1 1 182.16.128.38:6010 182.16.128.38:6003 d145 1.00 1 -30.56 146 1 1 182.16.128.38:6011 182.16.128.38:6003 d146 1.00 1 -30.56 147 1 1 182.16.128.38:6012 182.16.128.38:6003 d147 1.00 1 -30.56 148 1 1 182.16.128.38:6013 182.16.128.38:6003 d148 1.00 1 -30.56 149 1 1 182.16.128.38:6014 182.16.128.38:6003 d149 1.00 1 -30.56 150 1 1 182.16.128.38:6015 182.16.128.38:6003 d150 1.00 1 -30.56 151 1 1 182.16.128.39:6000 182.16.128.39:6003 d151 1.00 2 38.89 152 1 1 182.16.128.39:6006 182.16.128.39:6003 d152 1.00 2 38.89 153 1 1 182.16.128.39:6007 182.16.128.39:6003 d153 1.00 2 38.89 154 1 1 182.16.128.39:6008 182.16.128.39:6003 d154 1.00 2 38.89 155 1 1 182.16.128.39:6009 182.16.128.39:6003 d155 1.00 2 38.89 156 1 1 182.16.128.39:6010 182.16.128.39:6003 d156 1.00 1 -30.56 157 1 1 182.16.128.39:6011 182.16.128.39:6003 d157 1.00 1 -30.56 158 1 1 182.16.128.39:6012 182.16.128.39:6003 d158 1.00 1 -30.56 159 1 1 182.16.128.39:6013 182.16.128.39:6003 d159 1.00 1 -30.56 160 1 1 182.16.128.39:6014 182.16.128.39:6003 d160 1.00 1 -30.56 161 1 1 182.16.128.39:6015 182.16.128.39:6003 d161 1.00 1 -30.56 55 1 1 182.16.128.5:6000 182.16.128.5:6003 d55 2000.40 2789 -3.18 56 1 1 182.16.128.5:6006 182.16.128.5:6003 d56 2000.40 2789 -3.18 57 1 1 182.16.128.5:6007 182.16.128.5:6003 d57 2000.40 2789 -3.18 58 1 1 182.16.128.5:6008 182.16.128.5:6003 d58 2000.40 2789 -3.18 59 1 1 182.16.128.5:6009 182.16.128.5:6003 d59 2000.40 2789 -3.18 60 1 1 182.16.128.5:6010 182.16.128.5:6003 d60 2000.40 2789 -3.18 61 1 1 182.16.128.5:6011 182.16.128.5:6003 d61 2000.40 2789 -3.18 62 1 1 182.16.128.5:6012 182.16.128.5:6003 d62 2000.40 2789 -3.18 63 1 1 182.16.128.5:6013 182.16.128.5:6003 d63 2000.40 2789 -3.18 64 1 1 182.16.128.5:6014 182.16.128.5:6003 d64 2000.40 2789 -3.18 65 1 1 182.16.128.5:6015 182.16.128.5:6003 d65 2000.40 2788 -3.21 44 1 1 182.16.128.6:6000 182.16.128.6:6003 d44 269.43 376 -3.09 45 1 1 182.16.128.6:6006 182.16.128.6:6003 d45 269.43 376 -3.09 46 1 1 182.16.128.6:6007 182.16.128.6:6003 d46 269.43 376 -3.09 47 1 1 182.16.128.6:6008 182.16.128.6:6003 d47 269.43 376 -3.09 48 1 1 182.16.128.6:6009 182.16.128.6:6003 d48 269.43 376 -3.09 49 1 1 182.16.128.6:6010 182.16.128.6:6003 d49 269.43 376 -3.09 50 1 1 182.16.128.6:6011 182.16.128.6:6003 d50 269.43 376 -3.09 51 1 1 182.16.128.6:6012 182.16.128.6:6003 d51 269.43 375 -3.35 52 1 1 182.16.128.6:6013 182.16.128.6:6003 d52 269.43 375 -3.35 53 1 1 182.16.128.6:6014 182.16.128.6:6003 d53 269.43 375 -3.35 54 1 1 182.16.128.6:6015 182.16.128.6:6003 d54 269.43 375 -3.35 0 1 1 182.16.128.7:6000 182.16.128.7:6003 d0 2000.40 2789 -3.18 1 1 1 182.16.128.7:6006 182.16.128.7:6003 d1 2000.40 2789 -3.18 10 1 1 182.16.128.7:6015 182.16.128.7:6003 d10 2000.40 2788 -3.21 2 1 1 182.16.128.7:6007 182.16.128.7:6003 d2 2000.40 2789 -3.18 3 1 1 182.16.128.7:6008 182.16.128.7:6003 d3 2000.40 2789 -3.18 4 1 1 182.16.128.7:6009 182.16.128.7:6003 d4 2000.40 2789 -3.18 5 1 1 182.16.128.7:6010 182.16.128.7:6003 d5 2000.40 2789 -3.18 6 1 1 182.16.128.7:6011 182.16.128.7:6003 d6 2000.40 2789 -3.18 7 1 1 182.16.128.7:6012 182.16.128.7:6003 d7 2000.40 2789 -3.18 8 1 1 182.16.128.7:6013 182.16.128.7:6003 d8 2000.40 2789 -3.18 9 1 1 182.16.128.7:6014 182.16.128.7:6003 d9 2000.40 2789 -3.18 11 2 2 11.17.128.5:6000 11.17.128.5:6003 d11 2000.40 2979 3.42 12 2 2 11.17.128.5:6006 11.17.128.5:6003 d12 2000.40 2979 3.42 13 2 2 11.17.128.5:6007 11.17.128.5:6003 d13 2000.40 2979 3.42 15 2 2 11.17.128.5:6008 11.17.128.5:6003 d15 2000.40 2979 3.42 16 2 2 11.17.128.5:6009 11.17.128.5:6003 d16 2000.40 2979 3.42 17 2 2 11.17.128.5:6010 11.17.128.5:6003 d17 2000.40 2978 3.38 18 2 2 11.17.128.5:6011 11.17.128.5:6003 d18 2000.40 2978 3.38 19 2 2 11.17.128.5:6012 11.17.128.5:6003 d19 2000.40 2978 3.38 20 2 2 11.17.128.5:6013 11.17.128.5:6003 d20 2000.40 2978 3.38 21 2 2 11.17.128.5:6014 11.17.128.5:6003 d21 2000.40 2978 3.38 77 2 2 11.17.128.5:6015 11.17.128.5:6003 d77 2000.40 2978 3.38 33 2 2 11.17.128.6:6000 11.17.128.6:6003 d33 2000.40 2979 3.42 34 2 2 11.17.128.6:6006 11.17.128.6:6003 d34 2000.40 2979 3.42 35 2 2 11.17.128.6:6007 11.17.128.6:6003 d35 2000.40 2979 3.42 36 2 2 11.17.128.6:6008 11.17.128.6:6003 d36 2000.40 2979 3.42 37 2 2 11.17.128.6:6009 11.17.128.6:6003 d37 2000.40 2978 3.38 38 2 2 11.17.128.6:6010 11.17.128.6:6003 d38 2000.40 2978 3.38 39 2 2 11.17.128.6:6011 11.17.128.6:6003 d39 2000.40 2978 3.38 40 2 2 11.17.128.6:6012 11.17.128.6:6003 d40 2000.40 2978 3.38 41 2 2 11.17.128.6:6013 11.17.128.6:6003 d41 2000.40 2978 3.38 42 2 2 11.17.128.6:6014 11.17.128.6:6003 d42 2000.40 2978 3.38 43 2 2 11.17.128.6:6015 11.17.128.6:6003 d43 2000.40 2978 3.38 162 2 2 11.17.128.71:6000 11.17.128.71:6003 d162 1.00 2 38.89 163 2 2 11.17.128.71:6006 11.17.128.71:6003 d163 1.00 2 38.89 164 2 2 11.17.128.71:6007 11.17.128.71:6003 d164 1.00 2 38.89 165 2 2 11.17.128.71:6008 11.17.128.71:6003 d165 1.00 2 38.89 166 2 2 11.17.128.71:6009 11.17.128.71:6003 d166 1.00 1 -30.56 167 2 2 11.17.128.71:6010 11.17.128.71:6003 d167 1.00 1 -30.56 168 2 2 11.17.128.71:6011 11.17.128.71:6003 d168 1.00 1 -30.56

3 2

Re: [swift]: Swiftstack Zone/DC decommission
by Muralikrishna Gutha 14 Nov '23

14 Nov '23

Clary, Thanks for the update I was unable to locate the object.builder file in the instance where we are running swiftstack. We are currently on Swift Package Version: 2.24.0.3-1.el7 [root@hostname swift]# /opt/ss/bin/swift-ring-builder object.builder | head Ring Builder file does not exist: object.builder [root@hostname swift]# locate object.builder [root@hostname swift]#pwd /etc/swift Thanks, Murali -- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.

2 1

[PTL][release][stable] Yoga - final release before branch becomes unmaintained
by Elõd Illés 14 Nov '23

14 Nov '23

Hi teams, According to the new process [1] Yoga should transition to Unmaintained as soon as possible (the original deadline for Extended Maintenance transition was planned to 2023-11-02). Similarly like before, teams have the final opportunity to release from stable/yoga before the branches will be *deleted*. In order to help the teams I have generated the list of the current *open* and *unreleased* changes in stable/yoga for repositories [2] where there are relevant patches. Please do a *final* release on Yoga if needed, before stable/yoga transition to Unmaintained. Feel free to edit and extend these lists to track your team's progress! *NOTE*: teams, please focus on wrapping up your libraries first if there is any concern about the changes, in order to avoid broken (final!!) releases! [1] https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branc… [2] https://etherpad.opendev.org/p/yoga-final-release-before-unmaintained [3] https://review.opendev.org/q/topic:xena-em Thanks, Előd irc: elodilles @ #openstack-release

2 1

[neutron][ptg] 2024.1/Caracal vPTG summary
by Brian Haley 14 Nov '23

14 Nov '23

Sorry for the slowness of getting this out. Here is a summary of the discussions we had over our two days of meetings, as well as other related sessions. The etherpad link can be found at [0]. Monday, October 23rd -------------------- # OS DBPerf Cross project session [1] This session was to talk about devstack performance issues that are believed to be caused by excessive querying by Neutron and Keystone. By default, when devstack "stacks", DB counter values are collected on each of the subsystems. Keystone and Neutron are outliers on how many SELECTs they issue during this process. As this is such a high-level number (4x to 10x other users), more work is required to understand what to fix, or if it's just normal behavior. For example, there is data from just a "stack" in addition to a tempest run that might be able to narrow things down here. Slawek and Miguel have agreed to work with the oslo.db team and Mike Bayer to get a better understanding of things here. Tuesday, October 24th --------------------- # Horizon For the last few cycles the Neutron team (Lajos) has been trying to deprecate usage of neutronclient. While this work is mostly complete, there is usage in Horizon that needs to be addressed. Lajos and Amotoki will be working on this. Full notes at [2] Wednesday, October 25th ----------------------- # Retrospective As is customary the team did a retrospective of the good, bad and what could be improved. In short, the Neutron team has met its goals from last cycle of completing Sqlalchemy 2.0 and RBAC work, while continuing to deliver a few new features and a lot of bug fixes. Most of the team are long time members which helps. One place we are working on is our job stability, and we continue to look at how to increase our review velocity. Our initial proposal on the latter is to have review marathons, perhaps on a monthly basis, since it has the double effect of closing bugs in the process. # EOL/EM cadence Train just went EOL. Decision was to propose Ussuri for EOL this cycle. # Bugs The total number of open bugs against Neutron has crept up since the beginning of Bobcat, from 726 to 760. I asked people to go through any bugs they own and make sure they are in the right state or should be closed. This is something we've done in previous cycles that has reduced our count by 1/3. # Gate We now have voting SLURP jobs! I.e. from Antelope to Caracal. Rodolfo has also been working on fullstack performance to try and reduce the runtime since we hit the 3 hour limit occasionally. Action item was taken by Slawek to see if there was overlap with other tests to see what could be removed. There was a question on what versions of OVS and OVN we use in our gate, and if there is a rule on when we should bump it, especially since sometimes new features require newer versions. Action item was taken by me to create a requirements doc similar to what Nova does for libvirt. # Neutronclient deprecation As discussed above, Horizon will be the focus this cycle. # OVN drivers for stadium projects There have been efforts recently to make stadium projects work with OVN, for example VPNaaS, FWaas and Taas. Main ask for for review cycles from core contributors. # Hidden SG rules in OVN Background is at [3]. Short story is that there are a minimum set of things that must be allowed for a VM to configure and use the network, like ARP, DHCP, ND, RA, RS and MLD. ML2/OVS adds hidden SG rules for these, and OVN allows some via ACLs. Our api-ref actually documents these requirements since stateless SGs needed to add them to function properly. The only service that is not enabled by default with stateless SGs is metadata. Question was regarding if we should make these rules visible in a read-only state. # S-RBAC This is a community goal and Neutron has completed the first phase and is part of phase 2 (service role), additional testing and nova/neutron token passing TBD. Phase 3 is the Project Manager role, which is not planned for this cycle. Thursday, October 26th ---------------------- # RFE reviews Discussed three active RFEs, two related to neutron-dynamic-routing, and one for OVN interconnection. Further discussion to take place in reviews and/or drivers meeting. # Metadata changes I noticed there were a number of changes to the metadata code in flight, so we went through them to try and create order out of the chaos. Basically, we will try and merge any bug fixes first since they need to be backported, followed by features and refactors. See the etherpad for more details. # OVN DB sync utility This tool is typically run after a migration or upgrade to make sure the Neutron and OVN DBs are in sync. A number of bugs have been fixed and backported, but there might be more. An action item was taken to try and make it more robust, and also make sure usage is documented. # Active-Active L3 gateway multi-homing Frode Nordahl gave an update on this OVN work that started in the Bobcat cycle, necessary to support multiple external gateways. A little over half the code is merged, plan is to complete the rest this cycle. # Nova cross-project Sean Mooney proposed that Neutron starts making certain extensions mandatory, for example, multiple port bindings, that make migration possible. Since then other features have been added that require it as well. Should we stop supporting backends that have not implemented it? We agreed this is a good step forward to what should be the minimum requirements of Neutron. In addition: - Nova will deprecate the legacy code path that doesn't use multiple bindings and log a warning, to be deleted in E release - Neutron will do the same for any similar code - Sean will provide a blueprint explaining which API extensions will be mandatory by Nova, as well as file a Neutron RFE bug - Eventually, we'll provide release notes and a ML thread for explaining all of this (we being the Nova and Neutron PTLs) Complete notes in Nova etherpad [4]. Thanks, and if you think I missed something please send updates or corrections. -Brian [0] https://etherpad.opendev.org/p/oct2023-ptg-neutron [1] https://etherpad.opendev.org/p/oct2023-ptg-os-dbperf-crossproject [2] https://etherpad.opendev.org/p/horizon-caracal-ptg [3] https://bugs.launchpad.net/neutron/+bug/1926515 [4] https://etherpad.opendev.org/p/nova-caracal-ptg#L80

2 2

[ops][cinder][kolla] Cinder + Ceph under high load can lead to timeouts and a failure state.
by Forrest Fuqua 14 Nov '23

14 Nov '23

In my setup of Cinder Zed + Ceph Quincy under very high loads there is a race condition due to a low timeout by default in Openstack Kolla in Cinder.conf of rados_connect_timeout = 5 This can lead to a duplicate volume trying to be made on ceph and cinder erroring out a VM Build in Nova. Nova: nova.exception.BuildAbortException: Build of instance a529f7d5-5a81-44f6-a1fd-93caa39b2439 aborted: Volume b476094b-075e-4dc7-b63e-5bb58a0a9229 did not finish being created even after we waited 30 seconds or 4 attempts. And its status is error. Cinder-Volume: File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 230, in __init__\n self.volume = driver.rbd.Image(rados_ioctx,\n', ' File "rbd.pyx", line 2894, in rbd.Image.__init__\n', "rbd.Timeout: [errno 110] RBD operation timeout (error opening image b'volume-b476094b-075e-4dc7-b63e-5bb58a0a9229' at snapshot None)\n"] --- File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute\n six.reraise(c, e, tb)\n', ' File "/var/lib/kolla/venv/lib/python3.9/site-packages/six.py", line 719, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker\n rv = meth(*args, **kwargs)\n', ' File "rbd.pyx", line 698, in rbd.RBD.clone\n', 'rbd.ImageExists: [errno 17] RBD image already exists (error creating clone)\n'] --- I can confirm that the volume does exist in Ceph now but no longer being tracked in Cinder, leading to a leftover artifact that eats up resources and a failed VM build. This seems like behavior that should be accounted for in some way.

2 2

[swift] Swiftstack Zone/DC decomission
by Muralikrishna Gutha 14 Nov '23

14 Nov '23

Certainly, here's a rewritten version: We currently utilize SwiftStack Object Storage to meet our object storage requirements, and our cluster has a Primary Zone/Secondary Zone topology. Recently, we attempted to remove/disable the Secondary Zone. However, upon doing so, we observed a concerning trend: the storage utilization in the Primary Zone rapidly increased, reaching approximately 90%. This surge was particularly notable due to the use of the 'tworeplicaperregion' policy. In an attempt to address this, we brought the Secondary Zone nodes back online, expecting Swift to redistribute the data back to these nodes and alleviate the storage strain on the Primary Zone. Despite reintegrating the Secondary Zone nodes into the cluster, we continue to witness a persistent rise in storage utilization within the Primary Zone. Interestingly, two out of the three nodes in the Primary Zone seem to bear the brunt of the data and tokens, while the third node is not receiving a proportional distribution of data. At this juncture, we are seeking guidance on potential configurations or adjustments, such as specific knobs, that could expedite the rebalance process. Our objective is to efficiently distribute the data across nodes, particularly in the Primary Zone, and optimize the overall storage utilization in our SwiftStack Object Storage cluster. Any insights or recommendations on tuning parameters for a swifter rebalance would be highly appreciated. Thanks, Murali -- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.

1 0

[tc] Next meeting Tomorrow, Tuesday Nov 14, 1800 UTC
by Jay Faulkner 14 Nov '23

14 Nov '23

Hi all, The next meeting of the OpenStack Technical Committee is tomorrow, Tuesday November 14 at 1800 UTC in #openstack-tc on OFTC. The proposed agenda is: - Roll call - Follow up on tracked action items: https://meetings.opendev.org/meetings/tc/2023/tc.2023-11-07-18.00.html - Jay to sign up to rep TC at PTG OpenInfra Live recap - Gate health check - Leaderless projects (gmann) - Implementation of Unmaintained branch statuses - draft email to openstack-discuss while we figure this out: https://etherpad.opendev.org/p/early-caracal-unmaintained-transition - 2024.1 TC Tracker - https://etherpad.opendev.org/p/tc-2024.1-tracker - Open Discussion and Reviews Thanks all, Jay Faulkner OpenStack TC Chair

1 0

[swift][OpenStackSDK][keystone] Warnings about (my) swift endpoint URL by keystoneauth1 discovery
by Christian Rohmann 14 Nov '23

14 Nov '23

Hey openstack-discuss, we use Ceph RadosGW (Quincy release) to provide Swift as part of an OpenStack cloud. The endpoints are configured as such: > # openstack endpoint list > +----------------------------------+--------+--------------+----------------+---------+-----------+----------------------------------------------------------------------+ > | ID | Region | Service Name | Service > Type | Enabled | Interface | URL | > +----------------------------------+--------+--------------+----------------+---------+-----------+----------------------------------------------------------------------+ > | 1234567890 | region | swift | object-store | True | > public | > https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s > | > | 2345678901 | region | swift | object-store | True | > internal | > https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s > | > | 3456789012 | region | swift | object-store | True | > admin | > https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s > | > so according to https://docs.ceph.com/en/latest/radosgw/keystone/#cross-project-tenant-acce…, if I am not mistaken. I can also use the openstack client to access containers: > # openstack container list > +---------------+ > | Name | > +---------------+ > | containera | > | containerb | > +---------------+ and there is no warnings or other apparent issues. But when using "project cleanup" via e.g. "openstack project cleanup --dry-run --project $PROJECT" I see multiple warnings like > Failed to contact the endpoint at > https://object-store.region.cloud.example.com/swift/v1/AUTH_f2bc4bd34567ddc… > for discovery. Fallback to using that endpoint as the base url. > but only mentioning the swift endpoint. Following the warning string this originates from the discovery method of keystoneauth1, see https://github.com/openstack/keystoneauth/blob/28048af9593740315df8d9027c3b…. Could anybody help me understand what is issue might be? Why does this warning not appear for every use of object storage / swift via openstackclient? Is there any way for me to "fix" this? Could this even be a bug? Thanks and with kind regards Christian

4 10

report an error
by 许伟祥 13 Nov '23

13 Nov '23

carry out iaas-pre-host.sh report an error Downloading packages: warning: /var/cache/yum/x86_64/7/iaas/packages/python-fixtures-3.0.0-2.el7.noarch.rpm: Header V4 RSA/SHA1 Signature, key ID 764429e6: NOKEY Public key for python-fixtures-3.0.0-2.el7.noarch.rpm is not installed Please Reboot or Reconnect the terminal How to fix it 从 Windows 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>发送

1 0

[nova] what nova filter scheduler are you using?
by Satish Patel 13 Nov '23

13 Nov '23

Folks, Just curious about what schedulers people using in their cloud for better distribution or management of VMs? And why? In entire my life I never touch that config also go with default but just curious what people doing and how?

1 0

[kolla-ansible] Galera DR across multiple datacenters
by Jahns, Jay 13 Nov '23

13 Nov '23

We currently run OpenStack on top of two datacenters, and we need to ensure we have datacenter redundancy for the database. With this, we would have 2-3 nodes in each DC running galera. Its been recommended by multiple people that we use an externally managed database server (or cluster) that has its own redundancies built around the two DC layout, and not have kolla manage that. Is there a way we can use kolla to deploy a galera cluster per DC, and use that to provide redundancy in the event that we lose one of the DCs? We were currently looking at proxysql but folks have recommended against it. Thanks.

4 4

[neutron] Bug Deputy Report November 06 - 12
by Lucas Alvares Gomes 13 Nov '23

13 Nov '23

Hi, This is the Neutron bug report from November 6th to 12th. Quieter week than usual. High: * https://bugs.launchpad.net/neutron/+bug/2042938 - "[OVN] Neutron Service Restart Disrupts Octavia OVN Load Balancer Floating IP" Unassigned * https://bugs.launchpad.net/neutron/+bug/2042941 - "neutron-{ovn,ovs}-tempest-with-sqlalchemy-master jobs not installing sqlalchemy/alembic from source" Unassigned * https://bugs.launchpad.net/neutron/+bug/2043141 - " neutron-lib unit tests need update for sqlalchemy 2.0" Assigned to: Brian Haley Medium: * https://bugs.launchpad.net/neutron/+bug/2042925 - "DNS Integration throws expection logs entries were warnings would be sufficient" Assigned to: Jayce Houtman * https://bugs.launchpad.net/neutron/+bug/2042947 - "[stable branches] nftables job inherits from openvswitch-iptables_hybrid master job" Assigned to: Yatin Cheers, Lucas

1 0

[neutron][ovs][kolla-ansible] Does neutron-openvswitch-agent required for sriov only compute node
by Satish Patel 13 Nov '23

13 Nov '23

Folks, I have an openstack cloud just running SRIOV workload, We don't have a requirement to run standard vms on compute nodes. Currently I have noticed neutron-sriov-agent and openvswitch-agent running on all compute nodes. Technically in my case I don't have any use of openvswitch-agent so I am planning to get rid of those agents to reduce pressure on the neutron-server. Just wanted to check with export if that is ok to not run ovs agent. I hope neutron-server won't freak out.

2 2

[release] Release countdown for week R-20, Nov 13-17
by Elõd Illés 11 Nov '23

11 Nov '23

Development Focus ----------------- The Caracal-1 milestone is next week, on November 16th, 2023! Project team plans for the 2024.1 Caracal cycle should now be solidified. General Information ------------------- Libraries need to be released at least once per milestone period. Next week, the release team will propose releases for any library which had changes but has not been otherwise released since the 2023.2 Bobcat release. PTL's or release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well, by posting a -1. If we do not hear anything at all by the end of the week, we will assume things are OK to proceed. NB: If one of your libraries is still releasing 0.x versions, start thinking about when it will be appropriate to do a 1.0 version. The version number does signal the state, real or perceived, of the library, so we strongly encourage going to a full major version once things are in a good and usable state. Upcoming Deadlines & Dates -------------------------- Caracal-1 milestone: November 16th, 2023 Előd Illés irc: elodilles @ #openstack-release

1 0

[OpenstackAnsible][cinder][newtron][nova]
by Tecstim 10 Nov '23

10 Nov '23

Hi, I am testing to deploy a multi-node environment. I tried openstack-helm because I thought it was cool to have containers and virtual machines in the same environment, but it is not very stable and seems difficult to operate. Now I am testing with openstack ansible. I just deployed AIO in a VM. I want to achieve this: Use a physical interface for networks with vlans, and the gateway and dhcp are managed outside openstack. Use NetApp NFS as storage for persistent data of lxd containers of the openstack-ansible, images for VMs and deplyment and use this https://docs.openstack.org/cinder/2023.2/configuration/block-storage/driver… to create volumes for vms.

1 0

[neutron] CI meeting for next week is cancelled
by Yatin Karel 10 Nov '23

10 Nov '23

Hello Neutron Team !! Next Tuesday we have a Public Holiday so I will not be available to chair the meeting. So, the next Neutron CI Meeting will be on 21st November, 2023. Thanks and Regards Yatin Karel

1 0

[all] Failing sqlalchemy-tips jobs should now be passing again
by Stephen Finucane 10 Nov '23

10 Nov '23

Just FYI, for projects that have sqlalchemy-tips jobs that have been failing recently, they should now be passing again. The root cause was neither sqlalchemy nor alembic but rather oslo.utils [1]. It was fixed in oslo.utils 6.3.0 which was released [2] roughly an hour ago. A recheck should be sufficient to clear errors and as these tips jobs only run on master there should be no issues with stable branches. A further point to add: while I appreciate that broken gates are never a good thing, I would be careful making these jobs non-voting. I've already seen a few instances where SQLAlchemy 2.0 support has regressed and I for one have no desire to re-herd cats as we push to get SQLAlchemy uncapped in the coming weeks. If this happens again then we can re-examine things, naturally. Cheers, Stephen [1] https://review.opendev.org/c/openstack/oslo.utils/+/900343 [2] https://review.opendev.org/c/openstack/releases/+/900382

2 2

[kolla-ansible] group_vars question
by Satish Patel 10 Nov '23

10 Nov '23

Folks, I have multiple kinds of nic in cloud and want to configure SRIOV so instead of defining the following variable for each compute node, I would like to create a group and apply config to the groups. Am I right here to define groups like the following or is there a better way to handle this? What is the default path of group_vars and hosts_vars in kolla-ansible? # vi /etc/kolla/group_vars/sriov_eno49.yml neutron_sriov_physnet_mappings: physnet1: eno49 # vi /etc/kolla/group_vars/sriov_eno50.yml neutron_sriov_physnet_mappings: physnet1: eno50 # vi multinode [sriov_eno49] comp1 comp2 comp3 [sriov_eno50] comp4 comp5 comp6

2 2

[Diversity] Diversity and Inclusion WG November Meeting Change and New Time
by Amy Marrich 10 Nov '23

10 Nov '23

Due to travel next week we are bumping the D&I WG meeting to November 21st at our new winter time of 15:00 UTC. As always we welcome members of all the OpenInfra projects to join us to discuss diversity and WG related topics. Amy (spotz)

1 0

[kolla-ansible] Not to install openvswitch on sriov_agent compute nodes
by Satish Patel 10 Nov '23

10 Nov '23

Folks, I have pure sriov compute nodes where I don't want to deploy openvswitch because SRIOV agents do pass through PCI bugs to VM directly. Currently all my SRIOV compute nodes have two agents running. neutron_openvswitch_agent neutron_sriov_agent I don't think "neutron_openvswitch_agent" has any role on sriov compute nodes. If I want to tell kolla-ansible not to deploy neutron_openvswitch_agent on X compute node then how do I tell. Are there any easy ways to turn it off? ~S

1 0

hi
by Tecstim 10 Nov '23

10 Nov '23

1 0

[kolla-ansible] sriov setup
by Satish Patel 09 Nov '23

09 Nov '23

Folks, I am following this guide to deploy sriov compute with kolla-ansible (I am not using kayobe) - https://www.stackhpc.com/sriov-kayobe.html In the above doc last section neutron sriov agent, I am using the same code but those variables are not reflecting. It just prints everything as it is. # cat /etc/kolla/host_vars/os1-bos-sriov-comp-001.yml neutron_sriov_physnet_mappings: physnet1: eno49 # cat /etc/kayobe/kolla/config/neutron/sriov_agent.ini {% raw %} [sriov_nic] physical_device_mappings = {% for dev, physnet in sriov_physnet_mappings.items() %}{{ (loop.index0 > 0)|ternary(',','') }}{{ physnet }}:{{ dev }}{% endfor %} exclude_devices = {% endraw %} After running deploy it just prints the entire file as its in sriov_agent.ini file instead of proper values of variable. Does kolla-ansible support the above code?

2 2

[ptls] [tc] 2023 OpenStack Deployment Data
by Allison Price 09 Nov '23

09 Nov '23

Hi everyone, Apologies for sending this during the PTG as I intended to send this prior, but below is a link to the anonymized data including responses to many project or TC contributed questions. I have removed a significant number of fields to preserve the anonymity of the organizations, but if there is a particular data point that would help your team, please let me know. Cheers, Allison https://docs.google.com/spreadsheets/d/1ehlabuqOnNK4M7xtN1GXvhLoeHn4JBaISih…

2 1