November 2023 - openstack-discuss

[swift]: Swiftstack Zone/DC decommission
by Muralikrishna Gutha 15 Nov '23

15 Nov '23

Hello Team, We currently utilize SwiftStack Object Storage to meet our object storage requirements, and our cluster has a Primary Zone/Secondary Zone topology. Recently, we attempted to remove/disable the Secondary Zone. However, upon doing so, we observed a concerning trend: the storage utilization in the Primary Zone rapidly increased, reaching approximately 90%. This surge was particularly notable due to the use of the 'tworeplicaperregion' policy. In an attempt to address this, we brought the Secondary Zone nodes back online, expecting Swift to redistribute the data back to these nodes and alleviate the storage strain on the Primary Zone. Despite reintegrating the Secondary Zone nodes into the cluster, we continue to witness a persistent rise in storage utilization within the Primary Zone. Interestingly, two out of the three nodes in the Primary Zone seem to bear the brunt of the data and tokens, while the third node is not receiving a proportional distribution of data. At this juncture, we are seeking guidance on potential configurations or adjustments, such as specific knobs, that could expedite the rebalance process. Our objective is to efficiently distribute the data across nodes, particularly in the Primary Zone, and optimize the overall storage utilization in our SwiftStack Object Storage cluster. Any insights or recommendations on tuning parameters for a swifter rebalance would be highly appreciated. Thanks, Murali -- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.

3 7

[tc] Next TC Meeting will be Nov 28 (2023-11-21 meeting cancelled)
by Jay Faulkner 15 Nov '23

15 Nov '23

Hi all, The Technical Committee will not be meeting next week. The meeting was cancelled due to American Thanksgiving. Our next meeting will be November 28, 2023. Thanks, Jay Faulkner OpenStack TC Chair

1 0

Implementation review day will be Nov 15
by Sylvain Bauza 15 Nov '23

15 Nov '23

Hey folks, We just agreed during today's meeting on having our first implementation review day this week. In order to make this day efficient, I proposed an etherpad for tracking our efforts https://etherpad.opendev.org/p/nova-caracal-status Shall you want the nova community to look at your changes, you should please add them into the etherpad in the very last section ('proposed bugfixes') and then me and a few other cores will curate that list. Tomorrow, we'll scrub the list of approved blueprints's patches and the bugfixes' ones that got approved as a review priority and we'll iterate over them. Please try to look at your Gerrit emails, you could get comments ;-) Thanks, -Sylvain

1 1

the volume cannot be deleted, it is in deleting state
by kjme001＠gmail.com 14 Nov '23

14 Nov '23

hello, I'm having trouble deleting volumes in OpenStackl from a machine that no longer exists. How to execute cinder command reset-state --attach-status detached volume_name then I execute the command: # cinder delete volume_name Delete for volume cifar-10-copy failed: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-b2d3f2e4-7855-4831-a68d-f6710cc97e94) ERROR: Unable to delete any of the specified volumes. then executes the command: cinder reset-state --state available volume_name and cinder delete volume_name Request to delete volume volume_name has been accepted. when I check: openstack volume show volume_name is: | status | deleting The volume is not being deleted, it is in the deleting state. This is a volume from a non-existent node, but the volume is still visible in OpenStack and cannot be deleted because it is in the deleting state. Maybe someone can tell me how to remove such a volume? Regards

1 0

cinder-manage service remove cinder-volume xxxx@lvm-ssd - error
by kjme001＠gmail.com 14 Nov '23

14 Nov '23

cinder-manage service remove cinder-volume xxxx@lvm-ssd /usr/lib/python3/dist-packages/cinder/db/sqlalchemy/models.py:152: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery. last_heartbeat = column_property( /usr/lib/python3/dist-packages/cinder/db/sqlalchemy/models.py:160: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery. num_hosts = column_property( /usr/lib/python3/dist-packages/cinder/db/sqlalchemy/models.py:169: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery. num_down_hosts = column_property( 2023-11-11 14:34:00.928 898920 CRITICAL cinder [req-525973bb-e026-4999-95af-d852aa137066 - - - - -] Unhandled error: oslo_db.exception.DBNonExistentDatabase: (sqlite3.OperationalError) unable to open database file (Background on this error at: https://sqlalche.me/e/14/e3q8) 2023-11-11 14:34:00.928 898920 ERROR cinder Traceback (most recent call last): 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 3250, in _wrap_pool_connect 2023-11-11 14:34:00.928 898920 ERROR cinder return fn() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 310, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return _ConnectionFairy._checkout(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 868, in _checkout 2023-11-11 14:34:00.928 898920 ERROR cinder fairy = _ConnectionRecord.checkout(pool) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 476, in checkout 2023-11-11 14:34:00.928 898920 ERROR cinder rec = pool._do_get() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 256, in _do_get 2023-11-11 14:34:00.928 898920 ERROR cinder return self._create_connection() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 256, in _create_connection 2023-11-11 14:34:00.928 898920 ERROR cinder return _ConnectionRecord(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 371, in __init__ 2023-11-11 14:34:00.928 898920 ERROR cinder self.__connect() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 665, in __connect 2023-11-11 14:34:00.928 898920 ERROR cinder with util.safe_reraise(): 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ 2023-11-11 14:34:00.928 898920 ERROR cinder compat.raise_( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_ 2023-11-11 14:34:00.928 898920 ERROR cinder raise exception 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 661, in __connect 2023-11-11 14:34:00.928 898920 ERROR cinder self.dbapi_connection = connection = pool._invoke_creator(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/create.py", line 590, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return dialect.connect(*cargs, **cparams) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 597, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return self.dbapi.connect(*cargs, **cparams) 2023-11-11 14:34:00.928 898920 ERROR cinder sqlite3.OperationalError: unable to open database file 2023-11-11 14:34:00.928 898920 ERROR cinder 2023-11-11 14:34:00.928 898920 ERROR cinder The above exception was the direct cause of the following exception: 2023-11-11 14:34:00.928 898920 ERROR cinder 2023-11-11 14:34:00.928 898920 ERROR cinder Traceback (most recent call last): 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/bin/cinder-manage", line 10, in <module> 2023-11-11 14:34:00.928 898920 ERROR cinder sys.exit(main()) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/cmd/manage.py", line 1069, in main 2023-11-11 14:34:00.928 898920 ERROR cinder fn(**fn_kwargs) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/cmd/manage.py", line 711, in remove 2023-11-11 14:34:00.928 898920 ERROR cinder svc = objects.Service.get_by_args(ctxt, host_name, binary) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/objects/service.py", line 133, in get_by_args 2023-11-11 14:34:00.928 898920 ERROR cinder db_service = db.service_get(context, host=host, binary=binary_key) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/api.py", line 270, in service_get 2023-11-11 14:34:00.928 898920 ERROR cinder return IMPL.service_get(context, service_id, backend_match_level, 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 168, in wrapper 2023-11-11 14:34:00.928 898920 ERROR cinder return f(*args, **kwargs) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 745, in service_get 2023-11-11 14:34:00.928 898920 ERROR cinder query = _service_query(context, backend_match_level=backend_match_level, 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 686, in _service_query 2023-11-11 14:34:00.928 898920 ERROR cinder query = model_query(context, models.Service, session=session, 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 259, in model_query 2023-11-11 14:34:00.928 898920 ERROR cinder session = kwargs.get('session') or get_session() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 95, in get_session 2023-11-11 14:34:00.928 898920 ERROR cinder return main_context_manager._factory.get_legacy_facade().get_session( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 357, in get_legacy_facade 2023-11-11 14:34:00.928 898920 ERROR cinder self._start() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 504, in _start 2023-11-11 14:34:00.928 898920 ERROR cinder self._setup_for_connection( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 529, in _setup_for_connection 2023-11-11 14:34:00.928 898920 ERROR cinder engine = engines.create_engine( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator 2023-11-11 14:34:00.928 898920 ERROR cinder return wrapped(*args, **kwargs) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 211, in create_engine 2023-11-11 14:34:00.928 898920 ERROR cinder test_conn = _test_connection(engine, max_retries, retry_interval) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 386, in _test_connection 2023-11-11 14:34:00.928 898920 ERROR cinder return engine.connect() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 3204, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return self._connection_cls(self, close_with_result=close_with_result) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 96, in __init__ 2023-11-11 14:34:00.928 898920 ERROR cinder else engine.raw_connection() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 3283, in raw_connection 2023-11-11 14:34:00.928 898920 ERROR cinder return self._wrap_pool_connect(self.pool.connect, _connection) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 3253, in _wrap_pool_connect 2023-11-11 14:34:00.928 898920 ERROR cinder Connection._handle_dbapi_exception_noconnection( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2098, in _handle_dbapi_exception_noconnection 2023-11-11 14:34:00.928 898920 ERROR cinder util.raise_(newraise, with_traceback=exc_info[2], from_=e) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_ 2023-11-11 14:34:00.928 898920 ERROR cinder raise exception 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 3250, in _wrap_pool_connect 2023-11-11 14:34:00.928 898920 ERROR cinder return fn() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 310, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return _ConnectionFairy._checkout(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 868, in _checkout 2023-11-11 14:34:00.928 898920 ERROR cinder fairy = _ConnectionRecord.checkout(pool) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 476, in checkout 2023-11-11 14:34:00.928 898920 ERROR cinder rec = pool._do_get() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 256, in _do_get 2023-11-11 14:34:00.928 898920 ERROR cinder return self._create_connection() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 256, in _create_connection 2023-11-11 14:34:00.928 898920 ERROR cinder return _ConnectionRecord(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 371, in __init__ 2023-11-11 14:34:00.928 898920 ERROR cinder self.__connect() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 665, in __connect 2023-11-11 14:34:00.928 898920 ERROR cinder with util.safe_reraise(): 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ 2023-11-11 14:34:00.928 898920 ERROR cinder compat.raise_( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_ 2023-11-11 14:34:00.928 898920 ERROR cinder raise exception 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 661, in __connect 2023-11-11 14:34:00.928 898920 ERROR cinder self.dbapi_connection = connection = pool._invoke_creator(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/create.py", line 590, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return dialect.connect(*cargs, **cparams) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 597, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return self.dbapi.connect(*cargs, **cparams) 2023-11-11 14:34:00.928 898920 ERROR cinder oslo_db.exception.DBNonExistentDatabase: (sqlite3.OperationalError) unable to open database file 2023-11-11 14:34:00.928 898920 ERROR cinder (Background on this error at: https://sqlalche.me/e/14/e3q8) 2023-11-11 14:34:00.928 898920 ERROR cinder

3 4

(no subject)
by mgutha＠liveperson.com 14 Nov '23

14 Nov '23

I have observed an imbalance in the cluster based on the object.builder output. Is this possibly due to an ongoing rebalance process? Additionally, I'd like to confirm my understanding of 'min_part_hours.' Is it accurate to say that this parameter represents the number of hours an object needs to reside on a disk before it's considered for movement to another device during a rebalance? According to the Swift documentation, the default value is 24 hours. I'm curious if this value is applied to data written specifically by the rebalance process or if it also includes data written by other Swift services. object.builder, build version 775, id 61b2166cb8684b2b80d6fc370cef37a6 65536 partitions, 2.000000 replicas, 2 regions, 2 zones, 95 devices, 38.89 balance, 0.00 dispersion The minimum number of hours before a partition can be reassigned is 0 (0:00:00 remaining) The overload factor is 10.00% (0.100000) Ring file object.ring.gz not found, probably it hasn't been written yet Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta 129 1 1 182.16.128.36:6000 182.16.128.36:6003 d129 1.00 2 38.89 130 1 1 182.16.128.36:6006 182.16.128.36:6003 d130 1.00 2 38.89 131 1 1 182.16.128.36:6007 182.16.128.36:6003 d131 1.00 2 38.89 132 1 1 182.16.128.36:6008 182.16.128.36:6003 d132 1.00 2 38.89 133 1 1 182.16.128.36:6009 182.16.128.36:6003 d133 1.00 2 38.89 134 1 1 182.16.128.36:6010 182.16.128.36:6003 d134 1.00 1 -30.56 135 1 1 182.16.128.36:6011 182.16.128.36:6003 d135 1.00 1 -30.56 136 1 1 182.16.128.36:6012 182.16.128.36:6003 d136 1.00 1 -30.56 137 1 1 182.16.128.36:6013 182.16.128.36:6003 d137 1.00 1 -30.56 138 1 1 182.16.128.36:6014 182.16.128.36:6003 d138 1.00 1 -30.56 139 1 1 182.16.128.36:6015 182.16.128.36:6003 d139 1.00 1 -30.56 140 1 1 182.16.128.38:6000 182.16.128.38:6003 d140 1.00 2 38.89 141 1 1 182.16.128.38:6006 182.16.128.38:6003 d141 1.00 2 38.89 142 1 1 182.16.128.38:6007 182.16.128.38:6003 d142 1.00 2 38.89 143 1 1 182.16.128.38:6008 182.16.128.38:6003 d143 1.00 2 38.89 144 1 1 182.16.128.38:6009 182.16.128.38:6003 d144 1.00 2 38.89 145 1 1 182.16.128.38:6010 182.16.128.38:6003 d145 1.00 1 -30.56 146 1 1 182.16.128.38:6011 182.16.128.38:6003 d146 1.00 1 -30.56 147 1 1 182.16.128.38:6012 182.16.128.38:6003 d147 1.00 1 -30.56 148 1 1 182.16.128.38:6013 182.16.128.38:6003 d148 1.00 1 -30.56 149 1 1 182.16.128.38:6014 182.16.128.38:6003 d149 1.00 1 -30.56 150 1 1 182.16.128.38:6015 182.16.128.38:6003 d150 1.00 1 -30.56 151 1 1 182.16.128.39:6000 182.16.128.39:6003 d151 1.00 2 38.89 152 1 1 182.16.128.39:6006 182.16.128.39:6003 d152 1.00 2 38.89 153 1 1 182.16.128.39:6007 182.16.128.39:6003 d153 1.00 2 38.89 154 1 1 182.16.128.39:6008 182.16.128.39:6003 d154 1.00 2 38.89 155 1 1 182.16.128.39:6009 182.16.128.39:6003 d155 1.00 2 38.89 156 1 1 182.16.128.39:6010 182.16.128.39:6003 d156 1.00 1 -30.56 157 1 1 182.16.128.39:6011 182.16.128.39:6003 d157 1.00 1 -30.56 158 1 1 182.16.128.39:6012 182.16.128.39:6003 d158 1.00 1 -30.56 159 1 1 182.16.128.39:6013 182.16.128.39:6003 d159 1.00 1 -30.56 160 1 1 182.16.128.39:6014 182.16.128.39:6003 d160 1.00 1 -30.56 161 1 1 182.16.128.39:6015 182.16.128.39:6003 d161 1.00 1 -30.56 55 1 1 182.16.128.5:6000 182.16.128.5:6003 d55 2000.40 2789 -3.18 56 1 1 182.16.128.5:6006 182.16.128.5:6003 d56 2000.40 2789 -3.18 57 1 1 182.16.128.5:6007 182.16.128.5:6003 d57 2000.40 2789 -3.18 58 1 1 182.16.128.5:6008 182.16.128.5:6003 d58 2000.40 2789 -3.18 59 1 1 182.16.128.5:6009 182.16.128.5:6003 d59 2000.40 2789 -3.18 60 1 1 182.16.128.5:6010 182.16.128.5:6003 d60 2000.40 2789 -3.18 61 1 1 182.16.128.5:6011 182.16.128.5:6003 d61 2000.40 2789 -3.18 62 1 1 182.16.128.5:6012 182.16.128.5:6003 d62 2000.40 2789 -3.18 63 1 1 182.16.128.5:6013 182.16.128.5:6003 d63 2000.40 2789 -3.18 64 1 1 182.16.128.5:6014 182.16.128.5:6003 d64 2000.40 2789 -3.18 65 1 1 182.16.128.5:6015 182.16.128.5:6003 d65 2000.40 2788 -3.21 44 1 1 182.16.128.6:6000 182.16.128.6:6003 d44 269.43 376 -3.09 45 1 1 182.16.128.6:6006 182.16.128.6:6003 d45 269.43 376 -3.09 46 1 1 182.16.128.6:6007 182.16.128.6:6003 d46 269.43 376 -3.09 47 1 1 182.16.128.6:6008 182.16.128.6:6003 d47 269.43 376 -3.09 48 1 1 182.16.128.6:6009 182.16.128.6:6003 d48 269.43 376 -3.09 49 1 1 182.16.128.6:6010 182.16.128.6:6003 d49 269.43 376 -3.09 50 1 1 182.16.128.6:6011 182.16.128.6:6003 d50 269.43 376 -3.09 51 1 1 182.16.128.6:6012 182.16.128.6:6003 d51 269.43 375 -3.35 52 1 1 182.16.128.6:6013 182.16.128.6:6003 d52 269.43 375 -3.35 53 1 1 182.16.128.6:6014 182.16.128.6:6003 d53 269.43 375 -3.35 54 1 1 182.16.128.6:6015 182.16.128.6:6003 d54 269.43 375 -3.35 0 1 1 182.16.128.7:6000 182.16.128.7:6003 d0 2000.40 2789 -3.18 1 1 1 182.16.128.7:6006 182.16.128.7:6003 d1 2000.40 2789 -3.18 10 1 1 182.16.128.7:6015 182.16.128.7:6003 d10 2000.40 2788 -3.21 2 1 1 182.16.128.7:6007 182.16.128.7:6003 d2 2000.40 2789 -3.18 3 1 1 182.16.128.7:6008 182.16.128.7:6003 d3 2000.40 2789 -3.18 4 1 1 182.16.128.7:6009 182.16.128.7:6003 d4 2000.40 2789 -3.18 5 1 1 182.16.128.7:6010 182.16.128.7:6003 d5 2000.40 2789 -3.18 6 1 1 182.16.128.7:6011 182.16.128.7:6003 d6 2000.40 2789 -3.18 7 1 1 182.16.128.7:6012 182.16.128.7:6003 d7 2000.40 2789 -3.18 8 1 1 182.16.128.7:6013 182.16.128.7:6003 d8 2000.40 2789 -3.18 9 1 1 182.16.128.7:6014 182.16.128.7:6003 d9 2000.40 2789 -3.18 11 2 2 11.17.128.5:6000 11.17.128.5:6003 d11 2000.40 2979 3.42 12 2 2 11.17.128.5:6006 11.17.128.5:6003 d12 2000.40 2979 3.42 13 2 2 11.17.128.5:6007 11.17.128.5:6003 d13 2000.40 2979 3.42 15 2 2 11.17.128.5:6008 11.17.128.5:6003 d15 2000.40 2979 3.42 16 2 2 11.17.128.5:6009 11.17.128.5:6003 d16 2000.40 2979 3.42 17 2 2 11.17.128.5:6010 11.17.128.5:6003 d17 2000.40 2978 3.38 18 2 2 11.17.128.5:6011 11.17.128.5:6003 d18 2000.40 2978 3.38 19 2 2 11.17.128.5:6012 11.17.128.5:6003 d19 2000.40 2978 3.38 20 2 2 11.17.128.5:6013 11.17.128.5:6003 d20 2000.40 2978 3.38 21 2 2 11.17.128.5:6014 11.17.128.5:6003 d21 2000.40 2978 3.38 77 2 2 11.17.128.5:6015 11.17.128.5:6003 d77 2000.40 2978 3.38 33 2 2 11.17.128.6:6000 11.17.128.6:6003 d33 2000.40 2979 3.42 34 2 2 11.17.128.6:6006 11.17.128.6:6003 d34 2000.40 2979 3.42 35 2 2 11.17.128.6:6007 11.17.128.6:6003 d35 2000.40 2979 3.42 36 2 2 11.17.128.6:6008 11.17.128.6:6003 d36 2000.40 2979 3.42 37 2 2 11.17.128.6:6009 11.17.128.6:6003 d37 2000.40 2978 3.38 38 2 2 11.17.128.6:6010 11.17.128.6:6003 d38 2000.40 2978 3.38 39 2 2 11.17.128.6:6011 11.17.128.6:6003 d39 2000.40 2978 3.38 40 2 2 11.17.128.6:6012 11.17.128.6:6003 d40 2000.40 2978 3.38 41 2 2 11.17.128.6:6013 11.17.128.6:6003 d41 2000.40 2978 3.38 42 2 2 11.17.128.6:6014 11.17.128.6:6003 d42 2000.40 2978 3.38 43 2 2 11.17.128.6:6015 11.17.128.6:6003 d43 2000.40 2978 3.38 162 2 2 11.17.128.71:6000 11.17.128.71:6003 d162 1.00 2 38.89 163 2 2 11.17.128.71:6006 11.17.128.71:6003 d163 1.00 2 38.89 164 2 2 11.17.128.71:6007 11.17.128.71:6003 d164 1.00 2 38.89 165 2 2 11.17.128.71:6008 11.17.128.71:6003 d165 1.00 2 38.89 166 2 2 11.17.128.71:6009 11.17.128.71:6003 d166 1.00 1 -30.56 167 2 2 11.17.128.71:6010 11.17.128.71:6003 d167 1.00 1 -30.56 168 2 2 11.17.128.71:6011 11.17.128.71:6003 d168 1.00 1 -30.56

3 2

Re: [swift]: Swiftstack Zone/DC decommission
by Muralikrishna Gutha 14 Nov '23

14 Nov '23

Clary, Thanks for the update I was unable to locate the object.builder file in the instance where we are running swiftstack. We are currently on Swift Package Version: 2.24.0.3-1.el7 [root@hostname swift]# /opt/ss/bin/swift-ring-builder object.builder | head Ring Builder file does not exist: object.builder [root@hostname swift]# locate object.builder [root@hostname swift]#pwd /etc/swift Thanks, Murali -- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.

2 1

[PTL][release][stable] Yoga - final release before branch becomes unmaintained
by Elõd Illés 14 Nov '23

14 Nov '23

Hi teams, According to the new process [1] Yoga should transition to Unmaintained as soon as possible (the original deadline for Extended Maintenance transition was planned to 2023-11-02). Similarly like before, teams have the final opportunity to release from stable/yoga before the branches will be *deleted*. In order to help the teams I have generated the list of the current *open* and *unreleased* changes in stable/yoga for repositories [2] where there are relevant patches. Please do a *final* release on Yoga if needed, before stable/yoga transition to Unmaintained. Feel free to edit and extend these lists to track your team's progress! *NOTE*: teams, please focus on wrapping up your libraries first if there is any concern about the changes, in order to avoid broken (final!!) releases! [1] https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branc… [2] https://etherpad.opendev.org/p/yoga-final-release-before-unmaintained [3] https://review.opendev.org/q/topic:xena-em Thanks, Előd irc: elodilles @ #openstack-release

2 1

[neutron][ptg] 2024.1/Caracal vPTG summary
by Brian Haley 14 Nov '23

14 Nov '23

Sorry for the slowness of getting this out. Here is a summary of the discussions we had over our two days of meetings, as well as other related sessions. The etherpad link can be found at [0]. Monday, October 23rd -------------------- # OS DBPerf Cross project session [1] This session was to talk about devstack performance issues that are believed to be caused by excessive querying by Neutron and Keystone. By default, when devstack "stacks", DB counter values are collected on each of the subsystems. Keystone and Neutron are outliers on how many SELECTs they issue during this process. As this is such a high-level number (4x to 10x other users), more work is required to understand what to fix, or if it's just normal behavior. For example, there is data from just a "stack" in addition to a tempest run that might be able to narrow things down here. Slawek and Miguel have agreed to work with the oslo.db team and Mike Bayer to get a better understanding of things here. Tuesday, October 24th --------------------- # Horizon For the last few cycles the Neutron team (Lajos) has been trying to deprecate usage of neutronclient. While this work is mostly complete, there is usage in Horizon that needs to be addressed. Lajos and Amotoki will be working on this. Full notes at [2] Wednesday, October 25th ----------------------- # Retrospective As is customary the team did a retrospective of the good, bad and what could be improved. In short, the Neutron team has met its goals from last cycle of completing Sqlalchemy 2.0 and RBAC work, while continuing to deliver a few new features and a lot of bug fixes. Most of the team are long time members which helps. One place we are working on is our job stability, and we continue to look at how to increase our review velocity. Our initial proposal on the latter is to have review marathons, perhaps on a monthly basis, since it has the double effect of closing bugs in the process. # EOL/EM cadence Train just went EOL. Decision was to propose Ussuri for EOL this cycle. # Bugs The total number of open bugs against Neutron has crept up since the beginning of Bobcat, from 726 to 760. I asked people to go through any bugs they own and make sure they are in the right state or should be closed. This is something we've done in previous cycles that has reduced our count by 1/3. # Gate We now have voting SLURP jobs! I.e. from Antelope to Caracal. Rodolfo has also been working on fullstack performance to try and reduce the runtime since we hit the 3 hour limit occasionally. Action item was taken by Slawek to see if there was overlap with other tests to see what could be removed. There was a question on what versions of OVS and OVN we use in our gate, and if there is a rule on when we should bump it, especially since sometimes new features require newer versions. Action item was taken by me to create a requirements doc similar to what Nova does for libvirt. # Neutronclient deprecation As discussed above, Horizon will be the focus this cycle. # OVN drivers for stadium projects There have been efforts recently to make stadium projects work with OVN, for example VPNaaS, FWaas and Taas. Main ask for for review cycles from core contributors. # Hidden SG rules in OVN Background is at [3]. Short story is that there are a minimum set of things that must be allowed for a VM to configure and use the network, like ARP, DHCP, ND, RA, RS and MLD. ML2/OVS adds hidden SG rules for these, and OVN allows some via ACLs. Our api-ref actually documents these requirements since stateless SGs needed to add them to function properly. The only service that is not enabled by default with stateless SGs is metadata. Question was regarding if we should make these rules visible in a read-only state. # S-RBAC This is a community goal and Neutron has completed the first phase and is part of phase 2 (service role), additional testing and nova/neutron token passing TBD. Phase 3 is the Project Manager role, which is not planned for this cycle. Thursday, October 26th ---------------------- # RFE reviews Discussed three active RFEs, two related to neutron-dynamic-routing, and one for OVN interconnection. Further discussion to take place in reviews and/or drivers meeting. # Metadata changes I noticed there were a number of changes to the metadata code in flight, so we went through them to try and create order out of the chaos. Basically, we will try and merge any bug fixes first since they need to be backported, followed by features and refactors. See the etherpad for more details. # OVN DB sync utility This tool is typically run after a migration or upgrade to make sure the Neutron and OVN DBs are in sync. A number of bugs have been fixed and backported, but there might be more. An action item was taken to try and make it more robust, and also make sure usage is documented. # Active-Active L3 gateway multi-homing Frode Nordahl gave an update on this OVN work that started in the Bobcat cycle, necessary to support multiple external gateways. A little over half the code is merged, plan is to complete the rest this cycle. # Nova cross-project Sean Mooney proposed that Neutron starts making certain extensions mandatory, for example, multiple port bindings, that make migration possible. Since then other features have been added that require it as well. Should we stop supporting backends that have not implemented it? We agreed this is a good step forward to what should be the minimum requirements of Neutron. In addition: - Nova will deprecate the legacy code path that doesn't use multiple bindings and log a warning, to be deleted in E release - Neutron will do the same for any similar code - Sean will provide a blueprint explaining which API extensions will be mandatory by Nova, as well as file a Neutron RFE bug - Eventually, we'll provide release notes and a ML thread for explaining all of this (we being the Nova and Neutron PTLs) Complete notes in Nova etherpad [4]. Thanks, and if you think I missed something please send updates or corrections. -Brian [0] https://etherpad.opendev.org/p/oct2023-ptg-neutron [1] https://etherpad.opendev.org/p/oct2023-ptg-os-dbperf-crossproject [2] https://etherpad.opendev.org/p/horizon-caracal-ptg [3] https://bugs.launchpad.net/neutron/+bug/1926515 [4] https://etherpad.opendev.org/p/nova-caracal-ptg#L80

2 2

[ops][cinder][kolla] Cinder + Ceph under high load can lead to timeouts and a failure state.
by Forrest Fuqua 13 Nov '23

13 Nov '23

In my setup of Cinder Zed + Ceph Quincy under very high loads there is a race condition due to a low timeout by default in Openstack Kolla in Cinder.conf of rados_connect_timeout = 5 This can lead to a duplicate volume trying to be made on ceph and cinder erroring out a VM Build in Nova. Nova: nova.exception.BuildAbortException: Build of instance a529f7d5-5a81-44f6-a1fd-93caa39b2439 aborted: Volume b476094b-075e-4dc7-b63e-5bb58a0a9229 did not finish being created even after we waited 30 seconds or 4 attempts. And its status is error. Cinder-Volume: File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 230, in __init__\n self.volume = driver.rbd.Image(rados_ioctx,\n', ' File "rbd.pyx", line 2894, in rbd.Image.__init__\n', "rbd.Timeout: [errno 110] RBD operation timeout (error opening image b'volume-b476094b-075e-4dc7-b63e-5bb58a0a9229' at snapshot None)\n"] --- File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute\n six.reraise(c, e, tb)\n', ' File "/var/lib/kolla/venv/lib/python3.9/site-packages/six.py", line 719, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker\n rv = meth(*args, **kwargs)\n', ' File "rbd.pyx", line 698, in rbd.RBD.clone\n', 'rbd.ImageExists: [errno 17] RBD image already exists (error creating clone)\n'] --- I can confirm that the volume does exist in Ceph now but no longer being tracked in Cinder, leading to a leftover artifact that eats up resources and a failed VM build. This seems like behavior that should be accounted for in some way.

2 2