- openstack-discuss - lists.openstack.org

[Glance] Cancelling Weekly Meeting - 16th Nov
by Mridula Joshi 16 Nov '23

16 Nov '23

Hello All, Cancelling today's weekly meeting as most of the team members are not around for the meeting. See you next week ! Thanks, Mridula Joshi

1 0

[cinder] Festival of XS reviews (17 Nov)
by Rajat Dhasmana 16 Nov '23

16 Nov '23

Hello Argonauts, We will be having our monthly festival of reviews tomorrow i.e. 17th November (Friday) from 1400-1600 UTC. Following are the details: Date: 17th November, 2023 Time: 1400-1600 UTC Meeting link: https://meet.google.com/jqg-eigw-rku etherpad: https://etherpad.opendev.org/p/cinder-festival-of-reviews Thanks Rajat Dhasmana

1 0

openstack: aborted: Failed to allocate the network(s), not rescheduling.].
by kjme001＠gmail.com 16 Nov '23

16 Nov '23

when creating a new instance I get an error: aborted: Failed to allocate the network(s), not rescheduling.]. Can anyone tell me where to look for the problem? Regards

3 2

RE: openstack-discuss Digest, Vol 61, Issue 61- Nova- Compute Service Issue
by Asma Naz Shariq 15 Nov '23

15 Nov '23

Hello Openstack Team, I have deployed openstack zed release with multinode cluster through kolla-ansible controller with 1x storage, 1x compute node. The nova-compute service failed to register itself on the compute-host as attached; due to which there encounters an error of no availability zone for an instance to launch. Can any one guide me how to resolve this nova service issues. -----Original Message----- From: openstack-discuss-request(a)lists.openstack.org <openstack-discuss-request(a)lists.openstack.org> Sent: Thursday, November 16, 2023 2:15 AM To: openstack-discuss(a)lists.openstack.org Subject: openstack-discuss Digest, Vol 61, Issue 61 Send openstack-discuss mailing list submissions to openstack-discuss(a)lists.openstack.org To subscribe or unsubscribe via email, send a message with subject or body 'help' to openstack-discuss-request(a)lists.openstack.org You can reach the person managing the list at openstack-discuss-owner(a)lists.openstack.org When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. Re: [kolla-ansible][cinder-schedule] Problem between Ceph and cinder-schedule (Franck VEDEL) 2. Re: Re: [keystone] temporary workaround for deprecated _member_ role? (Michael Knox) 3. openstack: aborted: Failed to allocate the network(s), not rescheduling.]. (kjme001(a)gmail.com) ---------------------------------------------------------------------- Message: 1 Date: Wed, 15 Nov 2023 21:33:15 +0100 From: Franck VEDEL <franck.vedel(a)univ-grenoble-alpes.fr> Subject: Re: [kolla-ansible][cinder-schedule] Problem between Ceph and cinder-schedule To: Alan Bishop <abishop(a)redhat.com> Cc: openstack-discuss(a)lists.openstack.org Message-ID: <63B87539-94A5-4CA5-A6CA-9CB67124EAEF(a)univ-grenoble-alpes.fr> Content-Type: multipart/alternative; boundary="Apple-Mail=_880E21AE-FC09-43C2-8C33-2E2670B2D035" Alan ! Thanks a lot…. I had doubts about the last config because I only wanted to use the Ceph cluster, but cinder-scheduler was looking for a backend which it could not find. But the cluster was functional so: kolla-ansible -i multinode stop …..destroy…. deploy…. post-deploy…. init-runonce…. Et VOILA. This reset all configurations, adds and rollbacks to 0. And it works ! Thanks!! Franck > Le 15 nov. 2023 à 15:43, Alan Bishop <abishop(a)redhat.com> a écrit : > > > > On Wed, Nov 15, 2023 at 6:32 AM Franck VEDEL <franck.vedel(a)univ-grenoble-alpes.fr <mailto:franck.vedel@univ-grenoble-alpes.fr>> wrote: >> Hi Pierre >> Maybe I found something: >> In the container « cinder_volume », there is a file >> /etc/ceph/ceph.conf But, no /etc/ceph/ceph.conf file in « cinder_scheduler » container. >> I think it’s my problem !! > > Sorry, but the scheduler does not require access to /etc/ceph files. I > think you need to review the scheduler logs to see why it concluded > there are no available c-vol backends. Since you are able to create a volume, I assume the ceph backend is "up," but it would also be good to verify it remains up, especially during the time when you're trying to create a volume from an image. > > Alan > >> but why ? >> What is the option in globals.yml ? >> Or maybe there is something to add in /etc/kolla/config/cinder/ >> >> I have this (the last try): >> >> >> enable_cinder: "yes" >> enable_cinder_backup: "no" >> enable_cinder_backend_lvm: "no" >> external_ceph_cephx_enabled: "yes" >> >> # Glance >> ceph_glance_keyring: "ceph.client.glance.keyring" >> ceph_glance_user: "glance" >> ceph_glance_pool_name: "images" >> # Cinder >> ceph_cinder_keyring: "ceph.client.cinder.keyring" >> ceph_cinder_user: "cinder" >> ceph_cinder_pool_name: "volumes" >> >> #ceph_cinder_backup_keyring: "ceph.client.cinder-backup.keyring" >> #ceph_cinder_backup_keyring: "ceph.client.cinder.keyring" >> #ceph_cinder_backup_user: "cinder" >> #ceph_cinder_backup_user: "cinder-backup" >> #ceph_cinder_backup_pool_name: "backups" >> # Nova >> #ceph_nova_keyring: "{{ ceph_cinder_keyring }}" >> ceph_nova_keyring: "ceph.client.nova.keyring" >> ceph_nova_user: "nova" >> ceph_nova_pool_name: "vms" >> >> # Configure image backend. >> glance_backend_ceph: "yes" >> >> # Enable / disable Cinder backends >> cinder_backend_ceph: "yes" >> >> # Nova - Compute Options >> ######################## >> nova_backend_ceph: "yes" >> >> >> Thanks a lot >> >> >> >> Franck >> >> >>> Le 15 nov. 2023 à 09:54, Pierre Riteau <pierre(a)stackhpc.com <mailto:pierre@stackhpc.com>> a écrit : >>> >>> Hi Franck, >>> >>> It would help if you could share more details about the error (check both cinder-scheduler and cinder-volume logs). >>> >>> Do you have capabilities to access the images pool (presumably what is used by Glance) on your client.cinder user? >>> >>> Best wishes, >>> Pierre Riteau (priteau) >>> >>> On Wed, 15 Nov 2023 at 09:50, Franck VEDEL <franck.vedel(a)univ-grenoble-alpes.fr <mailto:franck.vedel@univ-grenoble-alpes.fr>> wrote: >>>> Good morning, >>>> I'll come back for help, thanks in advance. >>>> I am testing an Openstack 2023.1/Ceph Pacific environment. >>>> The cluster works, and from Horizon for example, if I create a simple volume, it works normally, it is placed in the "volumes" pool in the Ceph cluster, exactly as expected. >>>> >>>> if I create a volume from an image, ERROR cinder-schedule. >>>> In the cinder_volume and cinder_scheduler containers, same settings in cinder.conf. >>>> >>>> Before continuing to find out what the error could be, could there be a problem between Antelope (2023.1) and Ceph Pacific? (I'm on Ubuntu 22.04). >>>> >>>> Franck >>>> >>>> >>>> >>

1 0

[kolla-ansible][cinder-schedule] Problem between Ceph and cinder-schedule
by Franck VEDEL 15 Nov '23

15 Nov '23

Good morning, I'll come back for help, thanks in advance. I am testing an Openstack 2023.1/Ceph Pacific environment. The cluster works, and from Horizon for example, if I create a simple volume, it works normally, it is placed in the "volumes" pool in the Ceph cluster, exactly as expected. if I create a volume from an image, ERROR cinder-schedule. In the cinder_volume and cinder_scheduler containers, same settings in cinder.conf. Before continuing to find out what the error could be, could there be a problem between Antelope (2023.1) and Ceph Pacific? (I'm on Ubuntu 22.04). Franck

4 8

[cinder][ops] Deprecating vmdk volume driver
by Takashi Kajinami 15 Nov '23

15 Nov '23

Hello, # This is similar to my previous email[1] but this one is for a different component. Cinder has provided the following two volume drivers to manage volumes in VMWare product. VMwareVcVmdkDriver for VMWare vCenter server VMwareVStorageObjectDriver for VMWare VStorageObject These drivers may be useful for OpenStack deployments using VMWare hypervisors. However the vmware api virt driver is being removed during this cycle, and this use case is no longer supported. I'm sending this email to hear if anyone is using these volume drivers WITHOUT the vmware api virt driver. If no, then I'll propose deprecating these drivers in 2024.1 release, and remove these in 2024.2. In case you have any concerns (or agree with the plan) then please reply to this thread (or to me directly if needed). Thank you, Takashi [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…

1 0

Cinder Bug Report 2023-11-15
by Jon Bernard 15 Nov '23

15 Nov '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> There are exactly 0 new bugs this week, so no bug meeting today! Thanks, -- Jon

1 0

[largescale-sig] Next meeting: Nov 15, 8utc
by Thierry Carrez 15 Nov '23

15 Nov '23

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 8UTC, our EU+APAC-friendly time. Note that in some parts of the world, DST means that that meeting will be one hour earlier than usual. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20231115T08 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Thanks, -- Thierry Carrez (ttx)

1 1

[tc][releases][ptl][all] 'Unmaintained' branch update
by Brian Rosmaita 14 Nov '23

14 Nov '23

Hello Stackers, I'm sure you saw Elod's email eariler this week [0] calling for final releases from stable/yoga in preparation for its transition to 'Unmaintained' status. In addition to stable/yoga, the TC resolution [1] specified that the three most recent Extended Maintenance branches would also transition to 'Unmaintained', and branches older than that can be deleted. The TC and the release team have had some discussions over the past two weeks [2,3,4,5], and we are closing in on a consensus on how the proposal described in the 2023-07024 resolution should work. This is how it's looking: (1) branches ineligible for Unmaintained status These are stable/ussuri and stable/train. The release team will use the current tooling to generate the patches proposing the ussuri-eol and train-eol tags; teams will have a grace period to get any final changes merged; after the -eol tags are made, the branches will be deleted. (2) transitioning a branch out of Maintained status This is stable/yoga. Elod has already sent the reminder email [0] for project teams to propose a release patch if there's a reason to do a final release. The release team is working on tooling that will mark the HEAD of stable/yoga with yoga-eom (for "end of maintenance"), delete the stable/yoga branch, and then cut the unmaintained/yoga branch based on the yoga-eom tag. There will be a grace period for the community to decide whether the branch should be kept in Unmaintained status or deleted, as described in [1]. Eventually (either when the grace period expires, or later, when there is no longer interest in keeping unmaintained/yoga around), the HEAD will be tagged yoga-eol and the branch deleted. (3) transitioning current eligible Extended Maintenance branches to Unmaintained status These are stable/xena, stable/wallaby, and stable/victoria. Ericsson has expressed interest in keeping victoria in Unmaintained status. Subject to the guidelines in [1], wallaby, xena, and yoga will therefore be kept in Unmaintained status as well. Eventually, consistent with [1], these branches will be tagged $series-eol and deleted. For more details about (1)-(3) above, see the discussion in [6]. The current patch to the Project Team Guide [7] will be updated shortly to reflect the discussion in [6]. (4) amendment to the Unmaintained branch resolution The original plan in [1] was for there to be a $project-unmaintained-core Gerrit group for each OpenStack project that would have +2/+W powers on the unmaintained/$series branches. This would entail changes in the Gerrit ACL config files for all projects plus populating all the Gerrit groups. I've put up an amendment [8] to the resolution that would instead establish a single openstack-unmaintained-core Gerrit group that would have +2/+W powers over all the unmaintained/* branches in the openstack namespace. Projects that would prefer to stick with the original proposal and have tighter control over their unmaintained/* branches can still do so; they'll have to patch the appropriate Gerrit ACL config file(s) for their project and populate the $project-unmaintained-core group. Note that except for the Amendment mentioned in (4), this email only provides details about how [1] is to be implemented; it does not change the nature of the Unmaintained branch proposal that was adopted by the TC back in July 2023. If you have questions or comments, please reply to the openstack-discuss mailing list, or leave comments on [7] and [8]. [0] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [1] https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branc… [2] https://meetings.opendev.org/meetings/tc/2023/tc.2023-10-31-18.00.log.html#… [3] https://meetings.opendev.org/meetings/tc/2023/tc.2023-11-07-18.00.log.html#… video meeting recording: https://www.youtube.com/watch?v=qv6nSRhfV_M [4] https://meetings.opendev.org/meetings/releaseteam/2023/releaseteam.2023-11-… [5] https://meetings.opendev.org/meetings/tc/2023/tc.2023-11-14-18.00.log.html#… [6] https://etherpad.opendev.org/p/early-caracal-unmaintained-transition [7] https://review.opendev.org/c/openstack/project-team-guide/+/897505 [8] https://review.opendev.org/c/openstack/governance/+/900940

1 0

Discuss i18n IRC Meeting date
by Seongsoo Cho 14 Nov '23

14 Nov '23

Hi i18n Team. It's time to update our regular meeting date. This is the current meeting date in the doc, but actually, there are no meeting. [1] Every odd week: 16:00 UTC, Thursday Every even week: 07:00 UTC, Thursday So I want to set a new meeting date. For now, I'd like to have a monthly IRC meeting. Please let me know what days and times work best for you. Survey form: https://forms.gle/SZUFmwJS6QTs4akNA Best Regards Seongsoo [1] https://docs.openstack.org/i18n/latest/i18n_team_meeting.html -- Seongsoo Cho OpenStack Korea User Group / Community Leader IRC #seongsoocho

1 0

[ptls][PTG] OpenInfra Live: PTG Recap
by Kristin Barrientos 14 Nov '23

14 Nov '23

Hi Everyone, I wanted to reach out to see if any PTLs were interested in providing their highlights from last month’s PTG in an OpenInfra Live[1] episode on Thursday, December 7 at 1500 UTC. Ideally, we would like to have 4-6 projects represented (OpenStack, Kata, and/or StarlingX). A previous example of a PTG recap episode can be found here[2]. Please let me know if you’re interested and I can provide the next steps. If you would like to provide an update but that time doesn’t work for you, please share a recording with me and I can get it added. Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation [1] openinfra.dev/live [2] https://www.youtube.com/watch?v=yIt4dJvTQVg&t=231s

7 11

[swift]: Swiftstack Zone/DC decommission
by Muralikrishna Gutha 14 Nov '23

14 Nov '23

Hello Team, We currently utilize SwiftStack Object Storage to meet our object storage requirements, and our cluster has a Primary Zone/Secondary Zone topology. Recently, we attempted to remove/disable the Secondary Zone. However, upon doing so, we observed a concerning trend: the storage utilization in the Primary Zone rapidly increased, reaching approximately 90%. This surge was particularly notable due to the use of the 'tworeplicaperregion' policy. In an attempt to address this, we brought the Secondary Zone nodes back online, expecting Swift to redistribute the data back to these nodes and alleviate the storage strain on the Primary Zone. Despite reintegrating the Secondary Zone nodes into the cluster, we continue to witness a persistent rise in storage utilization within the Primary Zone. Interestingly, two out of the three nodes in the Primary Zone seem to bear the brunt of the data and tokens, while the third node is not receiving a proportional distribution of data. At this juncture, we are seeking guidance on potential configurations or adjustments, such as specific knobs, that could expedite the rebalance process. Our objective is to efficiently distribute the data across nodes, particularly in the Primary Zone, and optimize the overall storage utilization in our SwiftStack Object Storage cluster. Any insights or recommendations on tuning parameters for a swifter rebalance would be highly appreciated. Thanks, Murali -- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.

3 7

[tc] Next TC Meeting will be Nov 28 (2023-11-21 meeting cancelled)
by Jay Faulkner 14 Nov '23

14 Nov '23

Hi all, The Technical Committee will not be meeting next week. The meeting was cancelled due to American Thanksgiving. Our next meeting will be November 28, 2023. Thanks, Jay Faulkner OpenStack TC Chair

1 0

Implementation review day will be Nov 15
by Sylvain Bauza 14 Nov '23

14 Nov '23

Hey folks, We just agreed during today's meeting on having our first implementation review day this week. In order to make this day efficient, I proposed an etherpad for tracking our efforts https://etherpad.opendev.org/p/nova-caracal-status Shall you want the nova community to look at your changes, you should please add them into the etherpad in the very last section ('proposed bugfixes') and then me and a few other cores will curate that list. Tomorrow, we'll scrub the list of approved blueprints's patches and the bugfixes' ones that got approved as a review priority and we'll iterate over them. Please try to look at your Gerrit emails, you could get comments ;-) Thanks, -Sylvain

1 1

the volume cannot be deleted, it is in deleting state
by kjme001＠gmail.com 14 Nov '23

14 Nov '23

hello, I'm having trouble deleting volumes in OpenStackl from a machine that no longer exists. How to execute cinder command reset-state --attach-status detached volume_name then I execute the command: # cinder delete volume_name Delete for volume cifar-10-copy failed: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-b2d3f2e4-7855-4831-a68d-f6710cc97e94) ERROR: Unable to delete any of the specified volumes. then executes the command: cinder reset-state --state available volume_name and cinder delete volume_name Request to delete volume volume_name has been accepted. when I check: openstack volume show volume_name is: | status | deleting The volume is not being deleted, it is in the deleting state. This is a volume from a non-existent node, but the volume is still visible in OpenStack and cannot be deleted because it is in the deleting state. Maybe someone can tell me how to remove such a volume? Regards

1 0

cinder-manage service remove cinder-volume xxxx@lvm-ssd - error
by kjme001＠gmail.com 14 Nov '23

14 Nov '23

cinder-manage service remove cinder-volume xxxx@lvm-ssd /usr/lib/python3/dist-packages/cinder/db/sqlalchemy/models.py:152: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery. last_heartbeat = column_property( /usr/lib/python3/dist-packages/cinder/db/sqlalchemy/models.py:160: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery. num_hosts = column_property( /usr/lib/python3/dist-packages/cinder/db/sqlalchemy/models.py:169: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery. num_down_hosts = column_property( 2023-11-11 14:34:00.928 898920 CRITICAL cinder [req-525973bb-e026-4999-95af-d852aa137066 - - - - -] Unhandled error: oslo_db.exception.DBNonExistentDatabase: (sqlite3.OperationalError) unable to open database file (Background on this error at: https://sqlalche.me/e/14/e3q8) 2023-11-11 14:34:00.928 898920 ERROR cinder Traceback (most recent call last): 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 3250, in _wrap_pool_connect 2023-11-11 14:34:00.928 898920 ERROR cinder return fn() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 310, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return _ConnectionFairy._checkout(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 868, in _checkout 2023-11-11 14:34:00.928 898920 ERROR cinder fairy = _ConnectionRecord.checkout(pool) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 476, in checkout 2023-11-11 14:34:00.928 898920 ERROR cinder rec = pool._do_get() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 256, in _do_get 2023-11-11 14:34:00.928 898920 ERROR cinder return self._create_connection() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 256, in _create_connection 2023-11-11 14:34:00.928 898920 ERROR cinder return _ConnectionRecord(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 371, in __init__ 2023-11-11 14:34:00.928 898920 ERROR cinder self.__connect() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 665, in __connect 2023-11-11 14:34:00.928 898920 ERROR cinder with util.safe_reraise(): 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ 2023-11-11 14:34:00.928 898920 ERROR cinder compat.raise_( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_ 2023-11-11 14:34:00.928 898920 ERROR cinder raise exception 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 661, in __connect 2023-11-11 14:34:00.928 898920 ERROR cinder self.dbapi_connection = connection = pool._invoke_creator(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/create.py", line 590, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return dialect.connect(*cargs, **cparams) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 597, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return self.dbapi.connect(*cargs, **cparams) 2023-11-11 14:34:00.928 898920 ERROR cinder sqlite3.OperationalError: unable to open database file 2023-11-11 14:34:00.928 898920 ERROR cinder 2023-11-11 14:34:00.928 898920 ERROR cinder The above exception was the direct cause of the following exception: 2023-11-11 14:34:00.928 898920 ERROR cinder 2023-11-11 14:34:00.928 898920 ERROR cinder Traceback (most recent call last): 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/bin/cinder-manage", line 10, in <module> 2023-11-11 14:34:00.928 898920 ERROR cinder sys.exit(main()) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/cmd/manage.py", line 1069, in main 2023-11-11 14:34:00.928 898920 ERROR cinder fn(**fn_kwargs) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/cmd/manage.py", line 711, in remove 2023-11-11 14:34:00.928 898920 ERROR cinder svc = objects.Service.get_by_args(ctxt, host_name, binary) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/objects/service.py", line 133, in get_by_args 2023-11-11 14:34:00.928 898920 ERROR cinder db_service = db.service_get(context, host=host, binary=binary_key) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/api.py", line 270, in service_get 2023-11-11 14:34:00.928 898920 ERROR cinder return IMPL.service_get(context, service_id, backend_match_level, 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 168, in wrapper 2023-11-11 14:34:00.928 898920 ERROR cinder return f(*args, **kwargs) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 745, in service_get 2023-11-11 14:34:00.928 898920 ERROR cinder query = _service_query(context, backend_match_level=backend_match_level, 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 686, in _service_query 2023-11-11 14:34:00.928 898920 ERROR cinder query = model_query(context, models.Service, session=session, 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 259, in model_query 2023-11-11 14:34:00.928 898920 ERROR cinder session = kwargs.get('session') or get_session() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 95, in get_session 2023-11-11 14:34:00.928 898920 ERROR cinder return main_context_manager._factory.get_legacy_facade().get_session( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 357, in get_legacy_facade 2023-11-11 14:34:00.928 898920 ERROR cinder self._start() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 504, in _start 2023-11-11 14:34:00.928 898920 ERROR cinder self._setup_for_connection( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 529, in _setup_for_connection 2023-11-11 14:34:00.928 898920 ERROR cinder engine = engines.create_engine( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator 2023-11-11 14:34:00.928 898920 ERROR cinder return wrapped(*args, **kwargs) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 211, in create_engine 2023-11-11 14:34:00.928 898920 ERROR cinder test_conn = _test_connection(engine, max_retries, retry_interval) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 386, in _test_connection 2023-11-11 14:34:00.928 898920 ERROR cinder return engine.connect() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 3204, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return self._connection_cls(self, close_with_result=close_with_result) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 96, in __init__ 2023-11-11 14:34:00.928 898920 ERROR cinder else engine.raw_connection() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 3283, in raw_connection 2023-11-11 14:34:00.928 898920 ERROR cinder return self._wrap_pool_connect(self.pool.connect, _connection) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 3253, in _wrap_pool_connect 2023-11-11 14:34:00.928 898920 ERROR cinder Connection._handle_dbapi_exception_noconnection( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2098, in _handle_dbapi_exception_noconnection 2023-11-11 14:34:00.928 898920 ERROR cinder util.raise_(newraise, with_traceback=exc_info[2], from_=e) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_ 2023-11-11 14:34:00.928 898920 ERROR cinder raise exception 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 3250, in _wrap_pool_connect 2023-11-11 14:34:00.928 898920 ERROR cinder return fn() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 310, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return _ConnectionFairy._checkout(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 868, in _checkout 2023-11-11 14:34:00.928 898920 ERROR cinder fairy = _ConnectionRecord.checkout(pool) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 476, in checkout 2023-11-11 14:34:00.928 898920 ERROR cinder rec = pool._do_get() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 256, in _do_get 2023-11-11 14:34:00.928 898920 ERROR cinder return self._create_connection() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 256, in _create_connection 2023-11-11 14:34:00.928 898920 ERROR cinder return _ConnectionRecord(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 371, in __init__ 2023-11-11 14:34:00.928 898920 ERROR cinder self.__connect() 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 665, in __connect 2023-11-11 14:34:00.928 898920 ERROR cinder with util.safe_reraise(): 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ 2023-11-11 14:34:00.928 898920 ERROR cinder compat.raise_( 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_ 2023-11-11 14:34:00.928 898920 ERROR cinder raise exception 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 661, in __connect 2023-11-11 14:34:00.928 898920 ERROR cinder self.dbapi_connection = connection = pool._invoke_creator(self) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/create.py", line 590, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return dialect.connect(*cargs, **cparams) 2023-11-11 14:34:00.928 898920 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 597, in connect 2023-11-11 14:34:00.928 898920 ERROR cinder return self.dbapi.connect(*cargs, **cparams) 2023-11-11 14:34:00.928 898920 ERROR cinder oslo_db.exception.DBNonExistentDatabase: (sqlite3.OperationalError) unable to open database file 2023-11-11 14:34:00.928 898920 ERROR cinder (Background on this error at: https://sqlalche.me/e/14/e3q8) 2023-11-11 14:34:00.928 898920 ERROR cinder

3 4

(no subject)
by mgutha＠liveperson.com 13 Nov '23

13 Nov '23

I have observed an imbalance in the cluster based on the object.builder output. Is this possibly due to an ongoing rebalance process? Additionally, I'd like to confirm my understanding of 'min_part_hours.' Is it accurate to say that this parameter represents the number of hours an object needs to reside on a disk before it's considered for movement to another device during a rebalance? According to the Swift documentation, the default value is 24 hours. I'm curious if this value is applied to data written specifically by the rebalance process or if it also includes data written by other Swift services. object.builder, build version 775, id 61b2166cb8684b2b80d6fc370cef37a6 65536 partitions, 2.000000 replicas, 2 regions, 2 zones, 95 devices, 38.89 balance, 0.00 dispersion The minimum number of hours before a partition can be reassigned is 0 (0:00:00 remaining) The overload factor is 10.00% (0.100000) Ring file object.ring.gz not found, probably it hasn't been written yet Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta 129 1 1 182.16.128.36:6000 182.16.128.36:6003 d129 1.00 2 38.89 130 1 1 182.16.128.36:6006 182.16.128.36:6003 d130 1.00 2 38.89 131 1 1 182.16.128.36:6007 182.16.128.36:6003 d131 1.00 2 38.89 132 1 1 182.16.128.36:6008 182.16.128.36:6003 d132 1.00 2 38.89 133 1 1 182.16.128.36:6009 182.16.128.36:6003 d133 1.00 2 38.89 134 1 1 182.16.128.36:6010 182.16.128.36:6003 d134 1.00 1 -30.56 135 1 1 182.16.128.36:6011 182.16.128.36:6003 d135 1.00 1 -30.56 136 1 1 182.16.128.36:6012 182.16.128.36:6003 d136 1.00 1 -30.56 137 1 1 182.16.128.36:6013 182.16.128.36:6003 d137 1.00 1 -30.56 138 1 1 182.16.128.36:6014 182.16.128.36:6003 d138 1.00 1 -30.56 139 1 1 182.16.128.36:6015 182.16.128.36:6003 d139 1.00 1 -30.56 140 1 1 182.16.128.38:6000 182.16.128.38:6003 d140 1.00 2 38.89 141 1 1 182.16.128.38:6006 182.16.128.38:6003 d141 1.00 2 38.89 142 1 1 182.16.128.38:6007 182.16.128.38:6003 d142 1.00 2 38.89 143 1 1 182.16.128.38:6008 182.16.128.38:6003 d143 1.00 2 38.89 144 1 1 182.16.128.38:6009 182.16.128.38:6003 d144 1.00 2 38.89 145 1 1 182.16.128.38:6010 182.16.128.38:6003 d145 1.00 1 -30.56 146 1 1 182.16.128.38:6011 182.16.128.38:6003 d146 1.00 1 -30.56 147 1 1 182.16.128.38:6012 182.16.128.38:6003 d147 1.00 1 -30.56 148 1 1 182.16.128.38:6013 182.16.128.38:6003 d148 1.00 1 -30.56 149 1 1 182.16.128.38:6014 182.16.128.38:6003 d149 1.00 1 -30.56 150 1 1 182.16.128.38:6015 182.16.128.38:6003 d150 1.00 1 -30.56 151 1 1 182.16.128.39:6000 182.16.128.39:6003 d151 1.00 2 38.89 152 1 1 182.16.128.39:6006 182.16.128.39:6003 d152 1.00 2 38.89 153 1 1 182.16.128.39:6007 182.16.128.39:6003 d153 1.00 2 38.89 154 1 1 182.16.128.39:6008 182.16.128.39:6003 d154 1.00 2 38.89 155 1 1 182.16.128.39:6009 182.16.128.39:6003 d155 1.00 2 38.89 156 1 1 182.16.128.39:6010 182.16.128.39:6003 d156 1.00 1 -30.56 157 1 1 182.16.128.39:6011 182.16.128.39:6003 d157 1.00 1 -30.56 158 1 1 182.16.128.39:6012 182.16.128.39:6003 d158 1.00 1 -30.56 159 1 1 182.16.128.39:6013 182.16.128.39:6003 d159 1.00 1 -30.56 160 1 1 182.16.128.39:6014 182.16.128.39:6003 d160 1.00 1 -30.56 161 1 1 182.16.128.39:6015 182.16.128.39:6003 d161 1.00 1 -30.56 55 1 1 182.16.128.5:6000 182.16.128.5:6003 d55 2000.40 2789 -3.18 56 1 1 182.16.128.5:6006 182.16.128.5:6003 d56 2000.40 2789 -3.18 57 1 1 182.16.128.5:6007 182.16.128.5:6003 d57 2000.40 2789 -3.18 58 1 1 182.16.128.5:6008 182.16.128.5:6003 d58 2000.40 2789 -3.18 59 1 1 182.16.128.5:6009 182.16.128.5:6003 d59 2000.40 2789 -3.18 60 1 1 182.16.128.5:6010 182.16.128.5:6003 d60 2000.40 2789 -3.18 61 1 1 182.16.128.5:6011 182.16.128.5:6003 d61 2000.40 2789 -3.18 62 1 1 182.16.128.5:6012 182.16.128.5:6003 d62 2000.40 2789 -3.18 63 1 1 182.16.128.5:6013 182.16.128.5:6003 d63 2000.40 2789 -3.18 64 1 1 182.16.128.5:6014 182.16.128.5:6003 d64 2000.40 2789 -3.18 65 1 1 182.16.128.5:6015 182.16.128.5:6003 d65 2000.40 2788 -3.21 44 1 1 182.16.128.6:6000 182.16.128.6:6003 d44 269.43 376 -3.09 45 1 1 182.16.128.6:6006 182.16.128.6:6003 d45 269.43 376 -3.09 46 1 1 182.16.128.6:6007 182.16.128.6:6003 d46 269.43 376 -3.09 47 1 1 182.16.128.6:6008 182.16.128.6:6003 d47 269.43 376 -3.09 48 1 1 182.16.128.6:6009 182.16.128.6:6003 d48 269.43 376 -3.09 49 1 1 182.16.128.6:6010 182.16.128.6:6003 d49 269.43 376 -3.09 50 1 1 182.16.128.6:6011 182.16.128.6:6003 d50 269.43 376 -3.09 51 1 1 182.16.128.6:6012 182.16.128.6:6003 d51 269.43 375 -3.35 52 1 1 182.16.128.6:6013 182.16.128.6:6003 d52 269.43 375 -3.35 53 1 1 182.16.128.6:6014 182.16.128.6:6003 d53 269.43 375 -3.35 54 1 1 182.16.128.6:6015 182.16.128.6:6003 d54 269.43 375 -3.35 0 1 1 182.16.128.7:6000 182.16.128.7:6003 d0 2000.40 2789 -3.18 1 1 1 182.16.128.7:6006 182.16.128.7:6003 d1 2000.40 2789 -3.18 10 1 1 182.16.128.7:6015 182.16.128.7:6003 d10 2000.40 2788 -3.21 2 1 1 182.16.128.7:6007 182.16.128.7:6003 d2 2000.40 2789 -3.18 3 1 1 182.16.128.7:6008 182.16.128.7:6003 d3 2000.40 2789 -3.18 4 1 1 182.16.128.7:6009 182.16.128.7:6003 d4 2000.40 2789 -3.18 5 1 1 182.16.128.7:6010 182.16.128.7:6003 d5 2000.40 2789 -3.18 6 1 1 182.16.128.7:6011 182.16.128.7:6003 d6 2000.40 2789 -3.18 7 1 1 182.16.128.7:6012 182.16.128.7:6003 d7 2000.40 2789 -3.18 8 1 1 182.16.128.7:6013 182.16.128.7:6003 d8 2000.40 2789 -3.18 9 1 1 182.16.128.7:6014 182.16.128.7:6003 d9 2000.40 2789 -3.18 11 2 2 11.17.128.5:6000 11.17.128.5:6003 d11 2000.40 2979 3.42 12 2 2 11.17.128.5:6006 11.17.128.5:6003 d12 2000.40 2979 3.42 13 2 2 11.17.128.5:6007 11.17.128.5:6003 d13 2000.40 2979 3.42 15 2 2 11.17.128.5:6008 11.17.128.5:6003 d15 2000.40 2979 3.42 16 2 2 11.17.128.5:6009 11.17.128.5:6003 d16 2000.40 2979 3.42 17 2 2 11.17.128.5:6010 11.17.128.5:6003 d17 2000.40 2978 3.38 18 2 2 11.17.128.5:6011 11.17.128.5:6003 d18 2000.40 2978 3.38 19 2 2 11.17.128.5:6012 11.17.128.5:6003 d19 2000.40 2978 3.38 20 2 2 11.17.128.5:6013 11.17.128.5:6003 d20 2000.40 2978 3.38 21 2 2 11.17.128.5:6014 11.17.128.5:6003 d21 2000.40 2978 3.38 77 2 2 11.17.128.5:6015 11.17.128.5:6003 d77 2000.40 2978 3.38 33 2 2 11.17.128.6:6000 11.17.128.6:6003 d33 2000.40 2979 3.42 34 2 2 11.17.128.6:6006 11.17.128.6:6003 d34 2000.40 2979 3.42 35 2 2 11.17.128.6:6007 11.17.128.6:6003 d35 2000.40 2979 3.42 36 2 2 11.17.128.6:6008 11.17.128.6:6003 d36 2000.40 2979 3.42 37 2 2 11.17.128.6:6009 11.17.128.6:6003 d37 2000.40 2978 3.38 38 2 2 11.17.128.6:6010 11.17.128.6:6003 d38 2000.40 2978 3.38 39 2 2 11.17.128.6:6011 11.17.128.6:6003 d39 2000.40 2978 3.38 40 2 2 11.17.128.6:6012 11.17.128.6:6003 d40 2000.40 2978 3.38 41 2 2 11.17.128.6:6013 11.17.128.6:6003 d41 2000.40 2978 3.38 42 2 2 11.17.128.6:6014 11.17.128.6:6003 d42 2000.40 2978 3.38 43 2 2 11.17.128.6:6015 11.17.128.6:6003 d43 2000.40 2978 3.38 162 2 2 11.17.128.71:6000 11.17.128.71:6003 d162 1.00 2 38.89 163 2 2 11.17.128.71:6006 11.17.128.71:6003 d163 1.00 2 38.89 164 2 2 11.17.128.71:6007 11.17.128.71:6003 d164 1.00 2 38.89 165 2 2 11.17.128.71:6008 11.17.128.71:6003 d165 1.00 2 38.89 166 2 2 11.17.128.71:6009 11.17.128.71:6003 d166 1.00 1 -30.56 167 2 2 11.17.128.71:6010 11.17.128.71:6003 d167 1.00 1 -30.56 168 2 2 11.17.128.71:6011 11.17.128.71:6003 d168 1.00 1 -30.56

3 2

Re: [swift]: Swiftstack Zone/DC decommission
by Muralikrishna Gutha 13 Nov '23

13 Nov '23

Clary, Thanks for the update I was unable to locate the object.builder file in the instance where we are running swiftstack. We are currently on Swift Package Version: 2.24.0.3-1.el7 [root@hostname swift]# /opt/ss/bin/swift-ring-builder object.builder | head Ring Builder file does not exist: object.builder [root@hostname swift]# locate object.builder [root@hostname swift]#pwd /etc/swift Thanks, Murali -- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.

2 1

[PTL][release][stable] Yoga - final release before branch becomes unmaintained
by Elõd Illés 13 Nov '23

13 Nov '23

Hi teams, According to the new process [1] Yoga should transition to Unmaintained as soon as possible (the original deadline for Extended Maintenance transition was planned to 2023-11-02). Similarly like before, teams have the final opportunity to release from stable/yoga before the branches will be *deleted*. In order to help the teams I have generated the list of the current *open* and *unreleased* changes in stable/yoga for repositories [2] where there are relevant patches. Please do a *final* release on Yoga if needed, before stable/yoga transition to Unmaintained. Feel free to edit and extend these lists to track your team's progress! *NOTE*: teams, please focus on wrapping up your libraries first if there is any concern about the changes, in order to avoid broken (final!!) releases! [1] https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branc… [2] https://etherpad.opendev.org/p/yoga-final-release-before-unmaintained [3] https://review.opendev.org/q/topic:xena-em Thanks, Előd irc: elodilles @ #openstack-release

2 1

[neutron][ptg] 2024.1/Caracal vPTG summary
by Brian Haley 13 Nov '23

13 Nov '23

Sorry for the slowness of getting this out. Here is a summary of the discussions we had over our two days of meetings, as well as other related sessions. The etherpad link can be found at [0]. Monday, October 23rd -------------------- # OS DBPerf Cross project session [1] This session was to talk about devstack performance issues that are believed to be caused by excessive querying by Neutron and Keystone. By default, when devstack "stacks", DB counter values are collected on each of the subsystems. Keystone and Neutron are outliers on how many SELECTs they issue during this process. As this is such a high-level number (4x to 10x other users), more work is required to understand what to fix, or if it's just normal behavior. For example, there is data from just a "stack" in addition to a tempest run that might be able to narrow things down here. Slawek and Miguel have agreed to work with the oslo.db team and Mike Bayer to get a better understanding of things here. Tuesday, October 24th --------------------- # Horizon For the last few cycles the Neutron team (Lajos) has been trying to deprecate usage of neutronclient. While this work is mostly complete, there is usage in Horizon that needs to be addressed. Lajos and Amotoki will be working on this. Full notes at [2] Wednesday, October 25th ----------------------- # Retrospective As is customary the team did a retrospective of the good, bad and what could be improved. In short, the Neutron team has met its goals from last cycle of completing Sqlalchemy 2.0 and RBAC work, while continuing to deliver a few new features and a lot of bug fixes. Most of the team are long time members which helps. One place we are working on is our job stability, and we continue to look at how to increase our review velocity. Our initial proposal on the latter is to have review marathons, perhaps on a monthly basis, since it has the double effect of closing bugs in the process. # EOL/EM cadence Train just went EOL. Decision was to propose Ussuri for EOL this cycle. # Bugs The total number of open bugs against Neutron has crept up since the beginning of Bobcat, from 726 to 760. I asked people to go through any bugs they own and make sure they are in the right state or should be closed. This is something we've done in previous cycles that has reduced our count by 1/3. # Gate We now have voting SLURP jobs! I.e. from Antelope to Caracal. Rodolfo has also been working on fullstack performance to try and reduce the runtime since we hit the 3 hour limit occasionally. Action item was taken by Slawek to see if there was overlap with other tests to see what could be removed. There was a question on what versions of OVS and OVN we use in our gate, and if there is a rule on when we should bump it, especially since sometimes new features require newer versions. Action item was taken by me to create a requirements doc similar to what Nova does for libvirt. # Neutronclient deprecation As discussed above, Horizon will be the focus this cycle. # OVN drivers for stadium projects There have been efforts recently to make stadium projects work with OVN, for example VPNaaS, FWaas and Taas. Main ask for for review cycles from core contributors. # Hidden SG rules in OVN Background is at [3]. Short story is that there are a minimum set of things that must be allowed for a VM to configure and use the network, like ARP, DHCP, ND, RA, RS and MLD. ML2/OVS adds hidden SG rules for these, and OVN allows some via ACLs. Our api-ref actually documents these requirements since stateless SGs needed to add them to function properly. The only service that is not enabled by default with stateless SGs is metadata. Question was regarding if we should make these rules visible in a read-only state. # S-RBAC This is a community goal and Neutron has completed the first phase and is part of phase 2 (service role), additional testing and nova/neutron token passing TBD. Phase 3 is the Project Manager role, which is not planned for this cycle. Thursday, October 26th ---------------------- # RFE reviews Discussed three active RFEs, two related to neutron-dynamic-routing, and one for OVN interconnection. Further discussion to take place in reviews and/or drivers meeting. # Metadata changes I noticed there were a number of changes to the metadata code in flight, so we went through them to try and create order out of the chaos. Basically, we will try and merge any bug fixes first since they need to be backported, followed by features and refactors. See the etherpad for more details. # OVN DB sync utility This tool is typically run after a migration or upgrade to make sure the Neutron and OVN DBs are in sync. A number of bugs have been fixed and backported, but there might be more. An action item was taken to try and make it more robust, and also make sure usage is documented. # Active-Active L3 gateway multi-homing Frode Nordahl gave an update on this OVN work that started in the Bobcat cycle, necessary to support multiple external gateways. A little over half the code is merged, plan is to complete the rest this cycle. # Nova cross-project Sean Mooney proposed that Neutron starts making certain extensions mandatory, for example, multiple port bindings, that make migration possible. Since then other features have been added that require it as well. Should we stop supporting backends that have not implemented it? We agreed this is a good step forward to what should be the minimum requirements of Neutron. In addition: - Nova will deprecate the legacy code path that doesn't use multiple bindings and log a warning, to be deleted in E release - Neutron will do the same for any similar code - Sean will provide a blueprint explaining which API extensions will be mandatory by Nova, as well as file a Neutron RFE bug - Eventually, we'll provide release notes and a ML thread for explaining all of this (we being the Nova and Neutron PTLs) Complete notes in Nova etherpad [4]. Thanks, and if you think I missed something please send updates or corrections. -Brian [0] https://etherpad.opendev.org/p/oct2023-ptg-neutron [1] https://etherpad.opendev.org/p/oct2023-ptg-os-dbperf-crossproject [2] https://etherpad.opendev.org/p/horizon-caracal-ptg [3] https://bugs.launchpad.net/neutron/+bug/1926515 [4] https://etherpad.opendev.org/p/nova-caracal-ptg#L80

2 2

[ops][cinder][kolla] Cinder + Ceph under high load can lead to timeouts and a failure state.
by Forrest Fuqua 13 Nov '23

13 Nov '23

In my setup of Cinder Zed + Ceph Quincy under very high loads there is a race condition due to a low timeout by default in Openstack Kolla in Cinder.conf of rados_connect_timeout = 5 This can lead to a duplicate volume trying to be made on ceph and cinder erroring out a VM Build in Nova. Nova: nova.exception.BuildAbortException: Build of instance a529f7d5-5a81-44f6-a1fd-93caa39b2439 aborted: Volume b476094b-075e-4dc7-b63e-5bb58a0a9229 did not finish being created even after we waited 30 seconds or 4 attempts. And its status is error. Cinder-Volume: File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 230, in __init__\n self.volume = driver.rbd.Image(rados_ioctx,\n', ' File "rbd.pyx", line 2894, in rbd.Image.__init__\n', "rbd.Timeout: [errno 110] RBD operation timeout (error opening image b'volume-b476094b-075e-4dc7-b63e-5bb58a0a9229' at snapshot None)\n"] --- File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute\n six.reraise(c, e, tb)\n', ' File "/var/lib/kolla/venv/lib/python3.9/site-packages/six.py", line 719, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker\n rv = meth(*args, **kwargs)\n', ' File "rbd.pyx", line 698, in rbd.RBD.clone\n', 'rbd.ImageExists: [errno 17] RBD image already exists (error creating clone)\n'] --- I can confirm that the volume does exist in Ceph now but no longer being tracked in Cinder, leading to a leftover artifact that eats up resources and a failed VM build. This seems like behavior that should be accounted for in some way.

2 2

[swift] Swiftstack Zone/DC decomission
by Muralikrishna Gutha 13 Nov '23

13 Nov '23

Certainly, here's a rewritten version: We currently utilize SwiftStack Object Storage to meet our object storage requirements, and our cluster has a Primary Zone/Secondary Zone topology. Recently, we attempted to remove/disable the Secondary Zone. However, upon doing so, we observed a concerning trend: the storage utilization in the Primary Zone rapidly increased, reaching approximately 90%. This surge was particularly notable due to the use of the 'tworeplicaperregion' policy. In an attempt to address this, we brought the Secondary Zone nodes back online, expecting Swift to redistribute the data back to these nodes and alleviate the storage strain on the Primary Zone. Despite reintegrating the Secondary Zone nodes into the cluster, we continue to witness a persistent rise in storage utilization within the Primary Zone. Interestingly, two out of the three nodes in the Primary Zone seem to bear the brunt of the data and tokens, while the third node is not receiving a proportional distribution of data. At this juncture, we are seeking guidance on potential configurations or adjustments, such as specific knobs, that could expedite the rebalance process. Our objective is to efficiently distribute the data across nodes, particularly in the Primary Zone, and optimize the overall storage utilization in our SwiftStack Object Storage cluster. Any insights or recommendations on tuning parameters for a swifter rebalance would be highly appreciated. Thanks, Murali -- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.

1 0

[tc] Next meeting Tomorrow, Tuesday Nov 14, 1800 UTC
by Jay Faulkner 13 Nov '23

13 Nov '23

Hi all, The next meeting of the OpenStack Technical Committee is tomorrow, Tuesday November 14 at 1800 UTC in #openstack-tc on OFTC. The proposed agenda is: - Roll call - Follow up on tracked action items: https://meetings.opendev.org/meetings/tc/2023/tc.2023-11-07-18.00.html - Jay to sign up to rep TC at PTG OpenInfra Live recap - Gate health check - Leaderless projects (gmann) - Implementation of Unmaintained branch statuses - draft email to openstack-discuss while we figure this out: https://etherpad.opendev.org/p/early-caracal-unmaintained-transition - 2024.1 TC Tracker - https://etherpad.opendev.org/p/tc-2024.1-tracker - Open Discussion and Reviews Thanks all, Jay Faulkner OpenStack TC Chair

1 0

[swift][OpenStackSDK][keystone] Warnings about (my) swift endpoint URL by keystoneauth1 discovery
by Christian Rohmann 13 Nov '23

13 Nov '23

Hey openstack-discuss, we use Ceph RadosGW (Quincy release) to provide Swift as part of an OpenStack cloud. The endpoints are configured as such: > # openstack endpoint list > +----------------------------------+--------+--------------+----------------+---------+-----------+----------------------------------------------------------------------+ > | ID | Region | Service Name | Service > Type | Enabled | Interface | URL | > +----------------------------------+--------+--------------+----------------+---------+-----------+----------------------------------------------------------------------+ > | 1234567890 | region | swift | object-store | True | > public | > https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s > | > | 2345678901 | region | swift | object-store | True | > internal | > https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s > | > | 3456789012 | region | swift | object-store | True | > admin | > https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s > | > so according to https://docs.ceph.com/en/latest/radosgw/keystone/#cross-project-tenant-acce…, if I am not mistaken. I can also use the openstack client to access containers: > # openstack container list > +---------------+ > | Name | > +---------------+ > | containera | > | containerb | > +---------------+ and there is no warnings or other apparent issues. But when using "project cleanup" via e.g. "openstack project cleanup --dry-run --project $PROJECT" I see multiple warnings like > Failed to contact the endpoint at > https://object-store.region.cloud.example.com/swift/v1/AUTH_f2bc4bd34567ddc… > for discovery. Fallback to using that endpoint as the base url. > but only mentioning the swift endpoint. Following the warning string this originates from the discovery method of keystoneauth1, see https://github.com/openstack/keystoneauth/blob/28048af9593740315df8d9027c3b…. Could anybody help me understand what is issue might be? Why does this warning not appear for every use of object storage / swift via openstackclient? Is there any way for me to "fix" this? Could this even be a bug? Thanks and with kind regards Christian

4 10

report an error
by 许伟祥 13 Nov '23

13 Nov '23

carry out iaas-pre-host.sh report an error Downloading packages: warning: /var/cache/yum/x86_64/7/iaas/packages/python-fixtures-3.0.0-2.el7.noarch.rpm: Header V4 RSA/SHA1 Signature, key ID 764429e6: NOKEY Public key for python-fixtures-3.0.0-2.el7.noarch.rpm is not installed Please Reboot or Reconnect the terminal How to fix it 从 Windows 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>发送

1 0