December 2024 - openstack-discuss

OpenStack APIs
by rushit.ajudiya＠crestdata.ai 18 Dec '24

18 Dec '24

Hello, We want to retrieve all hypervisors available in OpenStack using the Hypervisor API documented at https://docs.openstack.org/api-ref/compute/#list-hypervisors-details. The documentation states that the "limit" and "marker" parameters can be used to manage pagination, but they don’t seem to work as expected. Is this a problem with the API, or is there another parameter not mentioned in the documentation that we should use?

2 1

[placement] isolating aggregates on kolla-ansible
by przemyslaw.kuczynski＠ochk.pl 18 Dec '24

18 Dec '24

Did someone manage to configure Windows host aggregate ? I tried same is in https://docs.openstack.org/nova/2024.2/reference/isolate-aggregates.html 1. Created 2 aggregates in 2 AZ , added 1 host to each 2. Set the scheduler.enable_isolated_aggregate_filtering config option to true in nova.conf and restart the nova-scheduler service cat /etc/kolla-pilot/config/nova.conf [libvirt] hw_machine_type=x86_64=q35 [scheduler] enable_isolated_aggregate_filtering = True 3. Add trait CUSTOM_LICENSED_WINDOWS to the resource providers for kvm01 and kvm03 in the Placement service.openstack resource provider list +--------------------------------------+-------+------------+--------------------------------------+----------------------+ | uuid | name | generation | root_provider_uuid | parent_provider_uuid | +--------------------------------------+-------+------------+--------------------------------------+----------------------+ | 36c0ae68-cc5f-4219-94bb-e46bcd3311d9 | kvm01 | 257 | 36c0ae68-cc5f-4219-94bb-e46bcd3311d9 | None | | 8abb4b5e-0407-405d-97d1-7a03a9a06cbe | kvm03 | 129 | 8abb4b5e-0407-405d-97d1-7a03a9a06cbe | None | | 14ed0962-d8ce-4bed-b90a-243b3a069f5d | kvm02 | 1096 | 14ed0962-d8ce-4bed-b90a-243b3a069f5d | None | | 8714da8e-7e9b-4823-93bb-df819c143e99 | kvm04 | 87 | 8714da8e-7e9b-4823-93bb-df819c143e99 | None | +--------------------------------------+-------+------------+--------------------------------------+----------------------+ openstack --os-placement-api-version 1.6 trait create CUSTOM_LICENSED_WINDOWStraits=$(openstack --os-placement-api-version 1.6 resource provider trait list -f value 36c0ae68-cc5f-4219-94bb-e46bcd3311d9 | sed 's/^/--trait /')openstack --os-placement-api-version 1.6 resource provider trait set $traits --trait CUSTOM_LICENSED_WINDOWS 36c0ae68-cc5f-4219-94bb-e46bcd3311d9traits=$(openstack --os-placement-api-version 1.6 resource provider trait list -f value 8abb4b5e-0407-405d-97d1-7a03a9a06cbe | sed 's/^/--trait /') openstack --os-placement-api-version 1.6 resource provider trait set $traits --trait CUSTOM_LICENSED_WINDOWS 8abb4b5e-0407-405d-97d1-7a03a9a06cbe openstack --os-compute-api-version 2.53 aggregate set --property trait:CUSTOM_LICENSED_WINDOWS=required WindowsW1 openstack --os-compute-api-version 2.53 aggregate set --property trait:CUSTOM_LICENSED_WINDOWS=required WindowsW2 Flavor metadata trait:CUSTOM_LICENSED_WINDOWS with value required nova.exception.NoValidHost: No valid host was found. There are not enough hosts available. Got no allocation candidates from the Placement API.

1 0

#eventlet-removal Thank you for 2024
by Herve Beraud 18 Dec '24

18 Dec '24

You are no doubt aware that we are currently facing a real challenge. We are living in a complex situation where we have to engage resources to exit from a problematic technology, eventlet, without especially gaining new features. This migration, although without immediate direct functional value, represente for us a considerable investment in time and resources. But such context is not far different from the Python 2/Python 3 migration. Remember that era. The core Python maintainers found themselves in a difficult situation: continue to maintain their code on Python 2, which had become insecure and unsupported, or go through the effort of completely migrating their codebase and toolchain to Python 3. All the community had to make a choice. Third-party libraries, frameworks were also on the edge case, to be upgraded or replaced with Python 3 compatible equivalents. At the time of the decision, these efforts appeared to represent a "non-value-added burden". These efforts did not bring any new functionality, did not respond to any new customer need, and even slowed down other more innovative developments. The Python maintainers could have chosen the easy way. They could have chosen to continue to support Python 2 again for years, they could have chosen to live in statu-quo, but that's not what happened [1]. They chose to fight the problem and they chose to engage strong efforts in this migration. They ensured the security and the perenniality of the Python programming language. Once the migration was completed, the benefits quickly became apparent. The innovation cadence increased. The community united behind one uniq major version. Better performances. More innovations. And last but not the least, a simpler life for the distros maintainers and for end-users, and so a reinforced confidence in Python by our industry. Python is now the number 1 technology, leaving Java and C in the dust [2]. This is the lesson we must learn. Necessary actions without immediate direct functional value can ultimately hide treasures of innovation. Engagement and dedication is the key to a sustainable future. It guarantees the long-term viability of a project, reduces risks, and opens the way to faster and more peaceful innovation. Today, I can tell you that I'm proud that our community engaged the same way. I'm proud to see all the milestones we reached during this exciting year. I'm proud to observe so much engagement coming from every side, from everybody, and from all our companies. That's a strong signal of trust from our industry. Our companies support this initiative, our companies support open source, our companies invest in the future of OpenStack. On behalf of this community initiative, let me thank you very much for all your efforts and engagements in 2024. Let me wish you happy holidays and enjoy your loved ones. [1] https://devguide.python.org/versions/ [2] https://www.tiobe.com/tiobe-index/ -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

[neutron] Hold the rechecks
by Yatin Karel 18 Dec '24

18 Dec '24

With the latest Ubuntu jammy kernel released, jobs running with openvswitch/linux bridge are impacted https://bugs.launchpad.net/neutron/+bug/2091990. Almost all branches are impacted, so hold any recheck until it's fixed. Other projects should also be seeing the issue when running such jobs. -- Thanks and Regards Yatin Karel

1 1

[qa] Team Organization Updates
by Martin Kopec 18 Dec '24

18 Dec '24

Hello everyone, I'm changing my work focus in the new year, therefore, I will have reduced bandwidth for QA and PTL duties for the rest of the Epoxy cycle. As a result, I'd like to announce a few changes: 1. Ghanshyam will take over moderating office hours starting in the new year. To better accommodate his timezone, the office hours will be held *bi-weekly on Wednesdays 18:00 UTC (even weeks).* The first office hour in the new year will be on *Wednesday January 8th*. 2. Ghanshyam will also act as PTL proxy for any emergencies, e.g. patches that require an approval asap. However, feel free to add me as a reviewer to patches that need a final review. 3. Luigi Toscano volunteered to help us with reviews, mainly with Tempest-related ones as he is a long-term contributor to cinder-tempest-plugin and several other projects - https://review.opendev.org/q/owner:ltoscano@redhat.com Thanks, -- Martin Kopec Red Hat EMEA IM: kopecmartin

1 0

[manila] Removal of the Standalone CephFS NFS protocol helper in Epoxy
by Ashley Rodriguez 18 Dec '24

18 Dec '24

Hi zorillas! As per subject line, we will be deprecating the standalone CephFS NFS protocol helper in this cycle. This is following our deprecation warning we put out in 2024.1 / Caracal [1] [2]. We've supported using a clustered Ceph NFS service since the Zed release. We've also added upgrade helpers to migrate from a standalone NFS-Ganesha service to the ceph-orch-deployed clustered NFS service; this code was backported to stable/2023.1 (Antelope). Although the upgrade path is disruptive, there is a UX improvement in Manila to ease this migration through surfacing “preferred” export paths. Moving forward, deployments must have “cephfs_nfs_cluster_id” in your configuration, otherwise the driver will fail to start. Manila will not make any export changes to NFS-Ganesha, so upgrading manila without an NFS cluster will mean that only the control plane is not going to work; but, data plane will remain unaffected. To assist migration, admins can also configure "cephfs_ganesha_export_ips" (or alternatively, "cephfs_ganesha_server_ip") alongside "cephfs_nfs_cluster_id". Setting these options will allow the CephFS driver to report additional export paths. These additional export paths will have the "preferred" metadata key set to False. Best, Manila team [1] https://review.opendev.org/c/openstack/manila/+/912648 [2] https://docs.openstack.org/manila/2024.2/configuration/shared-file-systems/…

1 0

[tc][all] No weekly meetings over the holidays (2024-12-24 and 2024-12-31)
by Goutham Pacha Ravi 17 Dec '24

17 Dec '24

Hello Stackers, As we approach the end-of-the-year holidays, please note that the OpenStack Technical Committee weekly meetings [1] previously scheduled for 2024-12-24 and 2024-12-31 are cancelled. We will resume our weekly meetings on 2025-01-07, and I will share the agenda closer to that date. Wishing everyone a happy and restful holiday season! Thank you, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

[openstack-ansible] Weekly meetings on Dec 24 and Dec 31 cancelled
by Dmitriy Rabotyagov 17 Dec '24

17 Dec '24

Hi, Last week we decided to cancel the weekly meetings on December 24 and December 31 2024 due to Christmas holidays as many people will not be around for the meeting. This means that the next meeting will happen only on January 7 2025. With that being said, I want to congratulate the community with the upcoming Christmas and New Year and wish to have plenty of fun during holidays! And thank for all the hard work that was put in during 2024.

1 0

[neutron] End of year meeting schedule
by Brian Haley 17 Dec '24

17 Dec '24

Hello Neutrinos: Just wanted to make sure everyone was aware of our meeting schedule for the rest of the year. As discussed in our meeting Tuesday, next week will be the last week of meetings for the year, specifically: December 13th - Drivers meeting cancelled December 16th - Neutron CI meeting December 17th - Neutron meeting December 20th - Drivers meeting cancelled December 23rd - Neutron CI meeting (?) will leave up to Yatin Meetings the week of December 23rd and December 30th are cancelled, starting up again the week of January 6th. As usual, reach out on IRC/gerrit/email with questions for me or the core team, but assume a potentially slow turn-around. I will be offline from December 19th to January 8th, checking-in occasionally to make sure there are no fires, Miguel has offered to lead the team meeting on January 7th in my absence (thanks Miguel!). -Brian, Neutron PTL

2 2

Vault cannot authorize approle
by Andy Speagle 17 Dec '24

17 Dec '24

Hey Team, I'm working with charmed openstack and it looks like I let my certs expire stupidly... and now I'm seeing this from vault: Vault cannot authorize approle Further info from the logs indicate this: unit-vault-0: 08:42:04 WARNING unit.vault/0.juju-log InternalServerError: Unable to authorize approle. This may indicate failure to communicate with the database unit-vault-0: 08:42:04 ERROR unit.vault/0.juju-log Traceback (most recent call last): File "/var/lib/juju/agents/unit-vault- 0/charm/reactive/vault_handlers.py", line 896, in client_approle_authorized vault.get_local_client() File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 339, in wrapped_f return self(f, *args, **kw) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 430, in __call__ do = self.iter(retry_state=retry_state) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 378, in iter raise retry_exc.reraise() File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 206, in reraise raise self.last_attempt.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 433, in __call__ result = fn(*args, **kwargs) File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault.py", line 258, in get_local_client client.auth_approle(app_role_id) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/v1/__init__.py", line 2072, in auth_approle return self.auth('/v1/auth/{0}/login'.format(mount_point), json=params, use_token=use_token) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/v1/__init__.py", line 1726, in auth return self._adapter.auth( File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/adapters.py", line 159, in auth response = self.post(url, **kwargs).json() File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/adapters.py", line 103, in post return self.request('post', url, **kwargs) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/adapters.py", line 233, in request utils.raise_for_error(response.status_code, text, errors=errors) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/utils.py", line 39, in raise_for_error raise exceptions.InternalServerError(message, errors=errors) hvac.exceptions.InternalServerError: internal error The database seems to be up and accessible... so, I gotta believe it's a cert issue... so, I feel like this might be a chicken-egg issue where it can't connect to the DB due to certs... but then can't issue certs because it can't connect to the DB. Any thoughts? ----- Andy Speagle

2 4

[neutron] bug deputy report for week of 2024-12-09+
by Ihar Hrachyshka 16 Dec '24

16 Dec '24

Hi all, Bugs ------- Gate issues ========= 1. fullstack failures in l3ha agent, placeholder: https://bugs.launchpad.net/neutron/+bug/2091351 May be related to https://bugs.launchpad.net/neutron/+bug/2083609 There's a fix for the latter: https://review.opendev.org/c/openstack/neutron/+/937758 2. functional suite OOMing when CPU number is low: https://bugs.launchpad.net/neutron/+bug/2091855 There's a fix to split the suite in serial runs: https://review.opendev.org/c/openstack/neutron/+/937759 The bug alludes that memory consumption could be improved; I'd recommend to limit the scope of this bug to fixing the CI failure and let the memory concerns be addressed elsewhere / by another owner if needed. 3. tempest failure in stable branch in psql job: https://bugs.launchpad.net/neutron/+bug/2091711 No fix; no owner. API / policy ========= All assigned to Slawek; not sure if he really wants to own all of them. 1. no quota on the number of tags breaks api: https://bugs.launchpad.net/neutron/+bug/2091410 DoS when too many tags are created; CVE adjacent. Need quotas for tags? 2. policy mechanism allows to work around some limitations on external connectivity when fwaas is used: https://bugs.launchpad.net/neutron/+bug/2091536 Not sure if it has a security angle or affects anything but fwaas. 3. custom policy rule failing due to missing fields: https://bugs.launchpad.net/neutron/+bug/2091493 Needs owner ========== 1. Unicast leakage on flat network: https://bugs.launchpad.net/neutron/+bug/2091211 The next step would be to set up a 2-compute devenv setup with flat and see if ml2/ovs openvswitch driver can reproduce. Additional tests to functional / fullstack / tempest validating leakage is not happening could be helpful too. Already fixed ========== 1. 500 when backend doesn't support vlan-transparent: https://bugs.launchpad.net/neutron/+bug/2091613 2. pylint change broke ovn agent delete: https://bugs.launchpad.net/neutron/+bug/2091071 New RFEs -------------- 1. use unicast mode for keepalived to avoid vxlan flooding when mcast is off: https://bugs.launchpad.net/neutron/+bug/2091599 Ihar

1 0

[Neutron] DHCP port status inactive
by Vinicius Afonso 16 Dec '24

16 Dec '24

Hello, I am new to Openstack environment and I need help with a Neutron problem. When I create a new network, the DHCP port launches with status inactive. Even though I tried to create a VM, the DHCP generate a port for the machine, However it takes a few minutes to give an error: "Build of instance ed6c8835-f2d4-4730-9638-74ded388361b aborted: Failed to allocate the network(s), not rescheduling." The only alternative solution to overcome this issue is to change the DHCP agent of the network. If I change it, the new DHCP port will have an active status and then I can launch a VM. Nothing conclusive was found in the logs, just the error to bind the port. It doesn't say anything to why the port is inactive, just that the port was created and with status active. Things that I already tried: -> See nova and neutron logs -> Restart neutron-server, neutron-dhcp-agent, neutron-openvswitch-agent -> Change manually the DHCP agent Anyone had the same problem as this one? Any tips to troubleshoot? Any help is welcome! Thanks.

2 1

Dalmatian: Dimensions of instance console
by berndbausch＠gmail.com 16 Dec '24

16 Dec '24

This is a Dalmatian-based Devstack. The instance console's height is too large for my screen. Documentation claims that console dimensions can be configured in a file named _detail_vnc.html [1], but there is no file with this name anywhere in the filesystem. I just submitted a bug [2] but wonder if somebody can help me out right now. How can the console dimensions be configured? Thanks! Bernd Bausch Instructor and tinkerer Here is my bug description: Under Frequently Asked Questions (https://docs.openstack.org/nova/2024.2/admin/remote-console-access.html#fre quently-asked-questions), the answer to Q: How do I adjust the dimensions of the VNC window image in the OpenStack dashboard? is incorrect. My Devstack installation contains no file named _detail_vnc.html. I found a file with the interesting name _detail_console.html under /opt/stack/horizon/openstack_dashboard/dashboards/project/instances/template s/instances, but it doesn't contain the answer's code for width and height. [1] https://docs.openstack.org/nova/2024.2/admin/remote-console-access.html#freq uently-asked-questions [2] https://docs.openstack.org/nova/2024.2/admin/remote-console-access.html#freq uently-asked-questions

1 0

[publiccloud-sig] Reminder: next meeting December 18th - 0800 UTC
by Felix Kronlage-Dammers 16 Dec '24

16 Dec '24

Hi everyone, This upcoming Wednesday (December 18th) the next meeting of the publiccloud-sig is going to happen. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> See ya in our pre-christmas special edition of the public cloud SIG :) felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-15)
by Goutham Pacha Ravi 15 Dec '24

15 Dec '24

Hello Stackers, We're 15 weeks away from the 2025.1 ("Epoxy") release date (2024-04-02) [1]. During the last week, the Technical Committee continued discussing the state of "unmaintained" code branches across OpenStack repositories and election officials proposed dates for the combined TC+PTL elections for the next release cycle [2]. We have decided to cancel the weekly meetings previously scheduled for 24th December and 31st December in lieu of the holidays. === Weekly Meeting === The OpenStack Technical Committee met over IRC on 2024-12-10 [3]. We revisited challenges with using CentOS 10 in the community CI system due to lack of AVX/AVX2 CPU flags on current cloud providers. Discussions revolved around finding providers with suitable hardware or reducing reliance on CentOS testing. We then revisited the state of "unmaintained" branches across OpenStack repositories. The oldest "unmaintained" branch tracks the "victoria" release (GA: 2020-10-14). The consensus amongst TC members was to enforce the policy to default old branches to EOL unless explicitly opted-in with required maintenance assurances. To do this, we need tooling to automate the transition, and a way to track and validate volunteer opt-ins for unmaintained branches. Volunteers stepped up to transition branches up until the 2023.1 ("Antelope") release to "end-of-life", beginning with the Victoria release branch [4]. If you're interested in keeping a particular branch open, for any repository, please express your interest directly on the gerrit change. All EOL transition patches will be merged after 1 month of community review. The next meeting of the OpenStack Technical Committee is on 2024-12-17 at 1800 UTC. This meeting will take place in the #openstack-tc channel on OFTC's IRC server. Please find the meeting agenda on the wiki [5]. Please remember that you can propose topics for upcoming TC meetings by directly editing the wiki. If you're unable to do so, please holler at me (IRC nick: gouthamr) or in the TC's IRC channel (#openstack-tc). === Governance Proposals === ==== Merged ==== - Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 ==== Open for Review ==== - Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 - Rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 - Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 - Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 - Allow more than 2 weeks for elections | https://review.opendev.org/c/openstack/governance/+/937741 === How to Contact the TC === You can reach the TC in several ways: - Email: send an email with the tag [tc] on this mailing list. - IRC: Ping us using the tc-members keyword on the #openstack-tc IRC channel on OFTC. - Weekly Meeting: The Technical Committee meets every week on Tuesdays at 1800 UTC [5]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] 2025.2/"F" elections proposal: https://review.opendev.org/c/openstack/election/+/937408 [3] TC Meeting IRC Log 2024-12-10: https://meetings.opendev.org/meetings/tc/2024/tc.2024-12-10-18.01.log.html [4] Transition unmaintained/victoria to EOL: https://review.opendev.org/c/openstack/releases/+/937515 [5] TC Meeting Agenda, 2024-12-17: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[openstack][octavia] How clients can debug their LB
by Nguyễn Hữu Khôi 15 Dec '24

15 Dec '24

Hello. I would like to know how clients can debug their LB such as access log or monitor LB performance? Thank you. Regards Nguyen Huu Khoi

3 4

[nova] Next 3 meetings are CANCELLED
by Sylvain Bauza 15 Dec '24

15 Dec '24

Hi folks, Apologies for this email, but I was needing to travel this week so I can't be around for this week's meeting and then I will be on PTO for Dec 24 and 31. Thanks, Sylvain

1 0

Diagnosing Sluggish Instance Boot Performance in OpenStack Cluster
by Abhijit Singh Anand 14 Dec '24

14 Dec '24

Dear All, Problem Statement: Instances remain stuck in the “Block Device Mapping” state for an extended period (~9 minutes) before eventually booting. This issue is observed consistently in one of my OpenStack clusters, while another cluster built on similar hardware and configuration spins up instances much faster. Environment Details: 1. Cluster Configuration: * Deployed using Kolla-Ansible on similar hardware specifications. * Both clusters connect to a common Ceph SSD-backed storage cluster. * Connection: 20 Gbps bonded interface over Cisco Nexus switches. * Same volumes used for images, VMs, backups. * Ceph keyrings and configurations are identical. 2. Performance Comparison: * “Good” cluster: Instances (even with large images >20 GB) boot to a login prompt in ~1-2 minutes. * “Bad” cluster: Instances with the same images and size take ~8-9 minutes. 3. Storage Backend Architecture: * “Good” cluster: Cinder-volume service is co-resident on compute nodes. * “Bad” cluster: Cinder-volume service runs only on controller nodes. 4. Instance Images: * Performance issue occurs with both RAW and QCOW2 images. * Example: A Windows 10 Pro RAW image boots in 1 min 40 sec in the "Good" cluster and takes 8 min 45 sec in the "Bad" cluster. What I’ve Tried So Far: * Configuration Matching: Reviewed and aligned as many configuration parameters as possible between the two clusters. * Resource Monitoring: Verified CPU and memory usage on nodes during instance launches—no significant resource contention observed. * Ceph Performance: Checked Ceph cluster performance; it seems healthy with no latency or throughput issues. * Network Testing: Verified 20 Gbps bond interfaces on both clusters; no observed bottlenecks. Hypothesis: I suspect the cinder-volume architecture could be contributing to the issue. In the "Good" cluster, cinder-volume is co-located with compute nodes, possibly optimizing data transfer paths. In contrast, the "Bad" cluster relies solely on controller-hosted cinder-volume services, which might introduce delays in data availability to compute nodes. Request for Community Input: 1. Does the difference in the placement of the cinder-volume service (co-resident vs. controller-only) align with the observed performance discrepancy? 2. Could there be other subtle configuration or architectural nuances that might explain the delays in the "Bad" cluster? 3. What specific logs or metrics would be most insightful in diagnosing this issue further? * I’ve reviewed Nova and Cinder logs but could dive deeper if pointed in the right direction. 4. Are there recommended best practices for tuning Cinder, Nova, or Ceph for environments with high storage performance requirements? Thank You! I appreciate any insights, suggestions, or pointers you can provide to help narrow down the root cause of this frustrating performance discrepancy. Regards, Abhijit S Anand

2 2

[manila] Epoxy Bugsquash: Dec 17th to 19th
by Carlos Silva 13 Dec '24

13 Dec '24

Hello zorillas and interested stackers, The first bugsquash of this cycle is right around the corner! Our bug backlog has been increasing for a while and we are seeing a trend where bugs are staying in progress for some time. For that reason, we will focus on reviews and following up on changes for the next week's bugsquash. Vida has kindly helped us put a list of bugs together [1]. We will be meeting at 15 UTC on Tuesday using meetpad [2], as agreed and we will use the next week's weekly meeting as a checkpoint for the bugsquash, where we'll discuss the progress of the bugs. [1] https://ethercalc.net/a4lehhy9ilp5 [2] https://meetpad.opendev.org/manila-epoxy-bugsquash I'm looking forward to it! carloss

1 0

[release] Release countdown for week R-15
by Thierry Carrez 13 Dec '24

13 Dec '24

Development Focus ----------------- The Epoxy-2 milestone will happen next month, on January 9th, 2025. 2025.1 "Epoxy"-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/epoxy/schedule.html for details. General Information ------------------- Please remember that libraries need to be released at least once per milestone period. At milestone 2, the release team will propose releases for any library that has not been otherwise released since milestone 1. Other non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. At milestone-2 we also freeze the contents of the final release. If you have a new deliverable that should be included in the final release, you should make sure it has a deliverable file in: https://opendev.org/openstack/releases/src/branch/master/deliverables/epoxy You should request a beta release (or intermediary release) for those new deliverables by milestone-2. We understand some may not be quite ready for a full release yet, but if you have something minimally viable to get released it would be good to do a 0.x release to exercise the release tooling for your deliverables. See the MembershipFreeze description for more details: https://releases.openstack.org/epoxy/schedule.html#e-mf Finally, now may be a good time for teams to check on any stable releases that need to be done for your deliverables. If you have bugfixes that have been backported, but no stable release getting those. If you are unsure what is out there committed but not released in the openstack/releases repo, running the command "tools/list_stable_unreleased_changes.sh <cycle_name>" gives a nice report. Upcoming Deadlines & Dates -------------------------- Epoxy-2 Milestone: January 9th, 2025 2025.1 Epoxy final release: April 2nd, 2025 -- Thierry Carrez (ttx)

1 0

[Kayobe] kolla_upper_constraints_file empty and getting openstacksdk version incompatability error
by Ali Hussain 13 Dec '24

13 Dec '24

Hi Maksim and Stackers, Trying out Kayobe, seed and control host are ready but upon executing deploy command " kayobe seed service deploy". The following task failed. Looks like "kolla_upper_constraints_file" is empty. What would be the value for it? TASK [ironic-inspector-rules : Ensure introspection rules exist] ************************************************************************************************************ failed: [seed] (item={'description': 'Set IPMI driver_info if no credentials', 'conditions': [{'field': 'node://driver_info.ipmi_username', 'op': 'is-empty'}, {'field': 'node://driver_info.ipmi_password', 'op': 'is-empty'}], 'actions': [{'action': 'set-attribute', 'path': 'driver_info/ipmi_username', 'value': ''}, {'action': 'set-attribute', 'path': 'driver_info/ipmi_password', 'value': ''}]}) => {"ansible_loop_var": "item", "changed": false, "item": {"actions": [{"action": "set-attribute", "path": "driver_info/ipmi_username", "value": ""}, {"action": "set-attribute", "path": "driver_info/ipmi_password", "value": ""}], "conditions": [{"field": "node://driver_info.ipmi_username", "op": "is-empty"}, {"field": "node://driver_info.ipmi_password", "op": "is-empty"}], "description": "Set IPMI driver_info if no credentials"}, "msg": "Incompatible openstacksdk library found: Version MUST be >=0.36 and <=0.98.999, but 1.0.2 is larger than maximum version 0.98.999."} Any suggestions? Thank you Best,

2 3

[cinder][ceph rbd driver] create volume in EC pool by download, but not clone
by Tony Liu 13 Dec '24

13 Dec '24

Hi, With Antelope, when create a volume in EC pool from image, Cinder downloads the image and upload it to volume, instead of cloning it as create volume in replicated pool. May I know the intention behind it? Why not clone it? Also, in case the intention is to remove dependency, clone and flatten works better. Thanks! Tony

2 7

Fwd: Regarding Trove-Openstack error
by skumari 13 Dec '24

13 Dec '24

---------- Original Message ---------- From: skumari <skumari(a)cdac.in> To: jimmy(a)openstack.org, communitymngr(a)openstack.org Date: December 9, 2024 at 10:03 AM Subject: Regarding Trove-Openstack error Hi Sir, In trove-openstack while iam trying to create a database instance iam getting this error: Polling request timed out. | | fault_date | 2024-12-06T10:45:59 | | fault_details | Traceback (most recent call last): | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/trove/common/utils.py", line 208, in wait_for_task | | | return polling_task.wait() | | | ^^^^^^^^^^^^^^^^^^^ | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/eventlet/event.py", line 124, in wait | | | result = hub.switch() | | | ^^^^^^^^^^^^ | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/eventlet/hubs/hub.py", line 310, in switch | | | return self.greenlet.switch() | | | ^^^^^^^^^^^^^^^^^^^^^^ | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_service/loopingcall.py", line 154, in _run_loop | | | idle = idle_for_func(result, self._elapsed(watch)) | | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_service/loopingcall.py", line 349, in _idle_for | | | raise LoopingCallTimeOut( | | | oslo_service.loopingcall.LoopingCallTimeOut: | | | Looping call timed out after 1808.67 seconds | | | | | | During handling of the above exception, another exception occurred: | | | | | | Traceback (most recent call last): | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/trove/taskmanager/models.py", line 447, in wait_for_instance | | | utils.poll_until(self._service_is_active, | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/trove/common/utils.py", line 224, in poll_until | | | return wait_for_task(task) | | | ^^^^^^^^^^^^^^^^^^^ | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/trove/common/utils.py", line 210, in wait_for_task | | | raise exception.PollTimeOut | | | trove.common.exception.PollTimeOut: Polling request timed out. I even increased poll request timeout also . can you please help me to solve the error Thankyou Kumari. ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

4 4

[nova][tc] Removing md5 in password injection
by Thomas Goirand 13 Dec '24

13 Dec '24

Hi, I didn't feel it would be controversial, though it seems removing md5 password injection is still up to debate: https://review.opendev.org/c/openstack/nova/+/935512 Of course, I'd like the TC to agree with me that injecting md5-hashed passwords is, in 2024, to be considered a security problem that should be fixed (and backported) ASAP. BTW, IMO this patch could be using the new feature from oslo_utils.secretutils that Takashi managed to get in: https://review.opendev.org/c/openstack/oslo.utils/+/931899 https://review.opendev.org/c/openstack/oslo.utils/+/935525 These, IMO, should also be backported to earlier oslo.utils releases, so we can fix earlier OpenStack releases in a nicer way. Cheers, Thomas Goirand (zigo)

6 14

[blazar] Next IRC meeting cancelled
by Pierre Riteau 12 Dec '24

12 Dec '24

Hello, Due to the holiday season, the next IRC meeting scheduled for December 26 is cancelled. We will meet again on January 9. I wish you all happy holidays. Cheers, Pierre Riteau (priteau)

1 0