December 2024 - openstack-discuss

[nova] Next 3 meetings are CANCELLED
by Sylvain Bauza 15 Dec '24

15 Dec '24

Hi folks, Apologies for this email, but I was needing to travel this week so I can't be around for this week's meeting and then I will be on PTO for Dec 24 and 31. Thanks, Sylvain

1 0

Diagnosing Sluggish Instance Boot Performance in OpenStack Cluster
by Abhijit Singh Anand 14 Dec '24

14 Dec '24

Dear All, Problem Statement: Instances remain stuck in the “Block Device Mapping” state for an extended period (~9 minutes) before eventually booting. This issue is observed consistently in one of my OpenStack clusters, while another cluster built on similar hardware and configuration spins up instances much faster. Environment Details: 1. Cluster Configuration: * Deployed using Kolla-Ansible on similar hardware specifications. * Both clusters connect to a common Ceph SSD-backed storage cluster. * Connection: 20 Gbps bonded interface over Cisco Nexus switches. * Same volumes used for images, VMs, backups. * Ceph keyrings and configurations are identical. 2. Performance Comparison: * “Good” cluster: Instances (even with large images >20 GB) boot to a login prompt in ~1-2 minutes. * “Bad” cluster: Instances with the same images and size take ~8-9 minutes. 3. Storage Backend Architecture: * “Good” cluster: Cinder-volume service is co-resident on compute nodes. * “Bad” cluster: Cinder-volume service runs only on controller nodes. 4. Instance Images: * Performance issue occurs with both RAW and QCOW2 images. * Example: A Windows 10 Pro RAW image boots in 1 min 40 sec in the "Good" cluster and takes 8 min 45 sec in the "Bad" cluster. What I’ve Tried So Far: * Configuration Matching: Reviewed and aligned as many configuration parameters as possible between the two clusters. * Resource Monitoring: Verified CPU and memory usage on nodes during instance launches—no significant resource contention observed. * Ceph Performance: Checked Ceph cluster performance; it seems healthy with no latency or throughput issues. * Network Testing: Verified 20 Gbps bond interfaces on both clusters; no observed bottlenecks. Hypothesis: I suspect the cinder-volume architecture could be contributing to the issue. In the "Good" cluster, cinder-volume is co-located with compute nodes, possibly optimizing data transfer paths. In contrast, the "Bad" cluster relies solely on controller-hosted cinder-volume services, which might introduce delays in data availability to compute nodes. Request for Community Input: 1. Does the difference in the placement of the cinder-volume service (co-resident vs. controller-only) align with the observed performance discrepancy? 2. Could there be other subtle configuration or architectural nuances that might explain the delays in the "Bad" cluster? 3. What specific logs or metrics would be most insightful in diagnosing this issue further? * I’ve reviewed Nova and Cinder logs but could dive deeper if pointed in the right direction. 4. Are there recommended best practices for tuning Cinder, Nova, or Ceph for environments with high storage performance requirements? Thank You! I appreciate any insights, suggestions, or pointers you can provide to help narrow down the root cause of this frustrating performance discrepancy. Regards, Abhijit S Anand

2 2

[manila] Epoxy Bugsquash: Dec 17th to 19th
by Carlos Silva 14 Dec '24

14 Dec '24

Hello zorillas and interested stackers, The first bugsquash of this cycle is right around the corner! Our bug backlog has been increasing for a while and we are seeing a trend where bugs are staying in progress for some time. For that reason, we will focus on reviews and following up on changes for the next week's bugsquash. Vida has kindly helped us put a list of bugs together [1]. We will be meeting at 15 UTC on Tuesday using meetpad [2], as agreed and we will use the next week's weekly meeting as a checkpoint for the bugsquash, where we'll discuss the progress of the bugs. [1] https://ethercalc.net/a4lehhy9ilp5 [2] https://meetpad.opendev.org/manila-epoxy-bugsquash I'm looking forward to it! carloss

1 0

[release] Release countdown for week R-15
by Thierry Carrez 13 Dec '24

13 Dec '24

Development Focus ----------------- The Epoxy-2 milestone will happen next month, on January 9th, 2025. 2025.1 "Epoxy"-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/epoxy/schedule.html for details. General Information ------------------- Please remember that libraries need to be released at least once per milestone period. At milestone 2, the release team will propose releases for any library that has not been otherwise released since milestone 1. Other non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. At milestone-2 we also freeze the contents of the final release. If you have a new deliverable that should be included in the final release, you should make sure it has a deliverable file in: https://opendev.org/openstack/releases/src/branch/master/deliverables/epoxy You should request a beta release (or intermediary release) for those new deliverables by milestone-2. We understand some may not be quite ready for a full release yet, but if you have something minimally viable to get released it would be good to do a 0.x release to exercise the release tooling for your deliverables. See the MembershipFreeze description for more details: https://releases.openstack.org/epoxy/schedule.html#e-mf Finally, now may be a good time for teams to check on any stable releases that need to be done for your deliverables. If you have bugfixes that have been backported, but no stable release getting those. If you are unsure what is out there committed but not released in the openstack/releases repo, running the command "tools/list_stable_unreleased_changes.sh <cycle_name>" gives a nice report. Upcoming Deadlines & Dates -------------------------- Epoxy-2 Milestone: January 9th, 2025 2025.1 Epoxy final release: April 2nd, 2025 -- Thierry Carrez (ttx)

1 0

[Kayobe] kolla_upper_constraints_file empty and getting openstacksdk version incompatability error
by Ali Hussain 13 Dec '24

13 Dec '24

Hi Maksim and Stackers, Trying out Kayobe, seed and control host are ready but upon executing deploy command " kayobe seed service deploy". The following task failed. Looks like "kolla_upper_constraints_file" is empty. What would be the value for it? TASK [ironic-inspector-rules : Ensure introspection rules exist] ************************************************************************************************************ failed: [seed] (item={'description': 'Set IPMI driver_info if no credentials', 'conditions': [{'field': 'node://driver_info.ipmi_username', 'op': 'is-empty'}, {'field': 'node://driver_info.ipmi_password', 'op': 'is-empty'}], 'actions': [{'action': 'set-attribute', 'path': 'driver_info/ipmi_username', 'value': ''}, {'action': 'set-attribute', 'path': 'driver_info/ipmi_password', 'value': ''}]}) => {"ansible_loop_var": "item", "changed": false, "item": {"actions": [{"action": "set-attribute", "path": "driver_info/ipmi_username", "value": ""}, {"action": "set-attribute", "path": "driver_info/ipmi_password", "value": ""}], "conditions": [{"field": "node://driver_info.ipmi_username", "op": "is-empty"}, {"field": "node://driver_info.ipmi_password", "op": "is-empty"}], "description": "Set IPMI driver_info if no credentials"}, "msg": "Incompatible openstacksdk library found: Version MUST be >=0.36 and <=0.98.999, but 1.0.2 is larger than maximum version 0.98.999."} Any suggestions? Thank you Best,

2 3

[cinder][ceph rbd driver] create volume in EC pool by download, but not clone
by Tony Liu 13 Dec '24

13 Dec '24

Hi, With Antelope, when create a volume in EC pool from image, Cinder downloads the image and upload it to volume, instead of cloning it as create volume in replicated pool. May I know the intention behind it? Why not clone it? Also, in case the intention is to remove dependency, clone and flatten works better. Thanks! Tony

2 7

Fwd: Regarding Trove-Openstack error
by skumari 13 Dec '24

13 Dec '24

---------- Original Message ---------- From: skumari <skumari(a)cdac.in> To: jimmy(a)openstack.org, communitymngr(a)openstack.org Date: December 9, 2024 at 10:03 AM Subject: Regarding Trove-Openstack error Hi Sir, In trove-openstack while iam trying to create a database instance iam getting this error: Polling request timed out. | | fault_date | 2024-12-06T10:45:59 | | fault_details | Traceback (most recent call last): | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/trove/common/utils.py", line 208, in wait_for_task | | | return polling_task.wait() | | | ^^^^^^^^^^^^^^^^^^^ | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/eventlet/event.py", line 124, in wait | | | result = hub.switch() | | | ^^^^^^^^^^^^ | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/eventlet/hubs/hub.py", line 310, in switch | | | return self.greenlet.switch() | | | ^^^^^^^^^^^^^^^^^^^^^^ | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_service/loopingcall.py", line 154, in _run_loop | | | idle = idle_for_func(result, self._elapsed(watch)) | | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_service/loopingcall.py", line 349, in _idle_for | | | raise LoopingCallTimeOut( | | | oslo_service.loopingcall.LoopingCallTimeOut: | | | Looping call timed out after 1808.67 seconds | | | | | | During handling of the above exception, another exception occurred: | | | | | | Traceback (most recent call last): | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/trove/taskmanager/models.py", line 447, in wait_for_instance | | | utils.poll_until(self._service_is_active, | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/trove/common/utils.py", line 224, in poll_until | | | return wait_for_task(task) | | | ^^^^^^^^^^^^^^^^^^^ | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/trove/common/utils.py", line 210, in wait_for_task | | | raise exception.PollTimeOut | | | trove.common.exception.PollTimeOut: Polling request timed out. I even increased poll request timeout also . can you please help me to solve the error Thankyou Kumari. ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

4 4

[nova][tc] Removing md5 in password injection
by Thomas Goirand 13 Dec '24

13 Dec '24

Hi, I didn't feel it would be controversial, though it seems removing md5 password injection is still up to debate: https://review.opendev.org/c/openstack/nova/+/935512 Of course, I'd like the TC to agree with me that injecting md5-hashed passwords is, in 2024, to be considered a security problem that should be fixed (and backported) ASAP. BTW, IMO this patch could be using the new feature from oslo_utils.secretutils that Takashi managed to get in: https://review.opendev.org/c/openstack/oslo.utils/+/931899 https://review.opendev.org/c/openstack/oslo.utils/+/935525 These, IMO, should also be backported to earlier oslo.utils releases, so we can fix earlier OpenStack releases in a nicer way. Cheers, Thomas Goirand (zigo)

6 14

[blazar] Next IRC meeting cancelled
by Pierre Riteau 12 Dec '24

12 Dec '24

Hello, Due to the holiday season, the next IRC meeting scheduled for December 26 is cancelled. We will meet again on January 9. I wish you all happy holidays. Cheers, Pierre Riteau (priteau)

1 0

[watcher] Canceling Dec 26th and Jan 2nd weekly meetings
by Douglas Viroel 12 Dec '24

12 Dec '24

Hi all o/ Since most of us will be out during the holidays, we agreed[1] on canceling watcher's weekly meetings[2] on Dec 26th and Jan 2nd. The last meeting of 2024 will be next week, on Dec 19th, and the first one of 2025 will be on Jan 9th. BR, [1] https://meetings.opendev.org/meetings/watcher/2024/watcher.2024-12-12-12.00… -- Douglas Viroel - dviroel

1 0