January 2025 - openstack-discuss - lists.openstack.org

[security-sig][ops][neutron] OVN Security issue can impact OpenStack users
by Jay Faulkner 22 Jan '25

22 Jan '25

Hi all, A recent security issue with OVN recently came to light and was fixed in a recent release. Full details are here: https://www.openwall.com/lists/oss-security/2025/01/22/5 . OpenStack deployers running OVN are strongly recommended to upgrade to the new fixed version. No CVE number has been issued yet for this resolved vulnerability. As a note, this was originally reported to the VMT as a neutron bug, but after triage it was responsibly reported to OVN and has been held under embargo. Thank you to the folks who participated in this triage process, including the OVN community for fixing the issue. Please see https://bugs.launchpad.net/ossa/+bug/2088280 for full details on OpenStack handling of this security report. Thanks, Jay Faulkner OpenStack VMT

1 0

[openstack-ansible]: GPU passthrough
by Rambo Rambo 22 Jan '25

22 Jan '25

Hi Experts Release: Wallaby Distribution: Openstack-ansible I am working on enabling GPU computes through PCIpassthrough: I managed to test the GPU PCI passthrough with the direct configuration below. *GPU compute:* /etc/nova/nova.conf [pci] device_spec = { "vendor_id": "10de", "product_id": "25b6" } passthrough_whitelist = { "vendor_id": "10de", "product_id": "25b6" } alias = { "vendor_id": "10de", "product_id": "25b6", "device_type": "type-PCI", "name": "nvidia" } *Nova API container:* /etc/nova/nova.conf enabled_filters = AvailabilityZoneFilter,ComputeFilter,AggregateNumInstancesFilter,AggregateIoOpsFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,NUMATopologyFilter, *PciPassthroughFilter* [pci] alias = { "vendor_id": "10de", "product_id": "25b6", "device_type": "type-PF", "name": "nvidia" } Now I need to put the configuration through openstack-ansible. So I need support for the same. *compute:* I intend to use hostvar for the compute configuration: /etc/openstack_deploy/host_vars/compute-<>.yml Can someone please confirm the parameters I need to set in this file and how to implement it? device_spec = ? passthrough_whitelist = ? nova_pci_alias: - '{ "name": "nvidia", "product_id": "25b6", "vendor_id": "10de" }' =====> how can I add *device_type field* here as per the mail.yaml it only have name,product_id and vendor_id field? Run openstack-ansible os-nova-install.yml --tags nova-config --limit <compute name> *For nova_api configuration,* I intend to use below: Below parameter in the user_variable file nova_scheduler_default_filters: - AvailabilityZoneFilter - ComputeFilter - AggregateInstanceExtraSpecsFilter - AggregateNumInstancesFilter - AggregateIoOpsFilter - ComputeCapabilitiesFilter - ImagePropertiesFilter - ServerGroupAntiAffinityFilter - ServerGroupAffinityFilter - NUMATopologyFilter - PciPassthroughFilter nova_pci_alias: - '{ "name": "nvidia", "product_id": "25b6", "vendor_id": "10de" }' =====> how can I add *device_type field here* as per the mail.yaml it only have name,product_id and vendor_id field? Execute: openstack-ansible os-nova-install.yml --tags nova-config --limit <nova api containers> Regards

2 2

[skyline-console] Issue with Skyline Console in Dalmatian Version of OpenStack
by Shubham Kumar Yadav 22 Jan '25

22 Jan '25

Dear Support Team, I hope this message finds you well. I am reaching out to report an issue I am experiencing with the Skyline console in the Dalmatian version of OpenStack. Issue Description: I am unable to list volumes and instances, as well as access volume-related services like Manila. This functionality is critical for my operations. Version Details: I have successfully set up the Skyline console for the Zed version of OpenStack, where everything is functioning as expected. However, the same cannot be said for the Dalmatian version. Troubleshooting Steps Taken: I have checked the service status and confirmed that all necessary services are running. I have attempted to refresh the console and clear the cache, but the issue persists. Request for Assistance: I would appreciate any guidance or troubleshooting steps you could provide to resolve this issue. If there are any logs or additional information you need from my end, please let me know. Also im using same image for both the openstack version - *99cloud/skyline:latest * These are the versions of nova-api & cinder-api *Zed version nova-manage version 26.2.2 cinder-manage version list 21.1.0* *Dalmation versionnova-manage version30.0.1cinder-manage version list* *25.0.0* Thank you for your attention to this matter. I look forward to your prompt response. Best regards, Shubham

1 0

reg trove - issues in deployment
by Rajendar K 22 Jan '25

22 Jan '25

Dear All, I have the following issues while trying trove service. (i) I have installed using kolla ansible and my trove container version is (trove-conductor:2024.1-debian-bookworm) and all my service is running in healthy state. (ii) trove-taskmanger.log as follows 2025-01-22 10:41:54.605 7 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib/python3.11/site-packages/trove/common/utils.py", line 195, in poll_and_check 2025-01-22 10:41:54.605 7 ERROR oslo.service.loopingcall obj = retriever() 2025-01-22 10:41:54.605 7 ERROR oslo.service.loopingcall ^^^^^^^^^^^ 2025-01-22 10:41:54.605 7 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib/python3.11/site-packages/trove/taskmanager/models.py", line 839, in _service_is_active 2025-01-22 10:41:54.605 7 ERROR oslo.service.loopingcall raise TroveError(_("Service not active, status: %s") % status) 2025-01-22 10:41:54.605 7 ERROR oslo.service.loopingcall trove.common.exception.TroveError: Service not active, status: failed to spawn 2025-01-22 10:41:54.605 7 ERROR oslo.service.loopingcall 2025-01-22 10:41:54.624 7 ERROR trove.taskmanager.models [None req-9aede034-a46c-47ec-a149-d7e164fa82e2 cad362f5da0d4f5b8aa87b0650379ed7 4e4b5450f3514da4af3c282d510d9f71 - - - -] Failed to create instance b53d9ebb-2fc6-4927-91c2-f82379b98cfa, error: Service not active, status: failed to spawn.: trove.common.exception.TroveError: Service not active, status: failed to spawn 2025-01-22 10:41:54.636 7 ERROR trove.taskmanager.models [None req-9aede034-a46c-47ec-a149-d7e164fa82e2 cad362f5da0d4f5b8aa87b0650379ed7 4e4b5450f3514da4af3c282d510d9f71 - - - -] Service status: ERROR, service error description: guestagent error: trove.common.exception.TroveError: Service not active, status: failed to spawn 2025-01-22 10:41:54.645 7 ERROR trove.taskmanager.models [None req-9aede034-a46c-47ec-a149-d7e164fa82e2 cad362f5da0d4f5b8aa87b0650379ed7 4e4b5450f3514da4af3c282d510d9f71 - - - -] Trove instance status: ERROR, Trove instance status description: Build error: guestagent timeout.: trove.common.exception.TroveError: Service not active, status: failed to spawn (iii) Does docker required for the configuration of DB or not need in case I use the image trove-master-guest-ubuntu-jammy.qcow2 directly Kindly help me in this regard, thanks Raj

1 0

[magnum] Proposing Jakub Darmach as core reviewer
by Michał Nasiadka 22 Jan '25

22 Jan '25

Hello, I’d like to propose Jakub Darmach (jakub(a)stackhpc.com) as a core reviewer. He has spent quite a good deal of time to get unmaintained branches CI in a green state. He has also contributed some Heat driver patches in the past and is actively reviewing patches (also going the extra mile of updating these patches to be mergable when the author is not available/responsive). Michał Nasiadka mnasiadka(a)gmail.com

3 3

[telemetry] Stepping down from core team
by Yadnesh Kulkarni 22 Jan '25

22 Jan '25

Dear Team, Due to my role change I will no longer be able to contribute to the Telemetry projects. I’ve decided to step down from the core team. Working on the Telemetry projects has been a truly rewarding and enriching learning experience. I’ve thoroughly enjoyed being part of this incredible journey and have gained a lot from collaborating with such talented and supportive individuals. Thank you all for your guidance, collaboration, and support throughout this time. Wishing the team continued success and growth! -- Yadnesh Kulkarni

2 1

[tc][freezer][watcher][release][requirement][oslo] projects need to refresh the DPL model for the next cycle
by Ghanshyam Mann 21 Jan '25

21 Jan '25

Hi All, As you know, we have five projects (listed below) in the DPL model. As per process[1], we need to reset the DPL model every cycle. The project team either needs to opt-in to the DPL model (refer to the document for exact steps[1]) or plan for the PTL election. 1. Freezer: https://review.opendev.org/c/openstack/governance/+/939484 2. Watcher: https://review.opendev.org/c/openstack/governance/+/939488 3. Release: https://review.opendev.org/c/openstack/governance/+/939486 4. Requirement: https://review.opendev.org/c/openstack/governance/+/939487 5. Oslo: https://review.opendev.org/c/openstack/governance/+/939485 [1] https://governance.openstack.org/tc/reference/distributed-project-leadershi… -gmann

1 1

[nova] Seeking Advice on Adding Multiple CPU Models in One Aggregate for OpenStack
by mina.mohammadi367＠gmail.com 21 Jan '25

21 Jan '25

Hello friends, I am currently working on a deployment scenario in OpenStack where I need to group multiple CPU models within a single aggregate. I’m reaching out to the community to ask if anyone has experience with this type of setup or has any advice on best practices. I would greatly appreciate any input from the community regarding potential challenges or recommendations for managing multiple CPU models within a single aggregate. In particular, any advice on ensuring compatibility and optimizing instance scheduling based on traits would be extremely valuable. For reference, I am considering the solution outlined in the CPU Model Selection documentation: https://specs.openstack.org/openstack/nova-specs/specs/train/implemented/cp… Thank you in advance for your time and assistance.

1 0

Assistance with Configuring fdb_age_threshold in OVN with Kolla-Ansible
by Jan Wasilewski 21 Jan '25

21 Jan '25

Hi Stackers, Over the past few days, we encountered an issue during the migration of VMs from a customer environment to our OpenStack cloud. Both environments were connected via a cross-connect network (a VLAN-type network spanning the cloud and the customer environment), which resulted in duplicate MAC addresses in our OVN database. This occurred because the migrated VM retained the same MAC address as previously used in the customer's environment(so "ovn-sbctl list fdb" returned a port which was no-longer used and leads to such duplication) . To address the issue, we used the following command to clear the FDB database: ovn-sbctl --all destroy fdb While researching further, we came across the `fdb_age_threshold` configuration [1], which seemed to be a useful feature to prevent similar issues in the future. We attempted to implement it in our Kolla-Ansible environment by adding the following configuration to ` */etc/kolla/config/neutron/ml2_conf.ini*`: [ovn] fdb_age_threshold = 920 After deploying the changes, however, we noticed that the setting did not take effect in OVN. The configuration only applied successfully after we executed a manual script to propagate the setting directly to OVN [2]. Anyway such change is applicable only for exsisting networks, so it is far from perfect solution. Could you provide guidance on whether this behavior is due to incorrect documentation, an error in our configuration process, or an additional requirement needed to enable this feature? Any insights would be greatly appreciated. Thank you in advance for your support! /Jan Wasilewski **References:** [1] https://docs.openstack.org/neutron/latest/configuration/ml2-conf.html#ovn.f… [2] https://paste.openstack.org/show/b9xUcnG6GXmCdSbSReG0/

2 2

[openstack-ansible] issue
by Danish Khan 21 Jan '25

21 Jan '25

Dear All, Whenever I do any activity like Operating system upgrade of compute nodes or openstack major version update then the name of compute host gets updated. I have my compute node with name compute1,2,3.... but after the activity they sometimes become compute1.openstack.local or compute2.mgmt.local. Due to this I get duplicate resource providers. But only the latest provider works properly. and I had manually unset all the vm from old provider and added then into new provider. Is there a way I can avoid this? means I don't want my compute nodes to get renamed with domain. And is there a way I can unset a vm from all provider and heal them automatically instead adding them into new resource proivder? I tried below docs but the healing part is not working so I go with manual process for all vm which is too time consuming. https://docs.openstack.org/nova/yoga/admin/troubleshooting/orphaned-allocat… Regards, Danish Khan

3 2

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-10)
by Goutham Pacha Ravi 20 Jan '25

20 Jan '25

Hello Stackers, We're 10 weeks away from the 2025.1 coordinated "Epoxy" release [1]. Starting this week, there are several deadlines that project teams have adopted to ensure that code, specifications, and documentation changes are adequately peer-reviewed. As Slawek Kaplonski (slaweq) shared to this mailing list [2], the nomination period for the upcoming Technical Committee and Project Team Leads elections will begin on 2025-02-05 and remain open until 2025-02-19. Early nominations are highly encouraged in case you'll be unavailable during this period. In the last week, the OpenStack Technical Committee selected a community goal to migrate away from the eventlet library across its codebase [3]. As with many community goals, this will be a multi-release goal and needs contributors. Please join the #openstack-eventlet-removal IRC channel on OFTC to participate in the discussion. === Weekly Meeting === Dmitriy Rabotyagov (noonedeadpunk) chaired the TC's weekly IRC meeting on 2025-01-14 [4]. We discussed the maintenance of the "openstackdocstheme" Sphinx plugin and called out some early attempts to switch documentation to a stock theme that's more maintainable and sustainable. We also reviewed a proposal from Freezer project maintainers to remove the project team from the "inactive" teams list [5]. The TC also considered requesting an exception for Freezer to release within 2025.1. In this vein, we also highlighted the need for consistent criteria to move projects out of inactive status and better tooling for indicating project health and activity. The next meeting of the OpenStack Technical Committee is today, 2025-01-21. This meeting will be hosted via IRC on OFTC's #openstack-tc channel. Please find the meeting agenda on the wiki [6]. I hope you'll be able to join us there. === Governance Proposals === ==== Merged ==== - Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 - Rework the eventlet-removal goal proposal | https://review.opendev.org/c/openstack/governance/+/931254 - Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 ==== Open for Review ==== - Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 - Retire Freezer DR | https://review.opendev.org/c/openstack/governance/+/938183 - Retire qdrouterd role | https://review.opendev.org/c/openstack/governance/+/938193 - Remove Freezer from inactive state | https://review.opendev.org/c/openstack/governance/+/938938 - Reset the DPL model for oslo project | https://review.opendev.org/c/openstack/governance/+/939485 - Reset the DPL model for Release project | https://review.opendev.org/c/openstack/governance/+/939486 - Reset the DPL model for Requirement project | https://review.opendev.org/c/openstack/governance/+/939487 - Reset the DPL model for Watcher project | https://review.opendev.org/c/openstack/governance/+/939488 - Define 2025 upstream investment opportunities | https://review.opendev.org/c/openstack/governance/+/939507 === Upcoming Events === - 2025-01-28: OpenInfra Board Monthly Meeting: https://board.openinfra.org/ - 2025-02-01: FOSDEM 2025 (https://fosdem.org/2025/) OpenStack's 15th Birthday Celebration - 2025-02-05: Nominations open for 2025.2 TC/PTL Elections: https://governance.openstack.org/election/ - 2025-02-28: 2025.1 ("Epoxy") Feature Freeze and release milestone 3 [1] - 2025-03-06: SCALE 2025 + OpenInfra Days NA (https://www.socallinuxexpo.org/scale/22x) Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] TC/PTL Election dates for the 2025.2 cycle: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [3] Goal to Remove Eventlet from OpenStack: https://governance.openstack.org/tc/goals/selected/remove-eventlet.html [4] TC Meeting IRC Log 2025-01-14: https://meetings.opendev.org/meetings/tc/2025/tc.2025-01-14-18.00.log.html [5] Remove Freezer from inactive state: https://review.opendev.org/c/openstack/governance/+/938938 [6] TC Meeting Agenda, 2025-01-21: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[Horizon] Need Help - Error with Compress
by info＠patrick-omland.de 20 Jan '25

20 Jan '25

I need help with Horizon; when I run the following command ./manage.py compress, I get the following error: (venv) root@fsn1-controller01 /var/lib/horizon # ./manage.py compress /var/lib/horizon/venv/lib/python3.11/site-packages/debreach/__init__.py:6: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. version_info = version.StrictVersion(__version__).version CommandError: An error occurred during rendering _stylesheets.html: Error: File to import not found or unreadable: /horizon/lib/font_awesome/scss/font-awesome.scss. on line 12:1 of dashboard/scss/horizon.scss from line 6:9 of stdin >> @import "/horizon/lib/font_awesome/scss/font-awesome.scss"; ^ (venv) root@fsn1-controller01 /var/lib/horizon # ll total 124 -rw-r--r-- 1 root root 77 Jan 20 19:13 babel-django.cfg -rw-r--r-- 1 root root 258 Jan 20 19:13 babel-djangojs.cfg -rw-r--r-- 1 root root 233 Jan 20 19:13 bindep.txt -rw-r--r-- 1 root root 600 Jan 20 19:13 CONTRIBUTING.rst drwxr-xr-x 3 root root 4096 Jan 20 19:13 doc -rw-r--r-- 1 root root 579 Jan 20 19:13 HACKING.rst drwxr-xr-x 19 root root 4096 Jan 20 19:15 horizon -rw-r--r-- 1 root root 10143 Jan 20 19:13 LICENSE -rwxr-xr-x 1 root root 831 Jan 20 19:13 manage.py -rw-r--r-- 1 root root 634 Jan 20 19:13 MANIFEST.in drwxr-xr-x 6 root root 4096 Jan 20 19:15 openstack_auth drwxr-xr-x 18 root root 4096 Jan 20 19:15 openstack_dashboard -rw-r--r-- 1 root root 1023 Jan 20 19:13 package.json drwxr-xr-x 8 root root 4096 Jan 20 19:13 playbooks -rw-r--r-- 1 root root 2188 Jan 20 19:13 plugin-registry.csv -rw-r--r-- 1 root root 1920 Jan 20 19:13 README.rst drwxr-xr-x 4 root root 4096 Jan 20 19:13 releasenotes -rw-r--r-- 1 root root 120 Jan 20 19:13 reno.yaml -rw-r--r-- 1 root root 2664 Jan 20 19:13 requirements.txt drwxr-xr-x 5 root root 4096 Jan 20 19:13 roles -rw-r--r-- 1 root root 1937 Jan 20 19:13 setup.cfg -rw-r--r-- 1 root root 698 Jan 20 19:13 setup.py drwxr-xr-x 9 root root 4096 Jan 20 19:15 static -rw-r--r-- 1 root root 379 Jan 20 19:13 test-requirements.txt -rw-r--r-- 1 root root 3198 Jan 20 19:13 test-shim.js drwxr-xr-x 2 root root 4096 Jan 20 19:13 tools -rw-r--r-- 1 root root 7537 Jan 20 19:13 tox.ini drwxr-xr-x 6 root root 4096 Jan 20 19:13 venv (venv) root@fsn1-controller01 /var/lib/horizon # I have installed Horizon in a virtual Python 3 environment (Debian Bookworm) and I also clone the Horizon Github Repo, then I installed all requirements using pip install -r requirements.txt. Anyone have an idea?

1 0

[Horizon] Need Help - Error with Compress
by info＠patrick-omland.de 20 Jan '25

20 Jan '25

I need help with Horizon; when I run the following command ./manage.py compress, I get the following error: (venv) root@fsn1-controller01 /var/lib/horizon # ./manage.py compress /var/lib/horizon/venv/lib/python3.11/site-packages/debreach/__init__.py:6: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. version_info = version.StrictVersion(__version__).version CommandError: An error occurred during rendering _stylesheets.html: Error: File to import not found or unreadable: /horizon/lib/font_awesome/scss/font-awesome.scss. on line 12:1 of dashboard/scss/horizon.scss from line 6:9 of stdin >> @import "/horizon/lib/font_awesome/scss/font-awesome.scss"; ^ (venv) root@fsn1-controller01 /var/lib/horizon # ll total 124 -rw-r--r-- 1 root root 77 Jan 20 19:13 babel-django.cfg -rw-r--r-- 1 root root 258 Jan 20 19:13 babel-djangojs.cfg -rw-r--r-- 1 root root 233 Jan 20 19:13 bindep.txt -rw-r--r-- 1 root root 600 Jan 20 19:13 CONTRIBUTING.rst drwxr-xr-x 3 root root 4096 Jan 20 19:13 doc -rw-r--r-- 1 root root 579 Jan 20 19:13 HACKING.rst drwxr-xr-x 19 root root 4096 Jan 20 19:15 horizon -rw-r--r-- 1 root root 10143 Jan 20 19:13 LICENSE -rwxr-xr-x 1 root root 831 Jan 20 19:13 manage.py -rw-r--r-- 1 root root 634 Jan 20 19:13 MANIFEST.in drwxr-xr-x 6 root root 4096 Jan 20 19:15 openstack_auth drwxr-xr-x 18 root root 4096 Jan 20 19:15 openstack_dashboard -rw-r--r-- 1 root root 1023 Jan 20 19:13 package.json drwxr-xr-x 8 root root 4096 Jan 20 19:13 playbooks -rw-r--r-- 1 root root 2188 Jan 20 19:13 plugin-registry.csv -rw-r--r-- 1 root root 1920 Jan 20 19:13 README.rst drwxr-xr-x 4 root root 4096 Jan 20 19:13 releasenotes -rw-r--r-- 1 root root 120 Jan 20 19:13 reno.yaml -rw-r--r-- 1 root root 2664 Jan 20 19:13 requirements.txt drwxr-xr-x 5 root root 4096 Jan 20 19:13 roles -rw-r--r-- 1 root root 1937 Jan 20 19:13 setup.cfg -rw-r--r-- 1 root root 698 Jan 20 19:13 setup.py drwxr-xr-x 9 root root 4096 Jan 20 19:15 static -rw-r--r-- 1 root root 379 Jan 20 19:13 test-requirements.txt -rw-r--r-- 1 root root 3198 Jan 20 19:13 test-shim.js drwxr-xr-x 2 root root 4096 Jan 20 19:13 tools -rw-r--r-- 1 root root 7537 Jan 20 19:13 tox.ini drwxr-xr-x 6 root root 4096 Jan 20 19:13 venv (venv) root@fsn1-controller01 /var/lib/horizon # I have installed Horizon in a virtual Python 3 environment (Debian Bookworm) and I also clone the Horizon Github Repo, then I installed all requirements using pip install -r requirements.txt. Anyone have an idea?

1 0

How to know the openstack client's default microversion?
by berndbausch＠gmail.com 20 Jan '25

20 Jan '25

Without the version option e.g. --os-compute-api-version, the openstack client submits API requests with certain default microversions. The precise microversions vary with the openstack client's version and the specific task. For example, the --debug option reveals that openstack client 7.1.3 uses Nova microversion 2.89 when attaching a volume to an instance. Is there a simpler way than --debug to find out which microversions the client uses? Is there a command, or perhaps a website where this is documented? Thanks for your ideas! Bernd.

2 1

[openstack-ansible] : libvirtd debug logs after Wallaby upgrade
by Rambo Rambo 20 Jan '25

20 Jan '25

Hi Team I have recently done Victoria to Wallaby upgrade on my openstack-ansible installation. After the upgrade we observe that /var/log/libvirt/ size on all the computes was growing due to libvirtd.log file. Further checking this file it can be seen that there are debug logs. 2025-01-19 11:15:20.817+0000: 2943565: *debug *: virNetlinkEventCallback:862 : dispatching to max 0 clients, called from event watch 7 2025-01-19 11:15:20.817+0000: 2943565:* debug* : virNetlinkEventCallback:876 : event not handled. 2025-01-19 11:15:20.821+0000: 2943605: *debug :* udevHandleOneDevice:1492 : udev action: 'add' ... libvirtd configuration file on compute shows log_level=1 cat libvirtd.conf # Ansible managed # enabling debug mode according to http://wiki.libvirt.org/page/DebugLogs log_level = 1 log_filters="3:remote 4:event 3:json 3:rpc" log_outputs="1:file:/var/log/libvirt/libvirtd.log" listen_tls = 0 listen_tcp = 0 unix_sock_group = "libvirt" unix_sock_ro_perms = "0777" unix_sock_rw_perms = "0770" auth_unix_ro = "none" auth_unix_rw = "none" auth_tcp = "sasl" No related parameter was changed/updated in user_variable file during the upgrade? Any information what could have enabled the debug logs for libvirtd on all the computes? Also how can I turn-off this using the openstack-ansible and what could be impact during the procedure? Regards

2 3

Openstack Ansible
by engineer2024 20 Jan '25

20 Jan '25

It seems that the ssh to containers from the deploy node, is removed in the caracal release by default in OSA. The new containers don't have the ssh service inside them. Is there a way to enable it via a variable ?

2 2

[neutron] Bug deputy week 3 2025 (2025-01-13 - 2025-01-19)
by Lajos Katona 20 Jan '25

20 Jan '25

Hi Neutrinos, The bug list for week 3 of 2025. * Critical ** https://bugs.launchpad.net/neutron/+bug/2094970 (rally ovn job timing out with eventlet 0.38.2) *** https://review.opendev.org/c/openstack/neutron/+/939315 Merged * High ** https://bugs.launchpad.net/neutron/+bug/2094736 ([OVN] Missing ``LogicalSwitchPortUpdateUpEvent`` when a port is bound) *** WIP: https://review.opendev.org/c/openstack/neutron/+/939345 ** https://bugs.launchpad.net/neutron/+bug/2094810 (neutron-status upgrade check fails on checking if ovn_alembic_version table exists) *** https://review.opendev.org/c/openstack/neutron/+/939248 ** https://bugs.launchpad.net/neutron/+bug/2094842 ("ovs.db.error.Error: ovsdb error: 0 values when type requires between 1 and 9223372036854775807" when router is attached to network without subnet ) *** Unassigned ** https://bugs.launchpad.net/neutron/+bug/2095000 (ovs-agent: Failed to store metadata for trunk ...: Cannot find Bridge with name=tbr-... ) Assigned ** https://bugs.launchpad.net/neutron/+bug/2095044 (test_ip_lib calls to sudo and fails (sometimes?)) *** https://review.opendev.org/c/openstack/neutron/+/939435 Merged ** https://bugs.launchpad.net/neutron/+bug/2095167 (BW limit not applied to external/provider network) *** https://review.opendev.org/c/x/whitebox-neutron-tempest-plugin/+/939539 ** https://bugs.launchpad.net/neutron/+bug/2095180 (test_agent_metadata_port_dhcp_reenable_event fails when ovsdb-server combines create and later update into a single create event for monitor) *** https://review.opendev.org/c/openstack/neutron/+/939542 ** https://bugs.launchpad.net/neutron/+bug/2095195 ( VM fails to receive IPv4 from DHCP agent because the agent has not updated the network config for dnsmasq after port was created) ** https://bugs.launchpad.net/neutron/+bug/2095196 (functional: test_virtual_port_host_update failed due to ovsdb SB transaction timeout, "Unreasonably long 25512ms poll interval (0ms user, 26ms system)" in ovsdb-server log) *** https://review.opendev.org/c/openstack/neutron/+/939558 ** https://bugs.launchpad.net/neutron/+bug/2095224 (Functional jobs randomly timed out, possible load issues on the host ) *** perhaps duplicate of bug/2095196 ( https://bugs.launchpad.net/neutron/+bug/209519) * Medium ** https://bugs.launchpad.net/neutron/+bug/2094846 (neutron.tests.functional.services.trunk.drivers.ovn.test_trunk_driver.TestOVNTrunkDriver.test_subport_delete fails because remove_subports fails to update OVN db due to revision number inconsistency) *** Unassigned ** https://bugs.launchpad.net/neutron/+bug/2095084 ([OVN] Error in ``_delete_router_remove_external_gateways``: "'NoneType' object has no attribute 'network_id'") *** https://review.opendev.org/c/openstack/neutron/+/939451 ** https://bugs.launchpad.net/neutron/+bug/2095097 (500 error creating resource with emoji for name) Unassigned ** https://bugs.launchpad.net/neutron/+bug/2095152 (ovs-agent: Leftover tpi/spi interfaces after VM boot/delete with trunk port(s)) Unassigned ** https://bugs.launchpad.net/neutron/+bug/2095185 test_legacy_router_conntrack_helper failed as "neutron_lib.exceptions.BridgeDoesNotExist: Bridge test-bre0e2da16 does not exist" when router attempted to plug interface *** Unassigned, Was seen on stable, but can happen on master * Low ** https://bugs.launchpad.net/neutron/+bug/2095091 (test_metadata_ipv6_only_network failed looking for QUERY_MSG) Unassigned Lajos Katona (lajoskatona)

1 0

[glance][ops] proposal to remove the glance-scrubber
by Brian Rosmaita 19 Jan '25

19 Jan '25

Hello Glance users, The Glance project team wants to determine whether anyone is using the glance-scrubber utility [0]. If not, we are thinking of deprecating it this cycle to be removed in the appropriate future cycle. If you have not previously heard of the glance-scrubber, that's most likely because it's only useful for particular deployment configurations of which yours is not one. See the doc [0] for details if you are curious. Our suspicion is that the glance-scrubber may have outlived its usefulness for modern OpenStack configurations. (The last time I remember it being discussed at a PTG was during the snowstorm in Dublin, which I believe was the Rocky development cycle). On the other hand, the team also has some ideas for improving the scrubber, but there's no point doing that if it's not actually being used in the first place. So if removal of the glance-scrubber would negatively affect you, please respond to this email so we can get some data points. Thanks! [0] https://docs.openstack.org/glance/latest/cli/glancescrubber.html

2 2

[i18n][infra][horizon] Zanata translation job failures after Python 3.8 deprecation
by Ian Y. Choi 19 Jan '25

19 Jan '25

Hi all, During last week's PTG, the I18n SIG identified Zanata translation job failures for Horizon (and likely related plugins). The root cause appears to be Python 3.8 deprecation on Horizon [1]. Failed jobs have been observed on Zuul [2], and translation proposals were made on Gerrit [3]. Since translation sync on Horizon stable branches is functioning correctly, the error message seems to be related to Python 3.8 deprecation: ERROR: Cannot install -r doc/requirements.txt (line 3) due to conflicting package version dependencies. To address this, I'd like to raise two points: 1/ To resolve the issue, would you support updating translation sync jobs with Bionic-based Zuul jobs by either upgrading to Jammy or modifying the Bionic image to support Python >3.8? 2/ At least me from I18n SIG acknowledges that recognizing these translation job errors was delayed. Moving forward, would it be beneficial to implement periodic (e.g., monthly) reporting on translation job successes and failures? Thank you, /Ian [1] https://docs.openstack.org/releasenotes/horizon/2024.2.html#upgrade-notes [2] https://zuul.opendev.org/t/openstack/builds?job_name=propose-translation-up… [3] https://review.opendev.org/q/owner:proposal-bot+project:openstack/horizon+b…

2 2

[magnum] GPU support with magnum
by Satish Patel 18 Jan '25

18 Jan '25

Folks, We are running magnum with openstack to spin up k8s clusters and everything works as expected. Now I want GPU support so we can do some AI/ML stuff with k8s. Does magnum support GPU and how does it work? In the past I used GPU in passthrough mode to expose hardware to VMs but not sure how it works for k8s and what are the challenges?

5 16

[neutron][upgrades] Linuxbridge driver removal and migration
by Brian Haley 17 Jan '25

17 Jan '25

Hi Stackers, In 2022, during the Zed cycle, the Neutron team took a step at marking some items as experimental. One of those was the Linuxbridge driver. The reason was due to it being unmaintained for a number of cycles, and no one was able to take on its ownership. The implication of this decision was that it was consider deprecated, even if not given an official warning message. In doing this, all gate jobs were moved into the experimental queue, feature-parity with other drivers was deemed unnecessary, and bug fixing was considered best-effort. Since then, both major distros have stopped supporting new deployments using Linuxbridge, and have been mainly focused on OVN. So what does this mean? We (the neutron cores) have come to the conclusion that it is time to take the next step and remove the code from the tree [1]. After 5 cycles of being considered experimental without any investment it just makes sense. We plan on doing this in the current Epoxy cycle. We realize that there are still deployments using Linuxbridge, and hopefully they have been looking at migration strategies since we originally marked the code experimental. This email is another push to the user community to start looking at this, especially if you are planning any new deployments. There has been at least one migration guide written at [2] (thanks Jim!), and at this point I would encourage those still using Linuxbridge to look at it and start asking any questions they have so we have the chance to make it better and can make that process easier. Thanks for reading, -Brian, Neutron PTL [0] https://review.opendev.org/c/openstack/neutron/+/845181 [1] https://review.opendev.org/c/openstack/neutron/+/927216 [2] https://www.jimmdenton.com/migrating-lxb-to-ovn/

11 14

openstack ssl
by engineer2024 17 Jan '25

17 Jan '25

The present model for openstack API services uses keystone for authentication mechanism for the REST API. is it possible to test the api calls directly using the server/port of a API service, bypassing the keystone URI ? does this require any modification in the configuration of the openstack service ? Thanks

1 0

[neutron] Using address scopes in a DVR environment breaks floating IPs
by Lukas Steiner 17 Jan '25

17 Jan '25

Hi all, I hope to get some advice here if we’re doing something wrong or ran into a bug. We have an OpenStack environment (wallaby) with Distributed Virtual Router (DVR) enabled (compute nodes use DVR_NO_EXTERNAL agent mode, network nodes use DVR_SNAT) and we would like to onboard our provider network to a subnet pool part of an address scope to allow routed self-service networks. But as soon as the provider network is part of an address scope, the floating IP feature stops working. After debugging we found that adding the network to an address scope results in an additional set of firewall rules in the router and snat network namespaces which blocks cross-scope traffic. But floating IPs should be the exception to that and there are indeed rules that explicitly allow them by setting a mark in the iptables mangle table: -A neutron-l3-agent-floatingip -d <floating-ip>/32 -i qg-<random-id> -j MARK --set-xmark 0x4000000/0xffff0000 We can find these rules in the qrouter namespaces, but they’re missing in the snat namespaces. If we add the rule manually to the snat namespace, the floating IP is working again. If we switch the compute nodes to DVR agent mode, it also works, as the snat namespace is not used. I have already searched for bugs and changes in the DVR router code since wallaby but can’t find anything which is not already patched in wallaby. So, I’m wondering if we have an unsupported setup with combining dynamic routing with DVR and centralized network nodes or facing a long running neutron bug. Maybe someone has a similar setup or can confirm that this use case should be supported? Regards, Lukas

1 0

Re: Dalmatian Devstack: Empty Port Binding section in Horizon
by Lajos Katona 17 Jan '25

17 Jan '25

Hi, I checked and I hoped that this SDK patch : https://review.opendev.org/c/openstack/openstacksdk/+/915902 or SDK 4.2.0 is enough (for binding details at least) but something else is missing. Could you please open a bug report for this issue, I try to check where to fix it. Lajos Katona (lajoskatona) <berndbausch(a)gmail.com> ezt írta (időpont: 2025. jan. 14., K, 13:25): > Thanks much, Lajos! > > > > It was installed by Devstack with git_clone https://opendev.org/openstack/horizon.git > /opt/stack/horizon stable/2024.2. Furthermore, under /opt/stack/horizon/ > horizon.egg-info I find a file named pbr.json, which contains > > > > {"git_version": "4bcc3dcea", "is_release": false} > > > > I hope that makes it clear what I am using! Or is there another way of > obtaining the Horizon release number? > > > > Of the list of patches you shared, the ones that seem relevant (SDK for > neutron ports, SDK for neutron networks) were merged in August and June, > respectively, which makes me think that should be included in my release. > > > > And since this is an HTML message now, I am adding two screenshots. Not > only port binding info, but also network provider info is missing. I can > confirm that this is the admin user. > > > > Kind regards, > > > > Bernd > > > > > > *From:* Lajos Katona <katonalala(a)gmail.com> > *Sent:* Tuesday, January 14, 2025 5:22 PM > *To:* berndbausch(a)gmail.com > *Cc:* openstack-discuss <openstack-discuss(a)lists.openstack.org> > *Subject:* Re: Dalmatian Devstack: Empty Port Binding section in Horizon > > > > Hi Bernd, > > Shall I ask which version of Horizon do you use? Recently some patches > were merged for changing to SDK in the background: > > > https://review.opendev.org/q/project:openstack/horizon+topic:%22bug/1999774… > > And to be sure if you hit an issue from that change series or something > else. > > > > Best wishes > > Lajos Katona (lajoskatona) > > > > <berndbausch(a)gmail.com> ezt írta (időpont: 2025. jan. 14., K, 4:43): > > Is there a known bug, either in Devstack or Horizon, that prevents Neutron > port binding details from being displayed? My googling did not find > anything. > > As admin, I open a port's details, and in the Binding section at the end of > the page, all I get is a bold text "VNIC Type". The CLI command "openstack > port show" does provide the details. Also on a Xena Devstack, Horizon > displays them[1]. Horizon's setup.py seems to be identical on both > releases, > as far as Neutron configuration is concerned. > > Thanks for any input. > > Bernd. > > [1] The Binding details I expect: > > Binding > > VNIC Type Normal > Host xena-202 > Profile None > VIF Type ovs > VIF Details > connectivity l2 > port_filter True > ovs_hybrid_plug False > datapath_type system > bridge_name br-int > >

2 1

[qa] QA office hours new time
by Ghanshyam Mann 16 Jan '25

16 Jan '25

Hi All, As informed by Martin[1], we are changing the QA office hours to Wed biweekly at 18.00 UTC. I have updated the meeting info[2] and wiki page[3]. This time might not suit many of you, so feel free to ping me on IRC if any topic/review needs urgent attention. [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [2] https://review.opendev.org/c/opendev/irc-meetings/+/939480 [3] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee -gmann

1 0

Adding a compute node using Openstack ansible
by Derek O keeffe 15 Jan '25

15 Jan '25

Hi all, I have something weird happening when trying to add a compute node to a working cluster. I added "compute3" to my openstack_user_config.yml compute_hosts stanza and ran the setup_hosts playbook. This created (re-created) the openstack_inventory.json file which I have reviewed thoroughly and I can see compute3 added everywhere compute1 and compute2 had been added from the deploy. When I run the second playbook setup_infrastructure, all the plays skip saying "no hosts matched" but the tasks all touch compute3.... I ran /opt/openstack-ansible/scripts/inventory-manage.py -G and I can see my compute3 listed in the output. The commands I run for the playbooks are as follows: openstack-ansible setup-hosts.yml --limit ,compute3openstack-ansible setup-infrastructure.yml --limit ,compute3openstack-ansible setup-openstack.yml --limit ,compute3 I've been trying to solve this since before Christmas with no luck, could anyone please shed some light on what the issue could be? Output of first playbook: https://paste.openstack.org/show/blArG04q6wgUCJy9JPh3/Output of second playbook: Paste #bSiabwqOk7qdns2iBYDA | LodgeIt! | | | | Paste #bSiabwqOk7qdns2iBYDA | LodgeIt! | | | Any advice or info would be greatly appreciated. Regards,Derek

2 3

[all] 2024 OpenInfra Foundation annual report
by Jeremy Stanley 15 Jan '25

15 Jan '25

The OpenInfra Foundation staff have published 2024's annual report: https://openinfra.org/annual-report/2024 As usual, it includes a section on OpenStack, covering some highlights of our community's achievements throughout the year. There's an accompanying post on the OpenStack Blog too: https://www.openstack.org/blog/openstack-2024-community-progress-highlights/ Please feel free to share these anywhere you think people may be interested in seeing our year in review, or that of our sibling communities and other activities supported by the foundation. -- Jeremy Stanley

1 0

[kolla-ansible][keystone]problem using LDAP and TLS
by Franck VEDEL 14 Jan '25

14 Jan '25

Hello, with all previous versions, to authenticate users from an Active Directory domain, using an LDAP connector worked without any problem on my labs, with or without using TLS. With TLS the file used is the following: /etc/kolla/config/keystone/domains/keystone.DOM1.conf [identity] driver = ldap [ldap] url = ldap://172.16.60.41 user = CN=ldapconnect,OU=openstack,DC=dom1,DC=loc password = XXXXXXXXXX suffix = DC=dom1,DC=loc user_tree_dn = OU=openstack,DC=dom1,DC=loc user_objectclass = person user_filter = (|(memberOf=CN=openstack-users,OU=openstack,DC=dom1,DC=loc)(memberOf=CN=openstack-admins,OU=openstack,DC=dom1,DC=loc)) user_id_attribute = sAMAccountName user_name_attribute = sAMAccountName user_mail_attribute = mail user_pass_attribute = user_enabled_attribute = userAccountControl user_enabled_mask = 2 user_enabled_default = 512 user_attribute_ignore = password,tenant_id,tenants group_objectclass = group group_tree_dn = OU=openstack,DC=dom1,DC=loc group_filter = (|(CN=openstack-users)(CN=openstack-admins)) group_id_attribute = cn group_name_attribute = name query_scope = sub chase_referrals = false use_tls = True tls_certificate = /etc/kolla/certificates/adserv.crt tls_req_cert = never I want to see if it works with Dalmatian (to update my lab, which I can't do with the Bobcat and Caracal versions because bugs block me). Without TLS (use_tls = False) , no problem, it works, my AD users can use Horizon and have access to their project. If I put "use_tls = True" and tls_certificate = /etc/kolla/certificates/adserv.crt .... It doesn't work with Dalmatian, certificate problem. Indeed, the certificate is not copied into the keystone container, so it can't use it. In the keystone logs (/var/kolla/keystone/keystone.log) there is an error message: 2025-01-15 08:36:04.477 1084 ERROR keystone.server.flask.request_processing.middleware.auth_context OSError: tls_cacertfile /etc/kolla/certificates/adserv.crt not found or is not a file However, my adserv.crt file has not changed, is valid, and is in /etc/kolla/certificates directory. What could have changed with the Dalmatian version? I tried to change parameters, but I can't find the solution. How should I proceed to make it work with TLS. I specify again that it works without TLS so the problem is really to use this certificate correctly, so to put it in the right place with the right parameter: tls_certificate or a new one? My lab use ubuntu 2204. Thanks a lot for your help Franck VEDEL

1 0

Facing issue while creating image in Open stack using Barbican
by Shalini Srivastava 14 Jan '25

14 Jan '25

Hi Sir, I have created openstack image with the help of Barbican key-manager but it is not coming in active mode from past two days. It is still showing in queue mode only. Since i have searched a lot reagarding the same issue but could not find any solution to fix the same. Hope to hear from you soon. Thanks & Regards Shalini Srivastava Project Associate /प्रोजेक्ट एसोसिएट Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294 ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

1 0

[dev] Invitation to participate in a survey on flaky tests spanning multiple OpenStack projects
by xiao＠ait.kyushu-u.ac.jp 14 Jan '25

14 Jan '25

Hello there, My name is Tao Xiao, Assistant Professor at Kyushu University (Japan), and togather with a team from Canada and China. Our research specifically investigates cross-project flakiness by analyzing information available on the Gerrit Code Review platform. Through our study, we have observed a significant number of flaky tests spanning multiple OpenStack projects and categorized several potential causes behind this phenomenon. Our next step is to validate our empirical findings and gather real developers' perceptions to improve the OpenStack ecosystem through a developer survey. If you have at most 10 minutes, we have designed a simple survey to capture your perceptions. The survey link is here: https://forms.gle/dUMWRL8MNALQE6kG8 This questionnaire is totally anonymized, and the survey was approved by the Kyushu University, Japan ethics board (2024-26). Again, thank you so much for your time, and please feel free to contact us for more information. Tao Xiao on Behalf of Our Team Assistant Professor Faculty of Information Science and Electrical Engineering Kyushu University, Japan xiao(a)ait.kyushu-u.ac.jp <mailto:xiao@ait.kyushu-u.ac.jp> Team Members and their homepages: Tao Xiao – https://tao-xiao.github.io/ Dong Wang – https://dong-w.github.io/ Shane McIntosh – https://shanemcintosh.org/ Hideaki Hata – https://hideakihata.github.io/ Yasutaka Kamei – https://posl.ait.kyushu-u.ac.jp/~kamei/

1 0

[openstack][octavia] How clients can debug their LB
by Nguyễn Hữu Khôi 14 Jan '25

14 Jan '25

Hello. I would like to know how clients can debug their LB such as access log or monitor LB performance? Thank you. Regards Nguyen Huu Khoi

5 7

[ironic] Ironic-Lib being deprecated
by Jay Faulkner 14 Jan '25

14 Jan '25

Hi all, As many of you know, at the recent PTG the Ironic community made a decision to move any functionality out of ironic-lib into ironic / ironic-python-agent. This library has been documented, since it's inception, as for use by Ironic projects only, so there should be little to no impact to deployers other than having one less python package to deploy in their IPA ramdisks and Ironic services. We will keep the repository's stable (and unmaintained) branches open for as long as their Ironic/IPA counterparts are also supported in order to fulfill our support obligation. Remaining patches to complete this deprecation are here: https://review.opendev.org/q/topic:%22ironic-lib-deprecation%22 -- please reach out if there are any questions or concerns. I thought it wise to inform the mailing list of this change. We'll ensure it's highlighted in Ironic cycle highlights for epoxy as well. Thanks, Jay Faulkner

1 0

[release] [upgrades] Seeking OpenStack Upgrade Success Stories
by Allison Price 14 Jan '25

14 Jan '25

Hi everyone, Ahead of the OpenStack 2025.1 ‘Epoxy’ release, the OpenInfra Foundation marketing team wants to work with the OpenStack community to highlight how upgrades have become simplified with the implementation of SLURP. This will include leveraging the governance docs to draft articles around the upstream progress as well as operator success stories for organizations who have upgraded their own deployments or customer deployments to a SLURP release. The goal is to further increase awareness on the improvements made to the upgrade process and reduce common perceptions that OpenStack upgrades remain complex. Here [1] is an etherpad where we are collecting names of volunteers who can either review materials promoting the upstream progress or share operator success stories. If you have any questions, please let me know. [1] https://etherpad.opendev.org/p/OpenStackUpgrades Cheers, Allison Allison Price VP of Marketing & Community p: +1-214-686-6821

1 0

[all] Foundation Individual Director election voting underway
by Jeremy Stanley 14 Jan '25

14 Jan '25

Voting is open now for the 2025 OpenInfra Foundation Individual Director Election. You must have joined the Open Infrastructure Foundation as an Individual Member by Sunday, July 14, 2024 to vote in this election, per the bylaws: https://openinfra.org/legal/bylaws Individual Directors serve as board representatives for the Individual Members who elect them, so are the most effective way we as participants in OpenStack can influence board decisions on the general direction of our supporting foundation. Voting in these elections, once you're eligible to do so, is also necessary in order to maintain your individual membership. To vote, check your email (or your spam folder) for your personal link to vote. It will have been sent to the primary address in your foundation member profile. The subject line is: "OpenInfra Foundation 2025 Individual Director Election" You can review the candidates here: https://openinfra.dev/election/2025-individual-director-election/candidates Voting will close this Friday, January 17th, 2025 at 1900 UTC. Please contact secretary(a)openinfra.dev with any questions you may have about the election. -- Jeremy Stanley

1 0

[kolla] [wallaby] [openvswitch]
by Albert Braden 14 Jan '25

14 Jan '25

One of our busiest clusters was rebuilt on Wallaby a few months ago (kolla-ansible) and deletions are taking a long time for VMs on some hypervisors. When I look at the ovs-vswitchd.log on the slow hypervisors I see lots of errors that apparently refer to a missing VM interface: 2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network device qvo0430a2f7-cd (No such device) For most network devices I see "added" and "deleted" lines in the log. This one has an "added" line but no "deleted". I tried restarting the neutron_openvswitch_agent and openvswitch_vswitchd containers but that didn't make a difference. How can I get OVS to stop choking on this missing interface?

2 17

[ironic][networking] Baremetal Networking Working Group resuming 2/15
by Julia Kreger 14 Jan '25

14 Jan '25

Greetings everyone, Now that we are in the new year and a number of ironic contributors are back from PTO, we're going to resume the networking meetings. One aspect which we discussed in December was to focus the discussion. To that end, after some follow-up discussion, we're going to try and focus the meeting a bit more using the agenda. We have an agenda posted to the etherpad[0], along with a meeting link. If you have additional topics, please feel free to add to that agenda, but also be mindful between the use cases of standalone usage and openstack integrated usage and explicitly call out the use case. As a reminder, the meeting will be at 1500 UTC. Meeting link in the etherpad! Thanks! -Julia [0]: https://etherpad.opendev.org/p/ironic-networking

1 0

[publiccloud-sig] Reminder: first meeting of the new year - January 15th at 0800 UTC
by Felix Kronlage-Dammers 14 Jan '25

14 Jan '25

Hi everyone, This upcoming Wednesday (January 15th) the next meeting of the publiccloud-sig is going to happen. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG felix -- Felix Kronlage-Dammers Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[all][tc] Election dates for the 2025.2 cycle
by Sławek Kapłoński 14 Jan '25

14 Jan '25

Hi, To align with the proposal from Goutham [1] we have updated election calendar for the upcoming elections [2] to make voting period slightly longer thus nomination will also start earlier. Please be aware of it. New calendar looks like: Release is at: 2025-04-02 Latest possible completion is at: 2025-03-19 Moving back to Wednesday: 2025-03-19 PTL Election from 2025-02-26T23:45 to 2025-03-19T23:45 TC Election from 2025-02-26T23:45 to 2025-03-19T23:45 TC Campaigning from 2025-02-19T23:45 to 2025-02-26T23:45 PTL Nominations from 2025-02-05T23:45 to 2025-02-19T23:45 TC Nominations from 2025-02-05T23:45 to 2025-02-19T23:45 Set email_deadline to 2025-02-19T00:00 Setting TC timeframe end to email_deadline Begining of 2024.2 Cycle @ 2024-03-15 00:00:00+00:00 End of 2025.1 cycle @ 2025-02-19 00:00:00+00:00 Election timeframe: 341 days, 0:00:00s --- [1] https://review.opendev.org/c/openstack/governance/+/937741 [2] https://review.opendev.org/c/openstack/election/+/938782/ -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

Dalmatian Devstack: Empty Port Binding section in Horizon
by berndbausch＠gmail.com 14 Jan '25

14 Jan '25

Is there a known bug, either in Devstack or Horizon, that prevents Neutron port binding details from being displayed? My googling did not find anything. As admin, I open a port's details, and in the Binding section at the end of the page, all I get is a bold text "VNIC Type". The CLI command "openstack port show" does provide the details. Also on a Xena Devstack, Horizon displays them[1]. Horizon's setup.py seems to be identical on both releases, as far as Neutron configuration is concerned. Thanks for any input. Bernd. [1] The Binding details I expect: Binding VNIC Type Normal Host xena-202 Profile None VIF Type ovs VIF Details connectivity l2 port_filter True ovs_hybrid_plug False datapath_type system bridge_name br-int

2 1

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-11)
by Goutham Pacha Ravi 13 Jan '25

13 Jan '25

Hello Stackers, Time flies! We're 11 weeks away from the 2025.1 "Epoxy" release day [1]. OpenStack project teams must be working on their deliverables according to the schedule shared by Előd Illés (elodilles) from the release team [2] this week. In the past week, the OpenStack Technical Committee (TC) worked with election officials to adjust the dates of the elections preceding the 2025.2 ("Flamingo" [3]) release cycle. This change was made in response to a revision in the TC's charter that now allows more time for polling. Election officials will share the updated schedule on this list soon. Additionally, the TC approved the retirement of openstack-ansible roles for former projects: Murano (Application Catalog), Senlin (Clustering Service), and Sahara (Data Processing). The OpenInfra Board's Individual Member Director elections are currently underway [4]. If you are an OpenInfra Foundation member, please check your email for a ballot and participate. The election will conclude on Friday, 2025-01-17. === Weekly Meeting === The OpenStack TC resumed its regular weekly meeting schedule last week following a brief hiatus due to the year-end holidays. Last week's meeting was held on 2025-01-07 at 1800 UTC, simultaneously on Zoom and IRC. Please find the meeting minutes on eavesdrop [5] and a recording on YouTube [6]. The meeting began with a discussion about the pending proposal [7] to delete the "unmaintained/victoria" branch on OpenStack git repositories. There is interest from at least one contributing organization in keeping this branch open for specific repositories. The deadline to merge this change is 2025-01-31. We aim to identify the specific repositories and the responsible maintainers by then. I'll keep you updated in future emails. The TC then discussed initiating the election cycle for the 2025.2 release. We recognize that long holidays (such as the upcoming Chinese New Year) could impact nominations from interested candidates. We hope to encourage nominations as early as possible since even a couple of extra weeks could make a difference in coordinating for such a geographically diverse community. The TC also reviewed the status of the community goal to migrate test jobs to Python 3.12 (and "Ubuntu 24.04 / Noble Numbat" where applicable). Ghanshyam Mann (gmann), the goal champion, shared that test jobs in three projects (Heat, Skyline, and devstack-plugin-container) need attention from their respective project teams [8]. We also expressed our gratitude to the OpenDev Infrastructure team for keeping the systems running smoothly during the holidays. The next OpenStack Technical Committee meeting is today, 2025-01-14, at 1800 UTC. This meeting will be held over IRC in OFTC's #openstack-tc channel. Please find the agenda on the meeting wiki [9]. I hope you can join us. Remember, any community member can propose meeting topics—just mention your IRC nick so the meeting chair can call upon you. === Governance Proposals === ==== Merged ==== - Allow more than 2 weeks for elections | https://review.opendev.org/c/openstack/governance/+/937741 - Put whitebox-tempest-plugin under release management | https://review.opendev.org/c/openstack/governance/+/938401 - Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 ==== Open for Review ==== - Rework the eventlet-removal goal proposal | https://review.opendev.org/c/openstack/governance/+/931254 - Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 - Retire Freezer DR | https://review.opendev.org/c/openstack/governance/+/938183 - Retire qdrouterd role | https://review.opendev.org/c/openstack/governance/+/938193 - Remove Freezer from inactive state | https://review.opendev.org/c/openstack/governance/+/938938 - Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 - Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 === How to Contact the TC === You can reach the TC in several ways: - Email: Send an email with the tag [tc] on this mailing list. - Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC. - Join us at our weekly meeting: The Technical Committee meets every week on Tuesdays at 1800 UTC [9]. === Upcoming Events === - 2025-01-17: 2025 OpenInfra Board Individual Member Director Elections conclude - 2025-02-01: FOSDEM 2025 (https://fosdem.org/2025/) OpenStack's 15th Birthday Celebration - 2025-02-28: 2025.1 ("Epoxy") Feature Freeze and release milestone 3 [1] - 2025-03-06: SCALE 2025 + OpenInfra Days NA (https://www.socallinuxexpo.org/scale/22x) Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] Release countdown for week R-11: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [3] OpenStack 2025.2 'F' Release Naming Poll: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [4] OpenInfra Foundation Board Elections: https://openinfra.dev/election/2025-individual-director-election [5] TC Meeting IRC Log 2025-01-07: https://meetings.opendev.org/meetings/tc/2025/tc.2025-01-07-18.00.log.html [6] TC Meeting Video Recording, 2025-01-07: https://youtu.be/-Nxul8_ykto [7] Transition unmaintained/victoria to EOL: https://review.opendev.org/c/openstack/releases/+/937515 [8] Projects failing the "migrate-to-noble" goal: https://etherpad.opendev.org/p/migrate-to-noble#L172 [9] TC Meeting Agenda, 2025-01-14: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[openstack][octavia] Override default 50000 OctaviaTimeoutMemberData
by Andy Speagle 13 Jan '25

13 Jan '25

Hi Folks, We're using charmed openstack and charmed k8s on top of that. In our k8s environment, we have an nginx ingress controller that uses the openstack-integrator charm to create and manage octavia LBs. I'm trying to figure out how to manage the haproxy frontend and backend timeout parameters that seem to default to 50000ms via: OctaviaTimeoutClientData OctaviaTimeoutMemberData Ideally, being able to specify this in the k8s service manifest that gets passed down into the octavia config would be "the way." Any ideas? Thanks! -- Andy Speagle

1 1

Assigning specific IP address to a host
by William Muriithi 13 Jan '25

13 Jan '25

Hello I have a system that I would like to assign a specific IP address. I am aware of a way I can set it up on creation but want to change it post creation so that I can prepare the host before swapping the IP address to bring it to production. This is what I have tried:- Set up a second interface: openstack server add fixed ip --fixed-ip-address 192.168.20.61 f3a20bef-bb03-4e9a-8996-77567ef96ff0 d1051b62-46a7-487e-b188-f742aec6d711 Remove the default security group: openstack port unset --security-group c0108624-dbd8-42bb-acaa-bbbfd32fbbae 6978e153-47dc-42d4-bdb8-63b325c77cf0 Assign the relevant security group: openstack port set --security-group 415a6ef0-e461-41fc-85f0-c92c043e5b50 6978e153-47dc-42d4-bdb8-63b325c77cf0 After that, when I ssh to the virtual machine, I can see the new interface and indeed also ping/ssh to the system using that IP address. However, its not reachable from outside the virtual machine What could I be missing here? Regards, William

2 1

[Neutron] Bug report Jan 6 to Jan 12
by Rodolfo Alonso Hernandez 13 Jan '25

13 Jan '25

Hello Neutrinos: This is the bug report from the past week. *HIGH* * DVR flow is not installed for allowed address pairs mac in DVR_TO_SRC_MAC table=1 (https://bugs.launchpad.net/neutron/+bug/2093248) Patch proposed: https://review.opendev.org/c/openstack/neutron/+/938657 * [OVN] Port unbinding operation receives rogue port UP event too ( https://bugs.launchpad.net/neutron/+bug/2093327) *Not assigned*. * [ovn-octavia-provider] first request is stuck on OVN txn ( https://bugs.launchpad.net/neutron/+bug/2093347) Patch proposed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/938797 *MEDIUM* * OVS Manager creation is failing at ovs-agent startup causing that values like inactivity_timeout are not set ( https://bugs.launchpad.net/neutron/+bug/2093433) Assigned. *LOW* * Missing Documentation for vpnaas with ovn ( https://bugs.launchpad.net/neutron/+bug/2093317) *Not assigned*. Regards.

1 0

[all][gate] Bandit version 1.8.1 issues
by Brian Haley 12 Jan '25

12 Jan '25

Hi Stackers, I was re-checking some simple Neutron changes and noticed our pep8 job started failing running bandit. Seems there was a new version (1.8.1) just released and it's generating false-positives [0]. I filed a bug against it at [1], and for now I'll just work around it by ignoring B106 warnings in Neutron with [2]. Didn't know if we wanted to pin/denylist it while the issue is addressed as I'm assuming others projects will start seeing the same failure. -Brian [0] https://bugs.launchpad.net/neutron/+bug/2093849 [1] https://github.com/PyCQA/bandit/issues/1216 [2] https://review.opendev.org/c/openstack/neutron/+/939004

1 1

keystone concepts
by Budai Laszlo 12 Jan '25

12 Jan '25

Dear all, reading the keystone documentation (https://docs.openstack.org/keystone/latest/getting-started/architecture.html) we can see the following sentence: "|Projects| represent the base unit of |ownership| in OpenStack, in that all resources in OpenStack should be owned by a specific project." What are "resources" in the previous statement? Thank you in advance, Laszlo

2 3

keystone concepts
by Budai Laszlo 12 Jan '25

12 Jan '25

Dear all, reading the keystone documentation (https://docs.openstack.org/keystone/latest/getting-started/architecture.html) we can see the following sentence: "|Projects| represent the base unit of |ownership| in OpenStack, in that all resources in OpenStack should be owned by a specific project." What are "resources" in the previous statement? Thank you in advance, Laszlo

1 0

[release] OpenStack 2025.2 'F' Release Naming Poll
by Allison Price 10 Jan '25

10 Jan '25

Hi everyone, Please join me in voting to name OpenStack 2025.2, starting with the letter ‘F’! https://civs1.civs.us/cgi-bin/vote.pl?id=E_7207f29f7e2e222d&akey=f80b0a4bbd… The deadline to vote is Friday, January 10 at 1700 UTC. Cheers, Allison

1 1

Step 2 of HA Wallaby overcloud fails with "ERROR: Can't run container mysql_wait_bundle"
by davorin.mikulic＠gmail.com 10 Jan '25

10 Jan '25

Dear Openstack community, I am trying to deploy Wallaby overcloud on HP Gen9 machines with CentOS 9 installed on them. The very similar setup with CentOS 8 deployed and worked successfully. Deploy fails on step 2 with: 2025-01-07 12:07:31.485297 | | WARNING | ERROR: Can't run container mysql_wait_bundle stderr: + STEP=2 + TAGS=file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user + CONFIG='include tripleo::profile::pacemaker::database::mysql_bundle' + EXTRA_ARGS= + '[' -d /tmp/puppet-etc ']' + cp -a /tmp/puppet-etc/devices /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet + echo '{"step": 2}' + export FACTER_deployment_type=containers + FACTER_deployment_type=containers + set +e + puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user -e 'noop_resource('\''package'\''); include tripleo::profile::pacemaker::database::mysql_bundle' Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 (file: /etc/puppet/hiera.yaml) Warning: Undefined variable '::deploy_config_name'; (file & line not available) Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html (file & line not available) Warning: Unknown variable: '::pacemaker::pcs_010'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 159, column: 6) Error: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0] Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0] Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[root@localhost]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/File[/root/.my.cnf]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@127.0.0.1]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@::1]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@localhost]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@%]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@localhost.localdomain]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@localhost.localdomain]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@overcloud-controller-0.tb27.cld]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@overcloud-controller-0.tb27.cld]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@overcloud-controller-0]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@overcloud-controller-0]: Skipping because of failed dependencies Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_database[test]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[root@%]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[clustercheck@localhost]: Skipping because of failed dependencies Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Mysql_database[cinder]: Skipping because of failed dependencies Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Mysql_database[glance]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Skipping because of failed dependencies Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Mysql_database[keystone]: Skipping because of failed dependencies Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Mysql_database[ovs_neutron]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Mysql_database[nova]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova_cell0]/Mysql_database[nova_cell0]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Mysql_database[nova_api]: Skipping because of failed dependencies Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Mysql_database[placement]: Skipping because of failed dependencies Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_user[cinder@%]: Skipping because of failed dependencies Error: Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_grant[cinder@%/cinder.*]: Skipping because of failed dependencies Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_user[glance@%]: Skipping because of failed dependencies Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_grant[glance@%/glance.*]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Skipping because of failed dependencies Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_user[keystone@%]: Skipping because of failed dependencies Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_grant[keystone@%/keystone.*]: Skipping because of failed dependencies Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_user[neutron@%]: Skipping because of failed dependencies Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_grant[neutron@%/ovs_neutron.*]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_user[nova@%]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_grant[nova@%/nova.*]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova_cell0]/Openstacklib::Db::Mysql::Host_access[nova_cell0_%]/Mysql_grant[nova@%/nova_cell0.*]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Openstacklib::Db::Mysql::Host_access[nova_api_%]/Mysql_user[nova_api@%]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Openstacklib::Db::Mysql::Host_access[nova_api_%]/Mysql_grant[nova_api@%/nova_api.*]: Skipping because of failed dependencies Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Openstacklib::Db::Mysql::Host_access[placement_%]/Mysql_user[placement@%]: Skipping because of failed dependencies Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Openstacklib::Db::Mysql::Host_access[placement_%]/Mysql_grant[placement@%/placement.*]: Skipping because of failed dependencies + rc=6 + set -e + set +ux 2025-01-07 12:07:31.488412 | | WARNING | ERROR: Can't run container rabbitmq_wait_bundle stderr: + STEP=2 + TAGS=file,file_line,concat,augeas,rabbitmq_policy,rabbitmq_user,rabbitmq_ready + CONFIG='include tripleo::profile::pacemaker::rabbitmq_bundle' + EXTRA_ARGS= + '[' -d /tmp/puppet-etc ']' + cp -a /tmp/puppet-etc/devices /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet + echo '{"step": 2}' + export FACTER_deployment_type=containers + FACTER_deployment_type=containers + set +e + puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,rabbitmq_policy,rabbitmq_user,rabbitmq_ready -e 'noop_resource('\''package'\''); include tripleo::profile::pacemaker::rabbitmq_bundle' Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 (file: /etc/puppet/hiera.yaml) Warning: Undefined variable '::deploy_config_name'; (file & line not available) Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html (file & line not available) Warning: Unknown variable: '::pacemaker::pcs_010'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 159, column: 6) Error: 'rabbitmqctl eval "rabbit_nodes:is_running(node(), rabbit)." | grep -q true && rabbitmqctl eval "rabbit_mnesia:is_clustered()." | grep -q true' returned 1 instead of one of [0] Error: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Exec[rabbitmq-ready]/returns: change from 'notrun' to ['0'] failed: 'rabbitmqctl eval "rabbit_nodes:is_running(node(), rabbit)." | grep -q true && rabbitmqctl eval "rabbit_mnesia:is_clustered()." | grep -q true' returned 1 instead of one of [0] Warning: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Rabbitmq_policy[ha-all@/]: Skipping because of failed dependencies + rc=6 + set -e + set +ux 2025-01-07 12:07:31.489548 | 000c29c7-8523-9a37-c0c4-00000000f883 | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | overcloud-controller-0 | error={"changed": false, "msg": "Failed containers: mysql_wait_bundle, rabbitmq_wait_bundle"} 2025-01-07 12:07:31.490512 | 000c29c7-8523-9a37-c0c4-00000000f883 | TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | overcloud-controller-0 | 0:48:23.163306 | 1989.39s PLAY RECAP ********************************************************************* localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 overcloud-cephstorage-0 : ok=273 changed=70 unreachable=0 failed=0 skipped=151 rescued=0 ignored=1 overcloud-cephstorage-1 : ok=275 changed=70 unreachable=0 failed=0 skipped=146 rescued=0 ignored=1 overcloud-cephstorage-2 : ok=275 changed=70 unreachable=0 failed=0 skipped=146 rescued=0 ignored=1 overcloud-computesriov-0 : ok=342 changed=92 unreachable=0 failed=0 skipped=163 rescued=0 ignored=1 overcloud-computesriov-1 : ok=349 changed=92 unreachable=0 failed=0 skipped=156 rescued=0 ignored=1 overcloud-controller-0 : ok=372 changed=99 unreachable=0 failed=1 skipped=217 rescued=0 ignored=1 overcloud-controller-1 : ok=392 changed=98 unreachable=0 failed=0 skipped=212 rescued=0 ignored=1 overcloud-controller-2 : ok=392 changed=98 unreachable=0 failed=0 skipped=212 rescued=0 ignored=1 undercloud : ok=76 changed=19 unreachable=0 failed=0 skipped=31 rescued=0 ignored=1 2025-01-07 12:07:31.561753 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.562447 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 1905 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.563026 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:48:23.235822 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.563565 | UUID | Info | Host | Task Name | Run Time 2025-01-07 12:07:31.564127 | 000c29c7-8523-9a37-c0c4-00000000f883 | SUMMARY | overcloud-controller-0 | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | 1989.39s 2025-01-07 12:07:31.564700 | 000c29c7-8523-9a37-c0c4-00000000bb0c | SUMMARY | undercloud | tripleo_container_image_prepare : Container image prepare | 137.67s 2025-01-07 12:07:31.565400 | 000c29c7-8523-9a37-c0c4-00000000e34e | SUMMARY | undercloud | tripleo_run_cephadm : run cephadm playbook | 59.69s 2025-01-07 12:07:31.565937 | 000c29c7-8523-9a37-c0c4-00000000e715 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for haproxy | 56.48s 2025-01-07 12:07:31.566525 | 000c29c7-8523-9a37-c0c4-00000000e747 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run pacemaker restart if the config file for the service changed | 46.63s 2025-01-07 12:07:31.567080 | 000c29c7-8523-9a37-c0c4-000000004353 | SUMMARY | overcloud-controller-1 | tripleo_firewall : Manage firewall rules | 36.87s 2025-01-07 12:07:31.567628 | 000c29c7-8523-9a37-c0c4-000000004353 | SUMMARY | overcloud-controller-0 | tripleo_firewall : Manage firewall rules | 36.75s 2025-01-07 12:07:31.568206 | 000c29c7-8523-9a37-c0c4-000000004353 | SUMMARY | overcloud-controller-2 | tripleo_firewall : Manage firewall rules | 36.65s 2025-01-07 12:07:31.568767 | 000c29c7-8523-9a37-c0c4-00000000e776 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run pacemaker restart if the config file for the service changed | 36.45s 2025-01-07 12:07:31.569335 | 000c29c7-8523-9a37-c0c4-00000000e745 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for mysql | 24.87s 2025-01-07 12:07:31.569886 | 000c29c7-8523-9a37-c0c4-00000000e774 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for oslo_messaging_rpc | 23.94s 2025-01-07 12:07:31.570454 | 000c29c7-8523-9a37-c0c4-00000000caa9 | SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 20.83s 2025-01-07 12:07:31.571025 | 000c29c7-8523-9a37-c0c4-00000000f709 | SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 20.82s 2025-01-07 12:07:31.571574 | 000c29c7-8523-9a37-c0c4-00000000be26 | SUMMARY | overcloud-controller-2 | Pre-fetch all the containers | 15.90s 2025-01-07 12:07:31.572173 | 000c29c7-8523-9a37-c0c4-00000000bdad | SUMMARY | overcloud-controller-1 | Pre-fetch all the containers | 15.88s 2025-01-07 12:07:31.572740 | 000c29c7-8523-9a37-c0c4-00000000bdad | SUMMARY | overcloud-controller-0 | Pre-fetch all the containers | 15.68s 2025-01-07 12:07:31.573331 | 000c29c7-8523-9a37-c0c4-0000000024d7 | SUMMARY | overcloud-computesriov-1 | tripleo_hieradata : Render hieradata from template | 11.32s 2025-01-07 12:07:31.573885 | 000c29c7-8523-9a37-c0c4-0000000024d7 | SUMMARY | overcloud-computesriov-0 | tripleo_hieradata : Render hieradata from template | 11.14s 2025-01-07 12:07:31.574467 | 000c29c7-8523-9a37-c0c4-00000000c4a0 | SUMMARY | overcloud-computesriov-1 | Wait for puppet host configuration to finish | 11.07s 2025-01-07 12:07:31.575030 | 000c29c7-8523-9a37-c0c4-00000000c452 | SUMMARY | overcloud-computesriov-0 | Wait for puppet host configuration to finish | 10.98s 2025-01-07 12:07:31.575572 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.576197 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.576756 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.577328 | The following node(s) had failures: overcloud-controller-0 2025-01-07 12:07:31.577879 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:32.979 180292 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleogr2nbip5 ] cleaned up 2025-01-07 12:07:32.979 180292 ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:32.981 180292 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh Host 10.100.27.127 not found in /home/stack/.ssh/known_hosts Overcloud Endpoint: http://10.100.27.127:5000 Overcloud Horizon Dashboard URL: http://10.100.27.127:80/dashboard Overcloud rc file: /home/stack/overcloud-deploy/overcloud/overcloudrc and /home/stack/overcloudrc Overcloud Deployed with error 2025-01-07 12:07:33.690 180292 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Stopping ephemeral heat. 2025-01-07 12:07:33.774 180292 INFO tripleoclient.heat_launcher [-] Pod does not exist: ephemeral-heat 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args) 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args) 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/cliff/command.py", line 185, in run 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1331, in take_action 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.set_deployment_status( 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.force_reraise() 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise self.value 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1303, in take_action 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.config_download( 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/workflows/deployment.py", line 386, in config_download 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud utils.run_ansible_playbook( 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 775, in run_ansible_playbook 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise RuntimeError(err_msg) 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 2025-01-07 12:07:36.434 180292 ERROR openstack [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:36.437 180292 INFO osc_lib.shell [-] END return value: 1 I can see the following logs on the failed controller-0 node: [root@overcloud-controller-0 log]# less /var/log/containers/rabbitmq/log/crash.log ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:29:03 =ERROR REPORT==== ** Generic server rabbit_disk_monitor terminating ** Last message in was update ** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld",50000000,1173484339200,100,10000,#Ref<0.112052652.1425801217.106754>,false,true,10,120000} ** Reason for termination == ** {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]} 2025-01-07 10:29:03 =CRASH REPORT==== crasher: initial call: rabbit_disk_monitor:init/1 pid: <0.309.0> registered_name: rabbit_disk_monitor exception exit: {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]} ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.301.0>] message_queue_len: 0 messages: [] links: [<0.308.0>] dictionary: [] trap_exit: false status: running heap_size: 2586 stack_size: 27 reductions: 314108420 neighbours: 2025-01-07 10:29:03 =SUPERVISOR REPORT==== Supervisor: {local,rabbit_disk_monitor_sup} Context: child_terminated Reason: {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"} Offender: [{pid,<0.309.0>},{name,rabbit_disk_monitor},{mfargs,{rabbit_disk_monitor,start_link,[50000000]}},{restart_type,{transient,1}},{shutdown,30000},{child_type,worker}] less /var/log/containers/rabbitmq/rabbit(a)overcloud-controller-0.internalapi.tb27.cld.log 2024-12-19 19:18:44.490 [info] <0.9.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_web_dispatch * rabbitmq_management_agent 2025-01-07 10:28:44.393 [error] <0.11277.105> ** Connection attempt from disallowed node 'rabbitmqcli-169019-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:44.404 [error] <0.11280.105> ** Connection attempt from disallowed node 'rabbitmqcli-169019-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:45.167 [error] <0.11284.105> ** Connection attempt from disallowed node 'rabbitmqcli-169251-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:45.179 [error] <0.11286.105> ** Connection attempt from disallowed node 'rabbitmqcli-169251-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.282 [error] <0.11282.105> ** Connection attempt from disallowed node 'rabbitmqcli-169491-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.293 [error] <0.11293.105> ** Connection attempt from disallowed node 'rabbitmqcli-169491-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.779 [error] <0.11298.105> ** Connection attempt from disallowed node 'rabbitmqcli-169710-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.788 [error] <0.11300.105> ** Connection attempt from disallowed node 'rabbitmqcli-169710-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:52.948 [error] <0.11303.105> ** Connection attempt from disallowed node 'rabbitmqcli-169935-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:52.958 [error] <0.11312.105> ** Connection attempt from disallowed node 'rabbitmqcli-169935-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:54.824 [error] <0.11317.105> ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:54.837 [error] <0.11318.105> ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55.645 [error] <0.11325.105> ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55.655 [error] <0.11327.105> ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56.160 [error] <0.11328.105> ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56.171 [error] <0.11334.105> ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:29:03.051 [error] <0.309.0> ** Generic server rabbit_disk_monitor terminating ** Last message in was update ** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld",50000000,1173484339200,100,10000,#Ref<0.112052652.1425801217.106754>,f alse,true,10,120000} ** Reason for termination == ** {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"sr c/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/r abbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,ini t_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]} 2025-01-07 10:29:03.051 [error] <0.309.0> CRASH REPORT Process rabbit_disk_monitor with 0 neighbours exited with reason: {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overclou d-controller-0.internalapi.tb27.cld: No such file or directory\n"} in rabbit_disk_monitor:parse_free_unix/1 line 272 2025-01-07 10:29:03.051 [error] <0.308.0> Supervisor rabbit_disk_monitor_sup had child rabbit_disk_monitor started with rabbit_disk_monitor:start_link(50000000) at <0.309.0> exit with reason {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"} in context child_terminated 2025-01-07 10:29:03.059 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq /mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 10 2025-01-07 10:31:03.068 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq /mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 9 2025-01-07 10:33:03.076 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq /mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 8 [root@overcloud-controller-0 ~]# pcs status --full Cluster name: tripleo_cluster Cluster Summary: * Stack: corosync (Pacemaker is running) * Current DC: overcloud-controller-1 (2) (version 2.1.9-1.el9-49aab9983) - partition with quorum * Last updated: Tue Jan 7 11:49:48 2025 on overcloud-controller-0 * Last change: Tue Jan 7 10:33:11 2025 by root via root on overcloud-controller-0 * 9 nodes configured * 26 resource instances configured (1 BLOCKED from further action due to failure) Node List: * GuestNode galera-bundle-0@overcloud-controller-2: online * GuestNode galera-bundle-1@overcloud-controller-0: online * GuestNode galera-bundle-2@overcloud-controller-1: online * Node overcloud-controller-0 (1): online, feature set 3.19.6 * Node overcloud-controller-1 (2): online, feature set 3.19.6 * Node overcloud-controller-2 (3): online, feature set 3.19.6 * GuestNode rabbitmq-bundle-0@overcloud-controller-2: online * GuestNode rabbitmq-bundle-1@overcloud-controller-0: online * GuestNode rabbitmq-bundle-2@overcloud-controller-1: online Full List of Resources: * ip-10.100.21.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-0 * ip-10.100.27.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-1 * ip-10.100.22.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-2 * ip-10.100.24.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-0 * ip-10.100.25.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-1 * Container bundle set: haproxy-bundle [cluster.common.tag/haproxy:pcmklatest]: * Replica[0] * haproxy-bundle-podman-0 (ocf:heartbeat:podman): Started overcloud-controller-0 * Replica[1] * haproxy-bundle-podman-1 (ocf:heartbeat:podman): Started overcloud-controller-1 * Replica[2] * haproxy-bundle-podman-2 (ocf:heartbeat:podman): Started overcloud-controller-2 * Container bundle set: galera-bundle [cluster.common.tag/mariadb:pcmklatest]: * Replica[0] * galera (ocf:heartbeat:galera): FAILED Promoted galera-bundle-0 (blocked) * galera-bundle-podman-0 (ocf:heartbeat:podman): Started overcloud-controller-2 * galera-bundle-0 (ocf:pacemaker:remote): Started overcloud-controller-2 * Replica[1] * galera (ocf:heartbeat:galera): Unpromoted galera-bundle-1 * galera-bundle-podman-1 (ocf:heartbeat:podman): Started overcloud-controller-0 * galera-bundle-1 (ocf:pacemaker:remote): Started overcloud-controller-0 * Replica[2] * galera (ocf:heartbeat:galera): Unpromoted galera-bundle-2 * galera-bundle-podman-2 (ocf:heartbeat:podman): Started overcloud-controller-1 * galera-bundle-2 (ocf:pacemaker:remote): Started overcloud-controller-1 * Container bundle set: rabbitmq-bundle [cluster.common.tag/rabbitmq:pcmklatest]: * Replica[0] * rabbitmq (ocf:heartbeat:rabbitmq-cluster): Started rabbitmq-bundle-0 * rabbitmq-bundle-podman-0 (ocf:heartbeat:podman): Started overcloud-controller-2 * rabbitmq-bundle-0 (ocf:pacemaker:remote): Started overcloud-controller-2 * Replica[1] * rabbitmq (ocf:heartbeat:rabbitmq-cluster): Started rabbitmq-bundle-1 * rabbitmq-bundle-podman-1 (ocf:heartbeat:podman): Started overcloud-controller-0 * rabbitmq-bundle-1 (ocf:pacemaker:remote): Started overcloud-controller-0 * Replica[2] * rabbitmq (ocf:heartbeat:rabbitmq-cluster): Started rabbitmq-bundle-2 * rabbitmq-bundle-podman-2 (ocf:heartbeat:podman): Started overcloud-controller-1 * rabbitmq-bundle-2 (ocf:pacemaker:remote): Started overcloud-controller-1 Node Attributes: * Node: overcloud-controller-0 (1): * galera-last-committed : -1 * galera-no-grastate : true * galera-role : true * haproxy-role : true * rabbitmq-role : true * rmq-node-attr-last-known-rabbitmq : rabbit(a)overcloud-controller-0.internalapi.tb27.cld * rmq-node-attr-rabbitmq : rabbit(a)overcloud-controller-0.internalapi.tb27.cld * Node: overcloud-controller-1 (2): * galera-last-committed : -1 * galera-no-grastate : true * galera-role : true * haproxy-role : true * rabbitmq-role : true * rmq-node-attr-last-known-rabbitmq : rabbit(a)overcloud-controller-1.internalapi.tb27.cld * rmq-node-attr-rabbitmq : rabbit(a)overcloud-controller-1.internalapi.tb27.cld * Node: overcloud-controller-2 (3): * galera-bootstrap : true * galera-role : true * haproxy-role : true * master-galera : 100 * rabbitmq-role : true * rmq-node-attr-last-known-rabbitmq : rabbit(a)overcloud-controller-2.internalapi.tb27.cld * rmq-node-attr-rabbitmq : rabbit(a)overcloud-controller-2.internalapi.tb27.cld Migration Summary: * Node: galera-bundle-0@overcloud-controller-2: * galera: migration-threshold=1000000 fail-count=1 last-failure='Tue Jan 7 10:32:38 2025' Failed Resource Actions: * galera_promote_0 on galera-bundle-0 'error' (1): call=116, status='complete', exitreason='Failed initial monitor action', last-rc-change='Tue Jan 7 10:32:28 2025', queued=0ms, exec=10603ms Tickets: PCSD Status: overcloud-controller-0: Online overcloud-controller-1: Online overcloud-controller-2: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled Does someone perhaps knows where the issue lies? Thank you very much!

3 4

[release] Release countdown for week R-11, Jan 13 - 17
by Elõd Illés 10 Jan '25

10 Jan '25

Development Focus ----------------- We are now past the Epoxy-2 milestone, and entering the last development phase of the cycle. Teams should be focused on implementing planned work for the cycle. Now is a good time to review those plans and reprioritize anything if needed based on the what progress has been made and what looks realistic to complete in the next few weeks. General Information ------------------- Looking ahead to the end of the release cycle, please be aware of the feature freeze dates. Those vary depending on deliverable type: * General libraries (except client libraries) need to have their last feature release before Non-client library freeze (February 20th, 2025). Their stable branches are cut early. * Client libraries (think python-*client libraries) need to have their last feature release before Client library freeze (February 27th, 2025) * Deliverables following a cycle-with-rc model (that would be most services) observe a Feature freeze on that same date, February 27th, 2025. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. After feature freeze, cycle-with-rc deliverables need to produce a first release candidate (and a stable branch) before RC1 deadline (March 13th, 2025) * Deliverables following cycle-with-intermediary model can release as necessary, but in all cases before Final RC deadline (March 27th, 2025) Upcoming Deadlines & Dates -------------------------- Non-client library freeze: February 20th, 2025 (R-6 week) Client library freeze: February 27th, 2025 (R-5 week) Epoxy-3 milestone: February 27th, 2025 (R-5 week) Előd Illés irc: elodilles @ #openstack-release

1 0

Openstack kolla networking setup
by matek7323＠interia.pl 10 Jan '25

10 Jan '25

I am setting up an Openstack Kolla environment with the goal of moving all virtualizations into one place At the moment I have 3 management controllers and 5 compute node for virtualization + CEPH I have there such settings as eth0 management network 172.16.50.0 and eth1 as external neutron 91.1.1.0/24 as public address And now I want to achieve two things: 1) How to make the vmka get the public address directly without natting it on the principle. Private address --> 10.224.1.15 --> public 91.1.1.15 2) Leave the option of nating after through the vrouter so that the machine assigns a private address and then a public IP address To all these servers for now the situation looks like this Port eth0 = vlan 750 access on the switch - MGMT port eth1 = vlan 888 access on switch - VLAN to operator so that I can broadcast public IP address network: version: 2 ethernets: eth0: match: macaddress: “***********” addresses: - “172.16.50.11/24” nameservers: addresses: - 8.8.8.8 set-name: eth0 routes: - to: default via: 172.16.50.254 eth1: match: macaddress: “**********” dhcp4: false dhcp6: false set-name: eth1 Each of the servers has the same config and now how to configure this to adjust the config of the coll, the servers as well as the openstack to be able to work natively and on the principle of assigning IP addresses directly to the machine by vrouter Translated with DeepL.com (free version)

1 0