- openstack-discuss - lists.openstack.org

[kolla] [wallaby] [openvswitch]
by Albert Braden 14 Jan '25

14 Jan '25

One of our busiest clusters was rebuilt on Wallaby a few months ago (kolla-ansible) and deletions are taking a long time for VMs on some hypervisors. When I look at the ovs-vswitchd.log on the slow hypervisors I see lots of errors that apparently refer to a missing VM interface: 2024-12-27T19:06:10.482Z|01793|bridge|WARN|could not open network device qvo0430a2f7-cd (No such device) For most network devices I see "added" and "deleted" lines in the log. This one has an "added" line but no "deleted". I tried restarting the neutron_openvswitch_agent and openvswitch_vswitchd containers but that didn't make a difference. How can I get OVS to stop choking on this missing interface?

2 17

[ironic][networking] Baremetal Networking Working Group resuming 2/15
by Julia Kreger 14 Jan '25

14 Jan '25

Greetings everyone, Now that we are in the new year and a number of ironic contributors are back from PTO, we're going to resume the networking meetings. One aspect which we discussed in December was to focus the discussion. To that end, after some follow-up discussion, we're going to try and focus the meeting a bit more using the agenda. We have an agenda posted to the etherpad[0], along with a meeting link. If you have additional topics, please feel free to add to that agenda, but also be mindful between the use cases of standalone usage and openstack integrated usage and explicitly call out the use case. As a reminder, the meeting will be at 1500 UTC. Meeting link in the etherpad! Thanks! -Julia [0]: https://etherpad.opendev.org/p/ironic-networking

1 0

[publiccloud-sig] Reminder: first meeting of the new year - January 15th at 0800 UTC
by Felix Kronlage-Dammers 14 Jan '25

14 Jan '25

Hi everyone, This upcoming Wednesday (January 15th) the next meeting of the publiccloud-sig is going to happen. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG felix -- Felix Kronlage-Dammers Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[all][tc] Election dates for the 2025.2 cycle
by Sławek Kapłoński 14 Jan '25

14 Jan '25

Hi, To align with the proposal from Goutham [1] we have updated election calendar for the upcoming elections [2] to make voting period slightly longer thus nomination will also start earlier. Please be aware of it. New calendar looks like: Release is at: 2025-04-02 Latest possible completion is at: 2025-03-19 Moving back to Wednesday: 2025-03-19 PTL Election from 2025-02-26T23:45 to 2025-03-19T23:45 TC Election from 2025-02-26T23:45 to 2025-03-19T23:45 TC Campaigning from 2025-02-19T23:45 to 2025-02-26T23:45 PTL Nominations from 2025-02-05T23:45 to 2025-02-19T23:45 TC Nominations from 2025-02-05T23:45 to 2025-02-19T23:45 Set email_deadline to 2025-02-19T00:00 Setting TC timeframe end to email_deadline Begining of 2024.2 Cycle @ 2024-03-15 00:00:00+00:00 End of 2025.1 cycle @ 2025-02-19 00:00:00+00:00 Election timeframe: 341 days, 0:00:00s --- [1] https://review.opendev.org/c/openstack/governance/+/937741 [2] https://review.opendev.org/c/openstack/election/+/938782/ -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

Dalmatian Devstack: Empty Port Binding section in Horizon
by berndbausch＠gmail.com 14 Jan '25

14 Jan '25

Is there a known bug, either in Devstack or Horizon, that prevents Neutron port binding details from being displayed? My googling did not find anything. As admin, I open a port's details, and in the Binding section at the end of the page, all I get is a bold text "VNIC Type". The CLI command "openstack port show" does provide the details. Also on a Xena Devstack, Horizon displays them[1]. Horizon's setup.py seems to be identical on both releases, as far as Neutron configuration is concerned. Thanks for any input. Bernd. [1] The Binding details I expect: Binding VNIC Type Normal Host xena-202 Profile None VIF Type ovs VIF Details connectivity l2 port_filter True ovs_hybrid_plug False datapath_type system bridge_name br-int

2 1

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-11)
by Goutham Pacha Ravi 13 Jan '25

13 Jan '25

Hello Stackers, Time flies! We're 11 weeks away from the 2025.1 "Epoxy" release day [1]. OpenStack project teams must be working on their deliverables according to the schedule shared by Előd Illés (elodilles) from the release team [2] this week. In the past week, the OpenStack Technical Committee (TC) worked with election officials to adjust the dates of the elections preceding the 2025.2 ("Flamingo" [3]) release cycle. This change was made in response to a revision in the TC's charter that now allows more time for polling. Election officials will share the updated schedule on this list soon. Additionally, the TC approved the retirement of openstack-ansible roles for former projects: Murano (Application Catalog), Senlin (Clustering Service), and Sahara (Data Processing). The OpenInfra Board's Individual Member Director elections are currently underway [4]. If you are an OpenInfra Foundation member, please check your email for a ballot and participate. The election will conclude on Friday, 2025-01-17. === Weekly Meeting === The OpenStack TC resumed its regular weekly meeting schedule last week following a brief hiatus due to the year-end holidays. Last week's meeting was held on 2025-01-07 at 1800 UTC, simultaneously on Zoom and IRC. Please find the meeting minutes on eavesdrop [5] and a recording on YouTube [6]. The meeting began with a discussion about the pending proposal [7] to delete the "unmaintained/victoria" branch on OpenStack git repositories. There is interest from at least one contributing organization in keeping this branch open for specific repositories. The deadline to merge this change is 2025-01-31. We aim to identify the specific repositories and the responsible maintainers by then. I'll keep you updated in future emails. The TC then discussed initiating the election cycle for the 2025.2 release. We recognize that long holidays (such as the upcoming Chinese New Year) could impact nominations from interested candidates. We hope to encourage nominations as early as possible since even a couple of extra weeks could make a difference in coordinating for such a geographically diverse community. The TC also reviewed the status of the community goal to migrate test jobs to Python 3.12 (and "Ubuntu 24.04 / Noble Numbat" where applicable). Ghanshyam Mann (gmann), the goal champion, shared that test jobs in three projects (Heat, Skyline, and devstack-plugin-container) need attention from their respective project teams [8]. We also expressed our gratitude to the OpenDev Infrastructure team for keeping the systems running smoothly during the holidays. The next OpenStack Technical Committee meeting is today, 2025-01-14, at 1800 UTC. This meeting will be held over IRC in OFTC's #openstack-tc channel. Please find the agenda on the meeting wiki [9]. I hope you can join us. Remember, any community member can propose meeting topics—just mention your IRC nick so the meeting chair can call upon you. === Governance Proposals === ==== Merged ==== - Allow more than 2 weeks for elections | https://review.opendev.org/c/openstack/governance/+/937741 - Put whitebox-tempest-plugin under release management | https://review.opendev.org/c/openstack/governance/+/938401 - Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 ==== Open for Review ==== - Rework the eventlet-removal goal proposal | https://review.opendev.org/c/openstack/governance/+/931254 - Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 - Retire Freezer DR | https://review.opendev.org/c/openstack/governance/+/938183 - Retire qdrouterd role | https://review.opendev.org/c/openstack/governance/+/938193 - Remove Freezer from inactive state | https://review.opendev.org/c/openstack/governance/+/938938 - Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 - Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 === How to Contact the TC === You can reach the TC in several ways: - Email: Send an email with the tag [tc] on this mailing list. - Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC. - Join us at our weekly meeting: The Technical Committee meets every week on Tuesdays at 1800 UTC [9]. === Upcoming Events === - 2025-01-17: 2025 OpenInfra Board Individual Member Director Elections conclude - 2025-02-01: FOSDEM 2025 (https://fosdem.org/2025/) OpenStack's 15th Birthday Celebration - 2025-02-28: 2025.1 ("Epoxy") Feature Freeze and release milestone 3 [1] - 2025-03-06: SCALE 2025 + OpenInfra Days NA (https://www.socallinuxexpo.org/scale/22x) Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] Release countdown for week R-11: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [3] OpenStack 2025.2 'F' Release Naming Poll: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [4] OpenInfra Foundation Board Elections: https://openinfra.dev/election/2025-individual-director-election [5] TC Meeting IRC Log 2025-01-07: https://meetings.opendev.org/meetings/tc/2025/tc.2025-01-07-18.00.log.html [6] TC Meeting Video Recording, 2025-01-07: https://youtu.be/-Nxul8_ykto [7] Transition unmaintained/victoria to EOL: https://review.opendev.org/c/openstack/releases/+/937515 [8] Projects failing the "migrate-to-noble" goal: https://etherpad.opendev.org/p/migrate-to-noble#L172 [9] TC Meeting Agenda, 2025-01-14: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[openstack][octavia] Override default 50000 OctaviaTimeoutMemberData
by Andy Speagle 13 Jan '25

13 Jan '25

Hi Folks, We're using charmed openstack and charmed k8s on top of that. In our k8s environment, we have an nginx ingress controller that uses the openstack-integrator charm to create and manage octavia LBs. I'm trying to figure out how to manage the haproxy frontend and backend timeout parameters that seem to default to 50000ms via: OctaviaTimeoutClientData OctaviaTimeoutMemberData Ideally, being able to specify this in the k8s service manifest that gets passed down into the octavia config would be "the way." Any ideas? Thanks! -- Andy Speagle

1 1

Assigning specific IP address to a host
by William Muriithi 13 Jan '25

13 Jan '25

Hello I have a system that I would like to assign a specific IP address. I am aware of a way I can set it up on creation but want to change it post creation so that I can prepare the host before swapping the IP address to bring it to production. This is what I have tried:- Set up a second interface: openstack server add fixed ip --fixed-ip-address 192.168.20.61 f3a20bef-bb03-4e9a-8996-77567ef96ff0 d1051b62-46a7-487e-b188-f742aec6d711 Remove the default security group: openstack port unset --security-group c0108624-dbd8-42bb-acaa-bbbfd32fbbae 6978e153-47dc-42d4-bdb8-63b325c77cf0 Assign the relevant security group: openstack port set --security-group 415a6ef0-e461-41fc-85f0-c92c043e5b50 6978e153-47dc-42d4-bdb8-63b325c77cf0 After that, when I ssh to the virtual machine, I can see the new interface and indeed also ping/ssh to the system using that IP address. However, its not reachable from outside the virtual machine What could I be missing here? Regards, William

2 1

[Neutron] Bug report Jan 6 to Jan 12
by Rodolfo Alonso Hernandez 13 Jan '25

13 Jan '25

Hello Neutrinos: This is the bug report from the past week. *HIGH* * DVR flow is not installed for allowed address pairs mac in DVR_TO_SRC_MAC table=1 (https://bugs.launchpad.net/neutron/+bug/2093248) Patch proposed: https://review.opendev.org/c/openstack/neutron/+/938657 * [OVN] Port unbinding operation receives rogue port UP event too ( https://bugs.launchpad.net/neutron/+bug/2093327) *Not assigned*. * [ovn-octavia-provider] first request is stuck on OVN txn ( https://bugs.launchpad.net/neutron/+bug/2093347) Patch proposed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/938797 *MEDIUM* * OVS Manager creation is failing at ovs-agent startup causing that values like inactivity_timeout are not set ( https://bugs.launchpad.net/neutron/+bug/2093433) Assigned. *LOW* * Missing Documentation for vpnaas with ovn ( https://bugs.launchpad.net/neutron/+bug/2093317) *Not assigned*. Regards.

1 0

[all][gate] Bandit version 1.8.1 issues
by Brian Haley 12 Jan '25

12 Jan '25

Hi Stackers, I was re-checking some simple Neutron changes and noticed our pep8 job started failing running bandit. Seems there was a new version (1.8.1) just released and it's generating false-positives [0]. I filed a bug against it at [1], and for now I'll just work around it by ignoring B106 warnings in Neutron with [2]. Didn't know if we wanted to pin/denylist it while the issue is addressed as I'm assuming others projects will start seeing the same failure. -Brian [0] https://bugs.launchpad.net/neutron/+bug/2093849 [1] https://github.com/PyCQA/bandit/issues/1216 [2] https://review.opendev.org/c/openstack/neutron/+/939004

1 1

keystone concepts
by Budai Laszlo 12 Jan '25

12 Jan '25

Dear all, reading the keystone documentation (https://docs.openstack.org/keystone/latest/getting-started/architecture.html) we can see the following sentence: "|Projects| represent the base unit of |ownership| in OpenStack, in that all resources in OpenStack should be owned by a specific project." What are "resources" in the previous statement? Thank you in advance, Laszlo

2 3

keystone concepts
by Budai Laszlo 12 Jan '25

12 Jan '25

Dear all, reading the keystone documentation (https://docs.openstack.org/keystone/latest/getting-started/architecture.html) we can see the following sentence: "|Projects| represent the base unit of |ownership| in OpenStack, in that all resources in OpenStack should be owned by a specific project." What are "resources" in the previous statement? Thank you in advance, Laszlo

1 0

[release] OpenStack 2025.2 'F' Release Naming Poll
by Allison Price 10 Jan '25

10 Jan '25

Hi everyone, Please join me in voting to name OpenStack 2025.2, starting with the letter ‘F’! https://civs1.civs.us/cgi-bin/vote.pl?id=E_7207f29f7e2e222d&akey=f80b0a4bbd… The deadline to vote is Friday, January 10 at 1700 UTC. Cheers, Allison

1 1

Step 2 of HA Wallaby overcloud fails with "ERROR: Can't run container mysql_wait_bundle"
by davorin.mikulic＠gmail.com 10 Jan '25

10 Jan '25

Dear Openstack community, I am trying to deploy Wallaby overcloud on HP Gen9 machines with CentOS 9 installed on them. The very similar setup with CentOS 8 deployed and worked successfully. Deploy fails on step 2 with: 2025-01-07 12:07:31.485297 | | WARNING | ERROR: Can't run container mysql_wait_bundle stderr: + STEP=2 + TAGS=file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user + CONFIG='include tripleo::profile::pacemaker::database::mysql_bundle' + EXTRA_ARGS= + '[' -d /tmp/puppet-etc ']' + cp -a /tmp/puppet-etc/devices /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet + echo '{"step": 2}' + export FACTER_deployment_type=containers + FACTER_deployment_type=containers + set +e + puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user -e 'noop_resource('\''package'\''); include tripleo::profile::pacemaker::database::mysql_bundle' Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 (file: /etc/puppet/hiera.yaml) Warning: Undefined variable '::deploy_config_name'; (file & line not available) Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html (file & line not available) Warning: Unknown variable: '::pacemaker::pcs_010'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 159, column: 6) Error: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0] Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0] Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[root@localhost]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/File[/root/.my.cnf]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@127.0.0.1]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@::1]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@localhost]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@%]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@localhost.localdomain]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@localhost.localdomain]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@overcloud-controller-0.tb27.cld]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@overcloud-controller-0.tb27.cld]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@overcloud-controller-0]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@overcloud-controller-0]: Skipping because of failed dependencies Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_database[test]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[root@%]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[clustercheck@localhost]: Skipping because of failed dependencies Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Mysql_database[cinder]: Skipping because of failed dependencies Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Mysql_database[glance]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Skipping because of failed dependencies Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Mysql_database[keystone]: Skipping because of failed dependencies Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Mysql_database[ovs_neutron]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Mysql_database[nova]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova_cell0]/Mysql_database[nova_cell0]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Mysql_database[nova_api]: Skipping because of failed dependencies Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Mysql_database[placement]: Skipping because of failed dependencies Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_user[cinder@%]: Skipping because of failed dependencies Error: Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_grant[cinder@%/cinder.*]: Skipping because of failed dependencies Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_user[glance@%]: Skipping because of failed dependencies Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_grant[glance@%/glance.*]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Skipping because of failed dependencies Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_user[keystone@%]: Skipping because of failed dependencies Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_grant[keystone@%/keystone.*]: Skipping because of failed dependencies Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_user[neutron@%]: Skipping because of failed dependencies Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_grant[neutron@%/ovs_neutron.*]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_user[nova@%]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_grant[nova@%/nova.*]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova_cell0]/Openstacklib::Db::Mysql::Host_access[nova_cell0_%]/Mysql_grant[nova@%/nova_cell0.*]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Openstacklib::Db::Mysql::Host_access[nova_api_%]/Mysql_user[nova_api@%]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Openstacklib::Db::Mysql::Host_access[nova_api_%]/Mysql_grant[nova_api@%/nova_api.*]: Skipping because of failed dependencies Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Openstacklib::Db::Mysql::Host_access[placement_%]/Mysql_user[placement@%]: Skipping because of failed dependencies Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Openstacklib::Db::Mysql::Host_access[placement_%]/Mysql_grant[placement@%/placement.*]: Skipping because of failed dependencies + rc=6 + set -e + set +ux 2025-01-07 12:07:31.488412 | | WARNING | ERROR: Can't run container rabbitmq_wait_bundle stderr: + STEP=2 + TAGS=file,file_line,concat,augeas,rabbitmq_policy,rabbitmq_user,rabbitmq_ready + CONFIG='include tripleo::profile::pacemaker::rabbitmq_bundle' + EXTRA_ARGS= + '[' -d /tmp/puppet-etc ']' + cp -a /tmp/puppet-etc/devices /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet + echo '{"step": 2}' + export FACTER_deployment_type=containers + FACTER_deployment_type=containers + set +e + puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,rabbitmq_policy,rabbitmq_user,rabbitmq_ready -e 'noop_resource('\''package'\''); include tripleo::profile::pacemaker::rabbitmq_bundle' Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 (file: /etc/puppet/hiera.yaml) Warning: Undefined variable '::deploy_config_name'; (file & line not available) Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html (file & line not available) Warning: Unknown variable: '::pacemaker::pcs_010'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 159, column: 6) Error: 'rabbitmqctl eval "rabbit_nodes:is_running(node(), rabbit)." | grep -q true && rabbitmqctl eval "rabbit_mnesia:is_clustered()." | grep -q true' returned 1 instead of one of [0] Error: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Exec[rabbitmq-ready]/returns: change from 'notrun' to ['0'] failed: 'rabbitmqctl eval "rabbit_nodes:is_running(node(), rabbit)." | grep -q true && rabbitmqctl eval "rabbit_mnesia:is_clustered()." | grep -q true' returned 1 instead of one of [0] Warning: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Rabbitmq_policy[ha-all@/]: Skipping because of failed dependencies + rc=6 + set -e + set +ux 2025-01-07 12:07:31.489548 | 000c29c7-8523-9a37-c0c4-00000000f883 | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | overcloud-controller-0 | error={"changed": false, "msg": "Failed containers: mysql_wait_bundle, rabbitmq_wait_bundle"} 2025-01-07 12:07:31.490512 | 000c29c7-8523-9a37-c0c4-00000000f883 | TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | overcloud-controller-0 | 0:48:23.163306 | 1989.39s PLAY RECAP ********************************************************************* localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 overcloud-cephstorage-0 : ok=273 changed=70 unreachable=0 failed=0 skipped=151 rescued=0 ignored=1 overcloud-cephstorage-1 : ok=275 changed=70 unreachable=0 failed=0 skipped=146 rescued=0 ignored=1 overcloud-cephstorage-2 : ok=275 changed=70 unreachable=0 failed=0 skipped=146 rescued=0 ignored=1 overcloud-computesriov-0 : ok=342 changed=92 unreachable=0 failed=0 skipped=163 rescued=0 ignored=1 overcloud-computesriov-1 : ok=349 changed=92 unreachable=0 failed=0 skipped=156 rescued=0 ignored=1 overcloud-controller-0 : ok=372 changed=99 unreachable=0 failed=1 skipped=217 rescued=0 ignored=1 overcloud-controller-1 : ok=392 changed=98 unreachable=0 failed=0 skipped=212 rescued=0 ignored=1 overcloud-controller-2 : ok=392 changed=98 unreachable=0 failed=0 skipped=212 rescued=0 ignored=1 undercloud : ok=76 changed=19 unreachable=0 failed=0 skipped=31 rescued=0 ignored=1 2025-01-07 12:07:31.561753 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.562447 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 1905 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.563026 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:48:23.235822 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.563565 | UUID | Info | Host | Task Name | Run Time 2025-01-07 12:07:31.564127 | 000c29c7-8523-9a37-c0c4-00000000f883 | SUMMARY | overcloud-controller-0 | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | 1989.39s 2025-01-07 12:07:31.564700 | 000c29c7-8523-9a37-c0c4-00000000bb0c | SUMMARY | undercloud | tripleo_container_image_prepare : Container image prepare | 137.67s 2025-01-07 12:07:31.565400 | 000c29c7-8523-9a37-c0c4-00000000e34e | SUMMARY | undercloud | tripleo_run_cephadm : run cephadm playbook | 59.69s 2025-01-07 12:07:31.565937 | 000c29c7-8523-9a37-c0c4-00000000e715 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for haproxy | 56.48s 2025-01-07 12:07:31.566525 | 000c29c7-8523-9a37-c0c4-00000000e747 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run pacemaker restart if the config file for the service changed | 46.63s 2025-01-07 12:07:31.567080 | 000c29c7-8523-9a37-c0c4-000000004353 | SUMMARY | overcloud-controller-1 | tripleo_firewall : Manage firewall rules | 36.87s 2025-01-07 12:07:31.567628 | 000c29c7-8523-9a37-c0c4-000000004353 | SUMMARY | overcloud-controller-0 | tripleo_firewall : Manage firewall rules | 36.75s 2025-01-07 12:07:31.568206 | 000c29c7-8523-9a37-c0c4-000000004353 | SUMMARY | overcloud-controller-2 | tripleo_firewall : Manage firewall rules | 36.65s 2025-01-07 12:07:31.568767 | 000c29c7-8523-9a37-c0c4-00000000e776 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run pacemaker restart if the config file for the service changed | 36.45s 2025-01-07 12:07:31.569335 | 000c29c7-8523-9a37-c0c4-00000000e745 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for mysql | 24.87s 2025-01-07 12:07:31.569886 | 000c29c7-8523-9a37-c0c4-00000000e774 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for oslo_messaging_rpc | 23.94s 2025-01-07 12:07:31.570454 | 000c29c7-8523-9a37-c0c4-00000000caa9 | SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 20.83s 2025-01-07 12:07:31.571025 | 000c29c7-8523-9a37-c0c4-00000000f709 | SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 20.82s 2025-01-07 12:07:31.571574 | 000c29c7-8523-9a37-c0c4-00000000be26 | SUMMARY | overcloud-controller-2 | Pre-fetch all the containers | 15.90s 2025-01-07 12:07:31.572173 | 000c29c7-8523-9a37-c0c4-00000000bdad | SUMMARY | overcloud-controller-1 | Pre-fetch all the containers | 15.88s 2025-01-07 12:07:31.572740 | 000c29c7-8523-9a37-c0c4-00000000bdad | SUMMARY | overcloud-controller-0 | Pre-fetch all the containers | 15.68s 2025-01-07 12:07:31.573331 | 000c29c7-8523-9a37-c0c4-0000000024d7 | SUMMARY | overcloud-computesriov-1 | tripleo_hieradata : Render hieradata from template | 11.32s 2025-01-07 12:07:31.573885 | 000c29c7-8523-9a37-c0c4-0000000024d7 | SUMMARY | overcloud-computesriov-0 | tripleo_hieradata : Render hieradata from template | 11.14s 2025-01-07 12:07:31.574467 | 000c29c7-8523-9a37-c0c4-00000000c4a0 | SUMMARY | overcloud-computesriov-1 | Wait for puppet host configuration to finish | 11.07s 2025-01-07 12:07:31.575030 | 000c29c7-8523-9a37-c0c4-00000000c452 | SUMMARY | overcloud-computesriov-0 | Wait for puppet host configuration to finish | 10.98s 2025-01-07 12:07:31.575572 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.576197 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.576756 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.577328 | The following node(s) had failures: overcloud-controller-0 2025-01-07 12:07:31.577879 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:32.979 180292 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleogr2nbip5 ] cleaned up 2025-01-07 12:07:32.979 180292 ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:32.981 180292 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh Host 10.100.27.127 not found in /home/stack/.ssh/known_hosts Overcloud Endpoint: http://10.100.27.127:5000 Overcloud Horizon Dashboard URL: http://10.100.27.127:80/dashboard Overcloud rc file: /home/stack/overcloud-deploy/overcloud/overcloudrc and /home/stack/overcloudrc Overcloud Deployed with error 2025-01-07 12:07:33.690 180292 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Stopping ephemeral heat. 2025-01-07 12:07:33.774 180292 INFO tripleoclient.heat_launcher [-] Pod does not exist: ephemeral-heat 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args) 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args) 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/cliff/command.py", line 185, in run 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1331, in take_action 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.set_deployment_status( 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.force_reraise() 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise self.value 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1303, in take_action 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.config_download( 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/workflows/deployment.py", line 386, in config_download 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud utils.run_ansible_playbook( 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 775, in run_ansible_playbook 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise RuntimeError(err_msg) 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 2025-01-07 12:07:36.434 180292 ERROR openstack [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:36.437 180292 INFO osc_lib.shell [-] END return value: 1 I can see the following logs on the failed controller-0 node: [root@overcloud-controller-0 log]# less /var/log/containers/rabbitmq/log/crash.log ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:29:03 =ERROR REPORT==== ** Generic server rabbit_disk_monitor terminating ** Last message in was update ** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld",50000000,1173484339200,100,10000,#Ref<0.112052652.1425801217.106754>,false,true,10,120000} ** Reason for termination == ** {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]} 2025-01-07 10:29:03 =CRASH REPORT==== crasher: initial call: rabbit_disk_monitor:init/1 pid: <0.309.0> registered_name: rabbit_disk_monitor exception exit: {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]} ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.301.0>] message_queue_len: 0 messages: [] links: [<0.308.0>] dictionary: [] trap_exit: false status: running heap_size: 2586 stack_size: 27 reductions: 314108420 neighbours: 2025-01-07 10:29:03 =SUPERVISOR REPORT==== Supervisor: {local,rabbit_disk_monitor_sup} Context: child_terminated Reason: {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"} Offender: [{pid,<0.309.0>},{name,rabbit_disk_monitor},{mfargs,{rabbit_disk_monitor,start_link,[50000000]}},{restart_type,{transient,1}},{shutdown,30000},{child_type,worker}] less /var/log/containers/rabbitmq/rabbit(a)overcloud-controller-0.internalapi.tb27.cld.log 2024-12-19 19:18:44.490 [info] <0.9.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_web_dispatch * rabbitmq_management_agent 2025-01-07 10:28:44.393 [error] <0.11277.105> ** Connection attempt from disallowed node 'rabbitmqcli-169019-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:44.404 [error] <0.11280.105> ** Connection attempt from disallowed node 'rabbitmqcli-169019-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:45.167 [error] <0.11284.105> ** Connection attempt from disallowed node 'rabbitmqcli-169251-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:45.179 [error] <0.11286.105> ** Connection attempt from disallowed node 'rabbitmqcli-169251-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.282 [error] <0.11282.105> ** Connection attempt from disallowed node 'rabbitmqcli-169491-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.293 [error] <0.11293.105> ** Connection attempt from disallowed node 'rabbitmqcli-169491-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.779 [error] <0.11298.105> ** Connection attempt from disallowed node 'rabbitmqcli-169710-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.788 [error] <0.11300.105> ** Connection attempt from disallowed node 'rabbitmqcli-169710-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:52.948 [error] <0.11303.105> ** Connection attempt from disallowed node 'rabbitmqcli-169935-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:52.958 [error] <0.11312.105> ** Connection attempt from disallowed node 'rabbitmqcli-169935-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:54.824 [error] <0.11317.105> ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:54.837 [error] <0.11318.105> ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55.645 [error] <0.11325.105> ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55.655 [error] <0.11327.105> ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56.160 [error] <0.11328.105> ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56.171 [error] <0.11334.105> ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:29:03.051 [error] <0.309.0> ** Generic server rabbit_disk_monitor terminating ** Last message in was update ** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld",50000000,1173484339200,100,10000,#Ref<0.112052652.1425801217.106754>,f alse,true,10,120000} ** Reason for termination == ** {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"sr c/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/r abbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,ini t_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]} 2025-01-07 10:29:03.051 [error] <0.309.0> CRASH REPORT Process rabbit_disk_monitor with 0 neighbours exited with reason: {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overclou d-controller-0.internalapi.tb27.cld: No such file or directory\n"} in rabbit_disk_monitor:parse_free_unix/1 line 272 2025-01-07 10:29:03.051 [error] <0.308.0> Supervisor rabbit_disk_monitor_sup had child rabbit_disk_monitor started with rabbit_disk_monitor:start_link(50000000) at <0.309.0> exit with reason {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"} in context child_terminated 2025-01-07 10:29:03.059 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq /mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 10 2025-01-07 10:31:03.068 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq /mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 9 2025-01-07 10:33:03.076 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq /mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 8 [root@overcloud-controller-0 ~]# pcs status --full Cluster name: tripleo_cluster Cluster Summary: * Stack: corosync (Pacemaker is running) * Current DC: overcloud-controller-1 (2) (version 2.1.9-1.el9-49aab9983) - partition with quorum * Last updated: Tue Jan 7 11:49:48 2025 on overcloud-controller-0 * Last change: Tue Jan 7 10:33:11 2025 by root via root on overcloud-controller-0 * 9 nodes configured * 26 resource instances configured (1 BLOCKED from further action due to failure) Node List: * GuestNode galera-bundle-0@overcloud-controller-2: online * GuestNode galera-bundle-1@overcloud-controller-0: online * GuestNode galera-bundle-2@overcloud-controller-1: online * Node overcloud-controller-0 (1): online, feature set 3.19.6 * Node overcloud-controller-1 (2): online, feature set 3.19.6 * Node overcloud-controller-2 (3): online, feature set 3.19.6 * GuestNode rabbitmq-bundle-0@overcloud-controller-2: online * GuestNode rabbitmq-bundle-1@overcloud-controller-0: online * GuestNode rabbitmq-bundle-2@overcloud-controller-1: online Full List of Resources: * ip-10.100.21.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-0 * ip-10.100.27.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-1 * ip-10.100.22.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-2 * ip-10.100.24.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-0 * ip-10.100.25.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-1 * Container bundle set: haproxy-bundle [cluster.common.tag/haproxy:pcmklatest]: * Replica[0] * haproxy-bundle-podman-0 (ocf:heartbeat:podman): Started overcloud-controller-0 * Replica[1] * haproxy-bundle-podman-1 (ocf:heartbeat:podman): Started overcloud-controller-1 * Replica[2] * haproxy-bundle-podman-2 (ocf:heartbeat:podman): Started overcloud-controller-2 * Container bundle set: galera-bundle [cluster.common.tag/mariadb:pcmklatest]: * Replica[0] * galera (ocf:heartbeat:galera): FAILED Promoted galera-bundle-0 (blocked) * galera-bundle-podman-0 (ocf:heartbeat:podman): Started overcloud-controller-2 * galera-bundle-0 (ocf:pacemaker:remote): Started overcloud-controller-2 * Replica[1] * galera (ocf:heartbeat:galera): Unpromoted galera-bundle-1 * galera-bundle-podman-1 (ocf:heartbeat:podman): Started overcloud-controller-0 * galera-bundle-1 (ocf:pacemaker:remote): Started overcloud-controller-0 * Replica[2] * galera (ocf:heartbeat:galera): Unpromoted galera-bundle-2 * galera-bundle-podman-2 (ocf:heartbeat:podman): Started overcloud-controller-1 * galera-bundle-2 (ocf:pacemaker:remote): Started overcloud-controller-1 * Container bundle set: rabbitmq-bundle [cluster.common.tag/rabbitmq:pcmklatest]: * Replica[0] * rabbitmq (ocf:heartbeat:rabbitmq-cluster): Started rabbitmq-bundle-0 * rabbitmq-bundle-podman-0 (ocf:heartbeat:podman): Started overcloud-controller-2 * rabbitmq-bundle-0 (ocf:pacemaker:remote): Started overcloud-controller-2 * Replica[1] * rabbitmq (ocf:heartbeat:rabbitmq-cluster): Started rabbitmq-bundle-1 * rabbitmq-bundle-podman-1 (ocf:heartbeat:podman): Started overcloud-controller-0 * rabbitmq-bundle-1 (ocf:pacemaker:remote): Started overcloud-controller-0 * Replica[2] * rabbitmq (ocf:heartbeat:rabbitmq-cluster): Started rabbitmq-bundle-2 * rabbitmq-bundle-podman-2 (ocf:heartbeat:podman): Started overcloud-controller-1 * rabbitmq-bundle-2 (ocf:pacemaker:remote): Started overcloud-controller-1 Node Attributes: * Node: overcloud-controller-0 (1): * galera-last-committed : -1 * galera-no-grastate : true * galera-role : true * haproxy-role : true * rabbitmq-role : true * rmq-node-attr-last-known-rabbitmq : rabbit(a)overcloud-controller-0.internalapi.tb27.cld * rmq-node-attr-rabbitmq : rabbit(a)overcloud-controller-0.internalapi.tb27.cld * Node: overcloud-controller-1 (2): * galera-last-committed : -1 * galera-no-grastate : true * galera-role : true * haproxy-role : true * rabbitmq-role : true * rmq-node-attr-last-known-rabbitmq : rabbit(a)overcloud-controller-1.internalapi.tb27.cld * rmq-node-attr-rabbitmq : rabbit(a)overcloud-controller-1.internalapi.tb27.cld * Node: overcloud-controller-2 (3): * galera-bootstrap : true * galera-role : true * haproxy-role : true * master-galera : 100 * rabbitmq-role : true * rmq-node-attr-last-known-rabbitmq : rabbit(a)overcloud-controller-2.internalapi.tb27.cld * rmq-node-attr-rabbitmq : rabbit(a)overcloud-controller-2.internalapi.tb27.cld Migration Summary: * Node: galera-bundle-0@overcloud-controller-2: * galera: migration-threshold=1000000 fail-count=1 last-failure='Tue Jan 7 10:32:38 2025' Failed Resource Actions: * galera_promote_0 on galera-bundle-0 'error' (1): call=116, status='complete', exitreason='Failed initial monitor action', last-rc-change='Tue Jan 7 10:32:28 2025', queued=0ms, exec=10603ms Tickets: PCSD Status: overcloud-controller-0: Online overcloud-controller-1: Online overcloud-controller-2: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled Does someone perhaps knows where the issue lies? Thank you very much!

3 4

[release] Release countdown for week R-11, Jan 13 - 17
by Elõd Illés 10 Jan '25

10 Jan '25

Development Focus ----------------- We are now past the Epoxy-2 milestone, and entering the last development phase of the cycle. Teams should be focused on implementing planned work for the cycle. Now is a good time to review those plans and reprioritize anything if needed based on the what progress has been made and what looks realistic to complete in the next few weeks. General Information ------------------- Looking ahead to the end of the release cycle, please be aware of the feature freeze dates. Those vary depending on deliverable type: * General libraries (except client libraries) need to have their last feature release before Non-client library freeze (February 20th, 2025). Their stable branches are cut early. * Client libraries (think python-*client libraries) need to have their last feature release before Client library freeze (February 27th, 2025) * Deliverables following a cycle-with-rc model (that would be most services) observe a Feature freeze on that same date, February 27th, 2025. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. After feature freeze, cycle-with-rc deliverables need to produce a first release candidate (and a stable branch) before RC1 deadline (March 13th, 2025) * Deliverables following cycle-with-intermediary model can release as necessary, but in all cases before Final RC deadline (March 27th, 2025) Upcoming Deadlines & Dates -------------------------- Non-client library freeze: February 20th, 2025 (R-6 week) Client library freeze: February 27th, 2025 (R-5 week) Epoxy-3 milestone: February 27th, 2025 (R-5 week) Előd Illés irc: elodilles @ #openstack-release

1 0

Openstack kolla networking setup
by matek7323＠interia.pl 10 Jan '25

10 Jan '25

I am setting up an Openstack Kolla environment with the goal of moving all virtualizations into one place At the moment I have 3 management controllers and 5 compute node for virtualization + CEPH I have there such settings as eth0 management network 172.16.50.0 and eth1 as external neutron 91.1.1.0/24 as public address And now I want to achieve two things: 1) How to make the vmka get the public address directly without natting it on the principle. Private address --> 10.224.1.15 --> public 91.1.1.15 2) Leave the option of nating after through the vrouter so that the machine assigns a private address and then a public IP address To all these servers for now the situation looks like this Port eth0 = vlan 750 access on the switch - MGMT port eth1 = vlan 888 access on switch - VLAN to operator so that I can broadcast public IP address network: version: 2 ethernets: eth0: match: macaddress: “***********” addresses: - “172.16.50.11/24” nameservers: addresses: - 8.8.8.8 set-name: eth0 routes: - to: default via: 172.16.50.254 eth1: match: macaddress: “**********” dhcp4: false dhcp6: false set-name: eth1 Each of the servers has the same config and now how to configure this to adjust the config of the coll, the servers as well as the openstack to be able to work natively and on the principle of assigning IP addresses directly to the machine by vrouter Translated with DeepL.com (free version)

1 0

[cinder][kubernetes] pvc stuck in reserved state
by Eugen Block 10 Jan '25

10 Jan '25

Hi, we have a couple of k8s clusters in our openstack environment (Victoria) and are facing some problems when pods get moved to different worker nodes. The PVCs get stuck in "reserved" state, and they only get attached after resetting the volumes to "available" state. First I thought it was due to the missing multiattach volume type which seems to work fine in our customer's cluster. I also found the "duplicate connector" error messages in the nova-compute.log: > Unable to update attachment.(Invalid volume: duplicate connectors > detected on volume But after adding the multiattach volume type and retyping the affected volumes, the PVCs still get stuck in "reserved" state while I don't see any error messages on the hypervisors anymore. So it seems that something has changed/improved, but now I don't have a trace where to look next. Could this be a cinder-csi driver issue or something? Our cinder backend is ceph (Pacific). Not sure which information could help so please ask for anything I can provide. Any pointers are highly appreciated! Thanks! Eugen

1 2

Third Party CI using Zuul v3 and nodepool - request for set up
by Roquesalane, Jean-Pierre 09 Jan '25

09 Jan '25

Hello OpenStack lovers! First, let me wish you all the best for this upcoming year. All our Cis are currently broken and we recently lost the team responsible for maintaining it. Unfortunately, there is a very poor documentation, and I guess we must rebuild something. I'm currently looking at Software Factory but can't get my hands on a starting point. I have a system deployed but would appreciate if you can lead me to people who have configured Zuul and nodepool. Our previous system was based on Jenkins and devstack-gate as a side note. I'll document the whole process as a help for future users. Thank you. Jean-Pierre Roquesalane SW System Sr Principal Engineer - OpenStack Champion Dell Technologies | Security and Ecosystem Software Mobile +33 6 21 86 79 04<tel:1-613-314-8106> Email : jeanpierre.roquesalane(a)dell.com<mailto:jeanpierre.roquesalane@dell.com> [cid:image001.png@01DB62C5.98BB1B40] Dell SAS, Si?ge Social 1 rond point Benjamin Franklin 34000 Montpellier, France. Capital 1,946,545.26 Euros. RCS Montpellier 351 528 229 - APE 4651Z. TVA Intracommunautaire FR 20 351 528 229. SIRET 351 528 229 00088. Vat Numbers: FR 20351528229 (France) / IT00001709997 (Italy) / ESN0012622G (Spain) Internal Use - Confidential

1 0

LXC containers
by engineer2024 09 Jan '25

09 Jan '25

I am trying to setup OSA branch 29.2.0 caracal. Whenever I try creating lxc containers using setup-hosts playbook, the new containers of version ubuntu 22 , as is the host, are missing openssh-server packages. The file /etc/ansible/roles/openstack_hosts/vars/ubuntu-22.04.yml, doesn't mention the ssh package in the distro packages. How to get ssh pkgs installed in the lxc containers ? Please help with this issue

2 1

LXC networking
by engineer2024 09 Jan '25

09 Jan '25

I have observed that in the OSA setups, on the controller nodes, each lxc container has two networks inside , eth0, eth1 interfaces. Ideally eth1 should be attached to br-mgmt bridge via veth pair. When I do 'ip link show type veth' on the host node, it is listing both the veth interfaces for each lxc container, but it is not listing the other member of that veth pair. --- # ip link show type veth 9f9add59_eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxcbr0 state UP mode DEFAULT group default qlen 1000 link/ether fe:b6:db:27:e4:a7 brd ff:ff:ff:ff:ff:ff link-netnsid 0 9f9add59_eth1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-mgmt state UP mode DEFAULT group default qlen 1000 link/ether fe:a8:30:7a:ac:3f brd ff:ff:ff:ff:ff:ff link-netnsid 0 --- --- # ip link show type veth | grep 9f9add59 72: 9f9add59_eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxcbr0 state UP mode DEFAULT group default qlen 1000 73: 9f9add59_eth1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-mgmt state UP mode DEFAULT group default qlen 1000 # --- But I can't find the other member of the veth pair, for example, for the interface 9f9add59_eth1, in the ip list. But the containers are running and reachable on the br-mgmt network. How to find out the other end of a veth pair, which is connected to the bridge ?

2 3

[blazar] reservation quota
by Winicius Allan 09 Jan '25

09 Jan '25

Is there a way to determine the quota of a project that the users can reserve? Regards.

2 1

[neutron] Proposal to deprecate and remove vlan_transparent config option
by Sławek Kapłoński 09 Jan '25

09 Jan '25

Hi, We have config option to enable vlan_transparent api extension currently. We will also have similar config option for vlan_qinq (it is now proposed in [1]). In both cases we have validation by mechanism drivers if option is or is not supported while networks are created. We can also filter out enabled API extensions by mechanism drivers, see [2] for example. So I proposed [3] to remove that config option and keep API extension to be loaded always if enabled ML2 drivers supports it. But before I will propose patch for that I would like to ask if there is maybe some use case for this config option really and maybe some of you somehow depends on it. In such case maybe we should keep it like it is now and don't remove it. Please let me know if you are using somehow this option. [1] https://review.opendev.org/c/openstack/neutron/+/937372 [2] https://github.com/openstack/neutron/blob/559672f777312d00c4358626650cc577e… [3] https://bugs.launchpad.net/neutron/+bug/2092174 -- Slawek Kaplonski Principal Software Engineer Red Hat

3 3

[Tempest] Devstack to configure tempest on a remote node
by Jakub Libosvar 08 Jan '25

08 Jan '25

Hello stackers, I have been attempting to deploy a “spine & leaf” like topology for upstream testing job of the ovn-bgp-agent [1]. The idea is to have an all-in-one devstack VM where all openstack services are running. This VM is plugged to two leaf switches respectively with its two interfaces, and each switch is connected to the “spine” switch. With the leaf switches behaving as route reflected we can validate routes are advertised to the spine. Now in order to validate that, I need to execute the tempest tests on the spine. That means I want to use devstack on the spine VM that calls OpenStack services API on the devstack all-in-one VM. The spine VM should not have any openstack service running. The problem I’m struggling with that the tempest is configured based on the ENABLED_SERVICES for the current box. I tried to hack the spine VM portion by calling configure_tempest with custom ENABLED_SERVICES from a local.sh [2] which seems to configure the tempest.conf correctly but the stack.sh script calls test-config [3] phase after local.sh is invoked, which re-configures tempest.conf back to the state as if no OpenStack services were enabled. I kinda hit the wall and ran out of ideas for hacks/workarounds with the current devstack codebase. I thought one way of solving this would be to decouple ENABLED_SERVICES into two separate variables, one being used to configure services and one for tempest purposes but before I dive into that, I wanted to ask if anybody has any experience doing something similar (configuring only tempest with devstack) or maybe there is already some variable I am not aware of. Thank you for any inputs, ideas or suggestions :) Kuba [1] https://review.opendev.org/c/openstack/ovn-bgp-agent/+/936968 [2] https://review.opendev.org/c/openstack/ovn-bgp-agent/+/936968/54/playbooks/… [3] https://opendev.org/openstack/devstack/src/branch/master/stack.sh#L1482-L15…

2 1

RDO User Survey
by Amy Marrich 08 Jan '25

08 Jan '25

Hi everyone, During today's RDO meeting we had a few topics that we need further information on in order to make decisions. As a result, we have created a 2 question survey[0] to collect the needed information. This survey will remain open for 2 weeks so we can make plans for the Epoxy release with the gathered information. The first question is in regards to CentOS Stream version(s) for the Epoxy release, CentOS Stream 9 only, 10 only, or both versions. The second question is to determine whether future RDO releases will continue to be all releases or only SLURP releases, if SLURP are the only releases folks are using. Thanks, Amy 0 - https://forms.gle/44atURvazYPXVvwZ8

1 0

[Neutron] Bug report last 2 weeks
by Rodolfo Alonso Hernandez 08 Jan '25

08 Jan '25

Hello Neutrinos: This is the bug report of the last 2 weeks. HIGH * Packet Logging Framework (OVN/ML2) ( https://bugs.launchpad.net/neutron/+bug/2092474) *Not assigned*. * with pyroute2 master dvr functional tests failing with route mismatch as returned routes are not from specific table ( https://bugs.launchpad.net/neutron/+bug/2092567) There is an open bug for pyroute2 (https://github.com/svinota/pyroute2/issues/1225) that should fix this problem. * with pyroute2 master dvr functional tests failing as pyroute.NetNS not honoring flags (https://bugs.launchpad.net/neutron/+bug/2092722) There is an open bug for pyroute2 (https://github.com/svinota/pyroute2/issues/1226) that should fix this problem. * Potential GPL violation in master branch due to GPL script embedded into Apache2 python code (https://bugs.launchpad.net/neutron/+bug/2092907) *Not assigned*. * Metadata service does not work when port security is disabled ( https://bugs.launchpad.net/neutron/+bug/2092971) *Not assigned*. MEDIUM * DeprecationWarning: create_subnetpool ( https://bugs.launchpad.net/neutron/+bug/2092657) *Not assigned*. * NoneType' object has no attribute 'registered_rules' ( https://bugs.launchpad.net/neutron/+bug/2092659) *Not assigned*. * OVN: Ports not activated on time when DPDK is used ( https://bugs.launchpad.net/neutron/+bug/2092407) Assigned to Ihar, patch proposed: https://review.opendev.org/c/openstack/neutron/+/938197. INCOMPLETE * deadlock： delete the network ( https://bugs.launchpad.net/neutron/+bug/2092983) Regards.

3 5