January 2025 - openstack-discuss - lists.openstack.org

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-11)
by Goutham Pacha Ravi 14 Jan '25

14 Jan '25

Hello Stackers, Time flies! We're 11 weeks away from the 2025.1 "Epoxy" release day [1]. OpenStack project teams must be working on their deliverables according to the schedule shared by Előd Illés (elodilles) from the release team [2] this week. In the past week, the OpenStack Technical Committee (TC) worked with election officials to adjust the dates of the elections preceding the 2025.2 ("Flamingo" [3]) release cycle. This change was made in response to a revision in the TC's charter that now allows more time for polling. Election officials will share the updated schedule on this list soon. Additionally, the TC approved the retirement of openstack-ansible roles for former projects: Murano (Application Catalog), Senlin (Clustering Service), and Sahara (Data Processing). The OpenInfra Board's Individual Member Director elections are currently underway [4]. If you are an OpenInfra Foundation member, please check your email for a ballot and participate. The election will conclude on Friday, 2025-01-17. === Weekly Meeting === The OpenStack TC resumed its regular weekly meeting schedule last week following a brief hiatus due to the year-end holidays. Last week's meeting was held on 2025-01-07 at 1800 UTC, simultaneously on Zoom and IRC. Please find the meeting minutes on eavesdrop [5] and a recording on YouTube [6]. The meeting began with a discussion about the pending proposal [7] to delete the "unmaintained/victoria" branch on OpenStack git repositories. There is interest from at least one contributing organization in keeping this branch open for specific repositories. The deadline to merge this change is 2025-01-31. We aim to identify the specific repositories and the responsible maintainers by then. I'll keep you updated in future emails. The TC then discussed initiating the election cycle for the 2025.2 release. We recognize that long holidays (such as the upcoming Chinese New Year) could impact nominations from interested candidates. We hope to encourage nominations as early as possible since even a couple of extra weeks could make a difference in coordinating for such a geographically diverse community. The TC also reviewed the status of the community goal to migrate test jobs to Python 3.12 (and "Ubuntu 24.04 / Noble Numbat" where applicable). Ghanshyam Mann (gmann), the goal champion, shared that test jobs in three projects (Heat, Skyline, and devstack-plugin-container) need attention from their respective project teams [8]. We also expressed our gratitude to the OpenDev Infrastructure team for keeping the systems running smoothly during the holidays. The next OpenStack Technical Committee meeting is today, 2025-01-14, at 1800 UTC. This meeting will be held over IRC in OFTC's #openstack-tc channel. Please find the agenda on the meeting wiki [9]. I hope you can join us. Remember, any community member can propose meeting topics—just mention your IRC nick so the meeting chair can call upon you. === Governance Proposals === ==== Merged ==== - Allow more than 2 weeks for elections | https://review.opendev.org/c/openstack/governance/+/937741 - Put whitebox-tempest-plugin under release management | https://review.opendev.org/c/openstack/governance/+/938401 - Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 ==== Open for Review ==== - Rework the eventlet-removal goal proposal | https://review.opendev.org/c/openstack/governance/+/931254 - Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 - Retire Freezer DR | https://review.opendev.org/c/openstack/governance/+/938183 - Retire qdrouterd role | https://review.opendev.org/c/openstack/governance/+/938193 - Remove Freezer from inactive state | https://review.opendev.org/c/openstack/governance/+/938938 - Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 - Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 === How to Contact the TC === You can reach the TC in several ways: - Email: Send an email with the tag [tc] on this mailing list. - Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC. - Join us at our weekly meeting: The Technical Committee meets every week on Tuesdays at 1800 UTC [9]. === Upcoming Events === - 2025-01-17: 2025 OpenInfra Board Individual Member Director Elections conclude - 2025-02-01: FOSDEM 2025 (https://fosdem.org/2025/) OpenStack's 15th Birthday Celebration - 2025-02-28: 2025.1 ("Epoxy") Feature Freeze and release milestone 3 [1] - 2025-03-06: SCALE 2025 + OpenInfra Days NA (https://www.socallinuxexpo.org/scale/22x) Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] Release countdown for week R-11: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [3] OpenStack 2025.2 'F' Release Naming Poll: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [4] OpenInfra Foundation Board Elections: https://openinfra.dev/election/2025-individual-director-election [5] TC Meeting IRC Log 2025-01-07: https://meetings.opendev.org/meetings/tc/2025/tc.2025-01-07-18.00.log.html [6] TC Meeting Video Recording, 2025-01-07: https://youtu.be/-Nxul8_ykto [7] Transition unmaintained/victoria to EOL: https://review.opendev.org/c/openstack/releases/+/937515 [8] Projects failing the "migrate-to-noble" goal: https://etherpad.opendev.org/p/migrate-to-noble#L172 [9] TC Meeting Agenda, 2025-01-14: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[openstack][octavia] Override default 50000 OctaviaTimeoutMemberData
by Andy Speagle 14 Jan '25

14 Jan '25

Hi Folks, We're using charmed openstack and charmed k8s on top of that. In our k8s environment, we have an nginx ingress controller that uses the openstack-integrator charm to create and manage octavia LBs. I'm trying to figure out how to manage the haproxy frontend and backend timeout parameters that seem to default to 50000ms via: OctaviaTimeoutClientData OctaviaTimeoutMemberData Ideally, being able to specify this in the k8s service manifest that gets passed down into the octavia config would be "the way." Any ideas? Thanks! -- Andy Speagle

1 1

Assigning specific IP address to a host
by William Muriithi 13 Jan '25

13 Jan '25

Hello I have a system that I would like to assign a specific IP address. I am aware of a way I can set it up on creation but want to change it post creation so that I can prepare the host before swapping the IP address to bring it to production. This is what I have tried:- Set up a second interface: openstack server add fixed ip --fixed-ip-address 192.168.20.61 f3a20bef-bb03-4e9a-8996-77567ef96ff0 d1051b62-46a7-487e-b188-f742aec6d711 Remove the default security group: openstack port unset --security-group c0108624-dbd8-42bb-acaa-bbbfd32fbbae 6978e153-47dc-42d4-bdb8-63b325c77cf0 Assign the relevant security group: openstack port set --security-group 415a6ef0-e461-41fc-85f0-c92c043e5b50 6978e153-47dc-42d4-bdb8-63b325c77cf0 After that, when I ssh to the virtual machine, I can see the new interface and indeed also ping/ssh to the system using that IP address. However, its not reachable from outside the virtual machine What could I be missing here? Regards, William

2 1

[Neutron] Bug report Jan 6 to Jan 12
by Rodolfo Alonso Hernandez 13 Jan '25

13 Jan '25

Hello Neutrinos: This is the bug report from the past week. *HIGH* * DVR flow is not installed for allowed address pairs mac in DVR_TO_SRC_MAC table=1 (https://bugs.launchpad.net/neutron/+bug/2093248) Patch proposed: https://review.opendev.org/c/openstack/neutron/+/938657 * [OVN] Port unbinding operation receives rogue port UP event too ( https://bugs.launchpad.net/neutron/+bug/2093327) *Not assigned*. * [ovn-octavia-provider] first request is stuck on OVN txn ( https://bugs.launchpad.net/neutron/+bug/2093347) Patch proposed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/938797 *MEDIUM* * OVS Manager creation is failing at ovs-agent startup causing that values like inactivity_timeout are not set ( https://bugs.launchpad.net/neutron/+bug/2093433) Assigned. *LOW* * Missing Documentation for vpnaas with ovn ( https://bugs.launchpad.net/neutron/+bug/2093317) *Not assigned*. Regards.

1 0

[all][gate] Bandit version 1.8.1 issues
by Brian Haley 13 Jan '25

13 Jan '25

Hi Stackers, I was re-checking some simple Neutron changes and noticed our pep8 job started failing running bandit. Seems there was a new version (1.8.1) just released and it's generating false-positives [0]. I filed a bug against it at [1], and for now I'll just work around it by ignoring B106 warnings in Neutron with [2]. Didn't know if we wanted to pin/denylist it while the issue is addressed as I'm assuming others projects will start seeing the same failure. -Brian [0] https://bugs.launchpad.net/neutron/+bug/2093849 [1] https://github.com/PyCQA/bandit/issues/1216 [2] https://review.opendev.org/c/openstack/neutron/+/939004

1 1

keystone concepts
by Budai Laszlo 13 Jan '25

13 Jan '25

Dear all, reading the keystone documentation (https://docs.openstack.org/keystone/latest/getting-started/architecture.html) we can see the following sentence: "|Projects| represent the base unit of |ownership| in OpenStack, in that all resources in OpenStack should be owned by a specific project." What are "resources" in the previous statement? Thank you in advance, Laszlo

2 3

keystone concepts
by Budai Laszlo 13 Jan '25

13 Jan '25

Dear all, reading the keystone documentation (https://docs.openstack.org/keystone/latest/getting-started/architecture.html) we can see the following sentence: "|Projects| represent the base unit of |ownership| in OpenStack, in that all resources in OpenStack should be owned by a specific project." What are "resources" in the previous statement? Thank you in advance, Laszlo

1 0

[release] OpenStack 2025.2 'F' Release Naming Poll
by Allison Price 11 Jan '25

11 Jan '25

Hi everyone, Please join me in voting to name OpenStack 2025.2, starting with the letter ‘F’! https://civs1.civs.us/cgi-bin/vote.pl?id=E_7207f29f7e2e222d&akey=f80b0a4bbd… The deadline to vote is Friday, January 10 at 1700 UTC. Cheers, Allison

1 1

Step 2 of HA Wallaby overcloud fails with "ERROR: Can't run container mysql_wait_bundle"
by davorin.mikulic＠gmail.com 11 Jan '25

11 Jan '25

Dear Openstack community, I am trying to deploy Wallaby overcloud on HP Gen9 machines with CentOS 9 installed on them. The very similar setup with CentOS 8 deployed and worked successfully. Deploy fails on step 2 with: 2025-01-07 12:07:31.485297 | | WARNING | ERROR: Can't run container mysql_wait_bundle stderr: + STEP=2 + TAGS=file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user + CONFIG='include tripleo::profile::pacemaker::database::mysql_bundle' + EXTRA_ARGS= + '[' -d /tmp/puppet-etc ']' + cp -a /tmp/puppet-etc/devices /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet + echo '{"step": 2}' + export FACTER_deployment_type=containers + FACTER_deployment_type=containers + set +e + puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user -e 'noop_resource('\''package'\''); include tripleo::profile::pacemaker::database::mysql_bundle' Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 (file: /etc/puppet/hiera.yaml) Warning: Undefined variable '::deploy_config_name'; (file & line not available) Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html (file & line not available) Warning: Unknown variable: '::pacemaker::pcs_010'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 159, column: 6) Error: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0] Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0] Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[root@localhost]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/File[/root/.my.cnf]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@127.0.0.1]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@::1]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@localhost]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@%]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@localhost.localdomain]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@localhost.localdomain]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@overcloud-controller-0.tb27.cld]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@overcloud-controller-0.tb27.cld]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@overcloud-controller-0]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@overcloud-controller-0]: Skipping because of failed dependencies Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_database[test]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[root@%]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[clustercheck@localhost]: Skipping because of failed dependencies Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Mysql_database[cinder]: Skipping because of failed dependencies Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Mysql_database[glance]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Skipping because of failed dependencies Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Mysql_database[keystone]: Skipping because of failed dependencies Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Mysql_database[ovs_neutron]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Mysql_database[nova]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova_cell0]/Mysql_database[nova_cell0]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Mysql_database[nova_api]: Skipping because of failed dependencies Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Mysql_database[placement]: Skipping because of failed dependencies Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_user[cinder@%]: Skipping because of failed dependencies Error: Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_grant[cinder@%/cinder.*]: Skipping because of failed dependencies Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_user[glance@%]: Skipping because of failed dependencies Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_grant[glance@%/glance.*]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Skipping because of failed dependencies Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_user[keystone@%]: Skipping because of failed dependencies Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_grant[keystone@%/keystone.*]: Skipping because of failed dependencies Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_user[neutron@%]: Skipping because of failed dependencies Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_grant[neutron@%/ovs_neutron.*]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_user[nova@%]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_grant[nova@%/nova.*]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova_cell0]/Openstacklib::Db::Mysql::Host_access[nova_cell0_%]/Mysql_grant[nova@%/nova_cell0.*]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Openstacklib::Db::Mysql::Host_access[nova_api_%]/Mysql_user[nova_api@%]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Openstacklib::Db::Mysql::Host_access[nova_api_%]/Mysql_grant[nova_api@%/nova_api.*]: Skipping because of failed dependencies Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Openstacklib::Db::Mysql::Host_access[placement_%]/Mysql_user[placement@%]: Skipping because of failed dependencies Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Openstacklib::Db::Mysql::Host_access[placement_%]/Mysql_grant[placement@%/placement.*]: Skipping because of failed dependencies + rc=6 + set -e + set +ux 2025-01-07 12:07:31.488412 | | WARNING | ERROR: Can't run container rabbitmq_wait_bundle stderr: + STEP=2 + TAGS=file,file_line,concat,augeas,rabbitmq_policy,rabbitmq_user,rabbitmq_ready + CONFIG='include tripleo::profile::pacemaker::rabbitmq_bundle' + EXTRA_ARGS= + '[' -d /tmp/puppet-etc ']' + cp -a /tmp/puppet-etc/devices /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet + echo '{"step": 2}' + export FACTER_deployment_type=containers + FACTER_deployment_type=containers + set +e + puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,rabbitmq_policy,rabbitmq_user,rabbitmq_ready -e 'noop_resource('\''package'\''); include tripleo::profile::pacemaker::rabbitmq_bundle' Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 (file: /etc/puppet/hiera.yaml) Warning: Undefined variable '::deploy_config_name'; (file & line not available) Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html (file & line not available) Warning: Unknown variable: '::pacemaker::pcs_010'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 159, column: 6) Error: 'rabbitmqctl eval "rabbit_nodes:is_running(node(), rabbit)." | grep -q true && rabbitmqctl eval "rabbit_mnesia:is_clustered()." | grep -q true' returned 1 instead of one of [0] Error: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Exec[rabbitmq-ready]/returns: change from 'notrun' to ['0'] failed: 'rabbitmqctl eval "rabbit_nodes:is_running(node(), rabbit)." | grep -q true && rabbitmqctl eval "rabbit_mnesia:is_clustered()." | grep -q true' returned 1 instead of one of [0] Warning: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Rabbitmq_policy[ha-all@/]: Skipping because of failed dependencies + rc=6 + set -e + set +ux 2025-01-07 12:07:31.489548 | 000c29c7-8523-9a37-c0c4-00000000f883 | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | overcloud-controller-0 | error={"changed": false, "msg": "Failed containers: mysql_wait_bundle, rabbitmq_wait_bundle"} 2025-01-07 12:07:31.490512 | 000c29c7-8523-9a37-c0c4-00000000f883 | TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | overcloud-controller-0 | 0:48:23.163306 | 1989.39s PLAY RECAP ********************************************************************* localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 overcloud-cephstorage-0 : ok=273 changed=70 unreachable=0 failed=0 skipped=151 rescued=0 ignored=1 overcloud-cephstorage-1 : ok=275 changed=70 unreachable=0 failed=0 skipped=146 rescued=0 ignored=1 overcloud-cephstorage-2 : ok=275 changed=70 unreachable=0 failed=0 skipped=146 rescued=0 ignored=1 overcloud-computesriov-0 : ok=342 changed=92 unreachable=0 failed=0 skipped=163 rescued=0 ignored=1 overcloud-computesriov-1 : ok=349 changed=92 unreachable=0 failed=0 skipped=156 rescued=0 ignored=1 overcloud-controller-0 : ok=372 changed=99 unreachable=0 failed=1 skipped=217 rescued=0 ignored=1 overcloud-controller-1 : ok=392 changed=98 unreachable=0 failed=0 skipped=212 rescued=0 ignored=1 overcloud-controller-2 : ok=392 changed=98 unreachable=0 failed=0 skipped=212 rescued=0 ignored=1 undercloud : ok=76 changed=19 unreachable=0 failed=0 skipped=31 rescued=0 ignored=1 2025-01-07 12:07:31.561753 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.562447 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 1905 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.563026 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:48:23.235822 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.563565 | UUID | Info | Host | Task Name | Run Time 2025-01-07 12:07:31.564127 | 000c29c7-8523-9a37-c0c4-00000000f883 | SUMMARY | overcloud-controller-0 | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | 1989.39s 2025-01-07 12:07:31.564700 | 000c29c7-8523-9a37-c0c4-00000000bb0c | SUMMARY | undercloud | tripleo_container_image_prepare : Container image prepare | 137.67s 2025-01-07 12:07:31.565400 | 000c29c7-8523-9a37-c0c4-00000000e34e | SUMMARY | undercloud | tripleo_run_cephadm : run cephadm playbook | 59.69s 2025-01-07 12:07:31.565937 | 000c29c7-8523-9a37-c0c4-00000000e715 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for haproxy | 56.48s 2025-01-07 12:07:31.566525 | 000c29c7-8523-9a37-c0c4-00000000e747 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run pacemaker restart if the config file for the service changed | 46.63s 2025-01-07 12:07:31.567080 | 000c29c7-8523-9a37-c0c4-000000004353 | SUMMARY | overcloud-controller-1 | tripleo_firewall : Manage firewall rules | 36.87s 2025-01-07 12:07:31.567628 | 000c29c7-8523-9a37-c0c4-000000004353 | SUMMARY | overcloud-controller-0 | tripleo_firewall : Manage firewall rules | 36.75s 2025-01-07 12:07:31.568206 | 000c29c7-8523-9a37-c0c4-000000004353 | SUMMARY | overcloud-controller-2 | tripleo_firewall : Manage firewall rules | 36.65s 2025-01-07 12:07:31.568767 | 000c29c7-8523-9a37-c0c4-00000000e776 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run pacemaker restart if the config file for the service changed | 36.45s 2025-01-07 12:07:31.569335 | 000c29c7-8523-9a37-c0c4-00000000e745 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for mysql | 24.87s 2025-01-07 12:07:31.569886 | 000c29c7-8523-9a37-c0c4-00000000e774 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for oslo_messaging_rpc | 23.94s 2025-01-07 12:07:31.570454 | 000c29c7-8523-9a37-c0c4-00000000caa9 | SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 20.83s 2025-01-07 12:07:31.571025 | 000c29c7-8523-9a37-c0c4-00000000f709 | SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 20.82s 2025-01-07 12:07:31.571574 | 000c29c7-8523-9a37-c0c4-00000000be26 | SUMMARY | overcloud-controller-2 | Pre-fetch all the containers | 15.90s 2025-01-07 12:07:31.572173 | 000c29c7-8523-9a37-c0c4-00000000bdad | SUMMARY | overcloud-controller-1 | Pre-fetch all the containers | 15.88s 2025-01-07 12:07:31.572740 | 000c29c7-8523-9a37-c0c4-00000000bdad | SUMMARY | overcloud-controller-0 | Pre-fetch all the containers | 15.68s 2025-01-07 12:07:31.573331 | 000c29c7-8523-9a37-c0c4-0000000024d7 | SUMMARY | overcloud-computesriov-1 | tripleo_hieradata : Render hieradata from template | 11.32s 2025-01-07 12:07:31.573885 | 000c29c7-8523-9a37-c0c4-0000000024d7 | SUMMARY | overcloud-computesriov-0 | tripleo_hieradata : Render hieradata from template | 11.14s 2025-01-07 12:07:31.574467 | 000c29c7-8523-9a37-c0c4-00000000c4a0 | SUMMARY | overcloud-computesriov-1 | Wait for puppet host configuration to finish | 11.07s 2025-01-07 12:07:31.575030 | 000c29c7-8523-9a37-c0c4-00000000c452 | SUMMARY | overcloud-computesriov-0 | Wait for puppet host configuration to finish | 10.98s 2025-01-07 12:07:31.575572 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.576197 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.576756 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.577328 | The following node(s) had failures: overcloud-controller-0 2025-01-07 12:07:31.577879 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:32.979 180292 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleogr2nbip5 ] cleaned up 2025-01-07 12:07:32.979 180292 ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:32.981 180292 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh Host 10.100.27.127 not found in /home/stack/.ssh/known_hosts Overcloud Endpoint: http://10.100.27.127:5000 Overcloud Horizon Dashboard URL: http://10.100.27.127:80/dashboard Overcloud rc file: /home/stack/overcloud-deploy/overcloud/overcloudrc and /home/stack/overcloudrc Overcloud Deployed with error 2025-01-07 12:07:33.690 180292 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Stopping ephemeral heat. 2025-01-07 12:07:33.774 180292 INFO tripleoclient.heat_launcher [-] Pod does not exist: ephemeral-heat 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args) 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args) 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/cliff/command.py", line 185, in run 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1331, in take_action 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.set_deployment_status( 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.force_reraise() 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise self.value 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1303, in take_action 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.config_download( 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/workflows/deployment.py", line 386, in config_download 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud utils.run_ansible_playbook( 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 775, in run_ansible_playbook 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise RuntimeError(err_msg) 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 2025-01-07 12:07:36.434 180292 ERROR openstack [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:36.437 180292 INFO osc_lib.shell [-] END return value: 1 I can see the following logs on the failed controller-0 node: [root@overcloud-controller-0 log]# less /var/log/containers/rabbitmq/log/crash.log ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:29:03 =ERROR REPORT==== ** Generic server rabbit_disk_monitor terminating ** Last message in was update ** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld",50000000,1173484339200,100,10000,#Ref<0.112052652.1425801217.106754>,false,true,10,120000} ** Reason for termination == ** {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]} 2025-01-07 10:29:03 =CRASH REPORT==== crasher: initial call: rabbit_disk_monitor:init/1 pid: <0.309.0> registered_name: rabbit_disk_monitor exception exit: {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]} ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.301.0>] message_queue_len: 0 messages: [] links: [<0.308.0>] dictionary: [] trap_exit: false status: running heap_size: 2586 stack_size: 27 reductions: 314108420 neighbours: 2025-01-07 10:29:03 =SUPERVISOR REPORT==== Supervisor: {local,rabbit_disk_monitor_sup} Context: child_terminated Reason: {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"} Offender: [{pid,<0.309.0>},{name,rabbit_disk_monitor},{mfargs,{rabbit_disk_monitor,start_link,[50000000]}},{restart_type,{transient,1}},{shutdown,30000},{child_type,worker}] less /var/log/containers/rabbitmq/rabbit(a)overcloud-controller-0.internalapi.tb27.cld.log 2024-12-19 19:18:44.490 [info] <0.9.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_web_dispatch * rabbitmq_management_agent 2025-01-07 10:28:44.393 [error] <0.11277.105> ** Connection attempt from disallowed node 'rabbitmqcli-169019-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:44.404 [error] <0.11280.105> ** Connection attempt from disallowed node 'rabbitmqcli-169019-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:45.167 [error] <0.11284.105> ** Connection attempt from disallowed node 'rabbitmqcli-169251-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:45.179 [error] <0.11286.105> ** Connection attempt from disallowed node 'rabbitmqcli-169251-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.282 [error] <0.11282.105> ** Connection attempt from disallowed node 'rabbitmqcli-169491-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.293 [error] <0.11293.105> ** Connection attempt from disallowed node 'rabbitmqcli-169491-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.779 [error] <0.11298.105> ** Connection attempt from disallowed node 'rabbitmqcli-169710-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.788 [error] <0.11300.105> ** Connection attempt from disallowed node 'rabbitmqcli-169710-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:52.948 [error] <0.11303.105> ** Connection attempt from disallowed node 'rabbitmqcli-169935-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:52.958 [error] <0.11312.105> ** Connection attempt from disallowed node 'rabbitmqcli-169935-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:54.824 [error] <0.11317.105> ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:54.837 [error] <0.11318.105> ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55.645 [error] <0.11325.105> ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55.655 [error] <0.11327.105> ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56.160 [error] <0.11328.105> ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56.171 [error] <0.11334.105> ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit(a)overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:29:03.051 [error] <0.309.0> ** Generic server rabbit_disk_monitor terminating ** Last message in was update ** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld",50000000,1173484339200,100,10000,#Ref<0.112052652.1425801217.106754>,f alse,true,10,120000} ** Reason for termination == ** {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"sr c/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/r abbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,ini t_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]} 2025-01-07 10:29:03.051 [error] <0.309.0> CRASH REPORT Process rabbit_disk_monitor with 0 neighbours exited with reason: {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overclou d-controller-0.internalapi.tb27.cld: No such file or directory\n"} in rabbit_disk_monitor:parse_free_unix/1 line 272 2025-01-07 10:29:03.051 [error] <0.308.0> Supervisor rabbit_disk_monitor_sup had child rabbit_disk_monitor started with rabbit_disk_monitor:start_link(50000000) at <0.309.0> exit with reason {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"} in context child_terminated 2025-01-07 10:29:03.059 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq /mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 10 2025-01-07 10:31:03.068 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq /mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 9 2025-01-07 10:33:03.076 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq /mnesia/rabbit(a)overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 8 [root@overcloud-controller-0 ~]# pcs status --full Cluster name: tripleo_cluster Cluster Summary: * Stack: corosync (Pacemaker is running) * Current DC: overcloud-controller-1 (2) (version 2.1.9-1.el9-49aab9983) - partition with quorum * Last updated: Tue Jan 7 11:49:48 2025 on overcloud-controller-0 * Last change: Tue Jan 7 10:33:11 2025 by root via root on overcloud-controller-0 * 9 nodes configured * 26 resource instances configured (1 BLOCKED from further action due to failure) Node List: * GuestNode galera-bundle-0@overcloud-controller-2: online * GuestNode galera-bundle-1@overcloud-controller-0: online * GuestNode galera-bundle-2@overcloud-controller-1: online * Node overcloud-controller-0 (1): online, feature set 3.19.6 * Node overcloud-controller-1 (2): online, feature set 3.19.6 * Node overcloud-controller-2 (3): online, feature set 3.19.6 * GuestNode rabbitmq-bundle-0@overcloud-controller-2: online * GuestNode rabbitmq-bundle-1@overcloud-controller-0: online * GuestNode rabbitmq-bundle-2@overcloud-controller-1: online Full List of Resources: * ip-10.100.21.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-0 * ip-10.100.27.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-1 * ip-10.100.22.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-2 * ip-10.100.24.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-0 * ip-10.100.25.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-1 * Container bundle set: haproxy-bundle [cluster.common.tag/haproxy:pcmklatest]: * Replica[0] * haproxy-bundle-podman-0 (ocf:heartbeat:podman): Started overcloud-controller-0 * Replica[1] * haproxy-bundle-podman-1 (ocf:heartbeat:podman): Started overcloud-controller-1 * Replica[2] * haproxy-bundle-podman-2 (ocf:heartbeat:podman): Started overcloud-controller-2 * Container bundle set: galera-bundle [cluster.common.tag/mariadb:pcmklatest]: * Replica[0] * galera (ocf:heartbeat:galera): FAILED Promoted galera-bundle-0 (blocked) * galera-bundle-podman-0 (ocf:heartbeat:podman): Started overcloud-controller-2 * galera-bundle-0 (ocf:pacemaker:remote): Started overcloud-controller-2 * Replica[1] * galera (ocf:heartbeat:galera): Unpromoted galera-bundle-1 * galera-bundle-podman-1 (ocf:heartbeat:podman): Started overcloud-controller-0 * galera-bundle-1 (ocf:pacemaker:remote): Started overcloud-controller-0 * Replica[2] * galera (ocf:heartbeat:galera): Unpromoted galera-bundle-2 * galera-bundle-podman-2 (ocf:heartbeat:podman): Started overcloud-controller-1 * galera-bundle-2 (ocf:pacemaker:remote): Started overcloud-controller-1 * Container bundle set: rabbitmq-bundle [cluster.common.tag/rabbitmq:pcmklatest]: * Replica[0] * rabbitmq (ocf:heartbeat:rabbitmq-cluster): Started rabbitmq-bundle-0 * rabbitmq-bundle-podman-0 (ocf:heartbeat:podman): Started overcloud-controller-2 * rabbitmq-bundle-0 (ocf:pacemaker:remote): Started overcloud-controller-2 * Replica[1] * rabbitmq (ocf:heartbeat:rabbitmq-cluster): Started rabbitmq-bundle-1 * rabbitmq-bundle-podman-1 (ocf:heartbeat:podman): Started overcloud-controller-0 * rabbitmq-bundle-1 (ocf:pacemaker:remote): Started overcloud-controller-0 * Replica[2] * rabbitmq (ocf:heartbeat:rabbitmq-cluster): Started rabbitmq-bundle-2 * rabbitmq-bundle-podman-2 (ocf:heartbeat:podman): Started overcloud-controller-1 * rabbitmq-bundle-2 (ocf:pacemaker:remote): Started overcloud-controller-1 Node Attributes: * Node: overcloud-controller-0 (1): * galera-last-committed : -1 * galera-no-grastate : true * galera-role : true * haproxy-role : true * rabbitmq-role : true * rmq-node-attr-last-known-rabbitmq : rabbit(a)overcloud-controller-0.internalapi.tb27.cld * rmq-node-attr-rabbitmq : rabbit(a)overcloud-controller-0.internalapi.tb27.cld * Node: overcloud-controller-1 (2): * galera-last-committed : -1 * galera-no-grastate : true * galera-role : true * haproxy-role : true * rabbitmq-role : true * rmq-node-attr-last-known-rabbitmq : rabbit(a)overcloud-controller-1.internalapi.tb27.cld * rmq-node-attr-rabbitmq : rabbit(a)overcloud-controller-1.internalapi.tb27.cld * Node: overcloud-controller-2 (3): * galera-bootstrap : true * galera-role : true * haproxy-role : true * master-galera : 100 * rabbitmq-role : true * rmq-node-attr-last-known-rabbitmq : rabbit(a)overcloud-controller-2.internalapi.tb27.cld * rmq-node-attr-rabbitmq : rabbit(a)overcloud-controller-2.internalapi.tb27.cld Migration Summary: * Node: galera-bundle-0@overcloud-controller-2: * galera: migration-threshold=1000000 fail-count=1 last-failure='Tue Jan 7 10:32:38 2025' Failed Resource Actions: * galera_promote_0 on galera-bundle-0 'error' (1): call=116, status='complete', exitreason='Failed initial monitor action', last-rc-change='Tue Jan 7 10:32:28 2025', queued=0ms, exec=10603ms Tickets: PCSD Status: overcloud-controller-0: Online overcloud-controller-1: Online overcloud-controller-2: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled Does someone perhaps knows where the issue lies? Thank you very much!

3 4

[release] Release countdown for week R-11, Jan 13 - 17
by Elõd Illés 10 Jan '25

10 Jan '25

Development Focus ----------------- We are now past the Epoxy-2 milestone, and entering the last development phase of the cycle. Teams should be focused on implementing planned work for the cycle. Now is a good time to review those plans and reprioritize anything if needed based on the what progress has been made and what looks realistic to complete in the next few weeks. General Information ------------------- Looking ahead to the end of the release cycle, please be aware of the feature freeze dates. Those vary depending on deliverable type: * General libraries (except client libraries) need to have their last feature release before Non-client library freeze (February 20th, 2025). Their stable branches are cut early. * Client libraries (think python-*client libraries) need to have their last feature release before Client library freeze (February 27th, 2025) * Deliverables following a cycle-with-rc model (that would be most services) observe a Feature freeze on that same date, February 27th, 2025. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. After feature freeze, cycle-with-rc deliverables need to produce a first release candidate (and a stable branch) before RC1 deadline (March 13th, 2025) * Deliverables following cycle-with-intermediary model can release as necessary, but in all cases before Final RC deadline (March 27th, 2025) Upcoming Deadlines & Dates -------------------------- Non-client library freeze: February 20th, 2025 (R-6 week) Client library freeze: February 27th, 2025 (R-5 week) Epoxy-3 milestone: February 27th, 2025 (R-5 week) Előd Illés irc: elodilles @ #openstack-release

1 0

Openstack kolla networking setup
by matek7323＠interia.pl 10 Jan '25

10 Jan '25

I am setting up an Openstack Kolla environment with the goal of moving all virtualizations into one place At the moment I have 3 management controllers and 5 compute node for virtualization + CEPH I have there such settings as eth0 management network 172.16.50.0 and eth1 as external neutron 91.1.1.0/24 as public address And now I want to achieve two things: 1) How to make the vmka get the public address directly without natting it on the principle. Private address --> 10.224.1.15 --> public 91.1.1.15 2) Leave the option of nating after through the vrouter so that the machine assigns a private address and then a public IP address To all these servers for now the situation looks like this Port eth0 = vlan 750 access on the switch - MGMT port eth1 = vlan 888 access on switch - VLAN to operator so that I can broadcast public IP address network: version: 2 ethernets: eth0: match: macaddress: “***********” addresses: - “172.16.50.11/24” nameservers: addresses: - 8.8.8.8 set-name: eth0 routes: - to: default via: 172.16.50.254 eth1: match: macaddress: “**********” dhcp4: false dhcp6: false set-name: eth1 Each of the servers has the same config and now how to configure this to adjust the config of the coll, the servers as well as the openstack to be able to work natively and on the principle of assigning IP addresses directly to the machine by vrouter Translated with DeepL.com (free version)

1 0

[cinder][kubernetes] pvc stuck in reserved state
by Eugen Block 10 Jan '25

10 Jan '25

Hi, we have a couple of k8s clusters in our openstack environment (Victoria) and are facing some problems when pods get moved to different worker nodes. The PVCs get stuck in "reserved" state, and they only get attached after resetting the volumes to "available" state. First I thought it was due to the missing multiattach volume type which seems to work fine in our customer's cluster. I also found the "duplicate connector" error messages in the nova-compute.log: > Unable to update attachment.(Invalid volume: duplicate connectors > detected on volume But after adding the multiattach volume type and retyping the affected volumes, the PVCs still get stuck in "reserved" state while I don't see any error messages on the hypervisors anymore. So it seems that something has changed/improved, but now I don't have a trace where to look next. Could this be a cinder-csi driver issue or something? Our cinder backend is ceph (Pacific). Not sure which information could help so please ask for anything I can provide. Any pointers are highly appreciated! Thanks! Eugen

1 2

Third Party CI using Zuul v3 and nodepool - request for set up
by Roquesalane, Jean-Pierre 10 Jan '25

10 Jan '25

Hello OpenStack lovers! First, let me wish you all the best for this upcoming year. All our Cis are currently broken and we recently lost the team responsible for maintaining it. Unfortunately, there is a very poor documentation, and I guess we must rebuild something. I'm currently looking at Software Factory but can't get my hands on a starting point. I have a system deployed but would appreciate if you can lead me to people who have configured Zuul and nodepool. Our previous system was based on Jenkins and devstack-gate as a side note. I'll document the whole process as a help for future users. Thank you. Jean-Pierre Roquesalane SW System Sr Principal Engineer - OpenStack Champion Dell Technologies | Security and Ecosystem Software Mobile +33 6 21 86 79 04<tel:1-613-314-8106> Email : jeanpierre.roquesalane(a)dell.com<mailto:jeanpierre.roquesalane@dell.com> [cid:image001.png@01DB62C5.98BB1B40] Dell SAS, Si?ge Social 1 rond point Benjamin Franklin 34000 Montpellier, France. Capital 1,946,545.26 Euros. RCS Montpellier 351 528 229 - APE 4651Z. TVA Intracommunautaire FR 20 351 528 229. SIRET 351 528 229 00088. Vat Numbers: FR 20351528229 (France) / IT00001709997 (Italy) / ESN0012622G (Spain) Internal Use - Confidential

1 0

LXC containers
by engineer2024 10 Jan '25

10 Jan '25

I am trying to setup OSA branch 29.2.0 caracal. Whenever I try creating lxc containers using setup-hosts playbook, the new containers of version ubuntu 22 , as is the host, are missing openssh-server packages. The file /etc/ansible/roles/openstack_hosts/vars/ubuntu-22.04.yml, doesn't mention the ssh package in the distro packages. How to get ssh pkgs installed in the lxc containers ? Please help with this issue

2 1

LXC networking
by engineer2024 10 Jan '25

10 Jan '25

I have observed that in the OSA setups, on the controller nodes, each lxc container has two networks inside , eth0, eth1 interfaces. Ideally eth1 should be attached to br-mgmt bridge via veth pair. When I do 'ip link show type veth' on the host node, it is listing both the veth interfaces for each lxc container, but it is not listing the other member of that veth pair. --- # ip link show type veth 9f9add59_eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxcbr0 state UP mode DEFAULT group default qlen 1000 link/ether fe:b6:db:27:e4:a7 brd ff:ff:ff:ff:ff:ff link-netnsid 0 9f9add59_eth1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-mgmt state UP mode DEFAULT group default qlen 1000 link/ether fe:a8:30:7a:ac:3f brd ff:ff:ff:ff:ff:ff link-netnsid 0 --- --- # ip link show type veth | grep 9f9add59 72: 9f9add59_eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxcbr0 state UP mode DEFAULT group default qlen 1000 73: 9f9add59_eth1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-mgmt state UP mode DEFAULT group default qlen 1000 # --- But I can't find the other member of the veth pair, for example, for the interface 9f9add59_eth1, in the ip list. But the containers are running and reachable on the br-mgmt network. How to find out the other end of a veth pair, which is connected to the bridge ?

2 3

[blazar] reservation quota
by Winicius Allan 09 Jan '25

09 Jan '25

Is there a way to determine the quota of a project that the users can reserve? Regards.

2 1

[neutron] Proposal to deprecate and remove vlan_transparent config option
by Sławek Kapłoński 09 Jan '25

09 Jan '25

Hi, We have config option to enable vlan_transparent api extension currently. We will also have similar config option for vlan_qinq (it is now proposed in [1]). In both cases we have validation by mechanism drivers if option is or is not supported while networks are created. We can also filter out enabled API extensions by mechanism drivers, see [2] for example. So I proposed [3] to remove that config option and keep API extension to be loaded always if enabled ML2 drivers supports it. But before I will propose patch for that I would like to ask if there is maybe some use case for this config option really and maybe some of you somehow depends on it. In such case maybe we should keep it like it is now and don't remove it. Please let me know if you are using somehow this option. [1] https://review.opendev.org/c/openstack/neutron/+/937372 [2] https://github.com/openstack/neutron/blob/559672f777312d00c4358626650cc577e… [3] https://bugs.launchpad.net/neutron/+bug/2092174 -- Slawek Kaplonski Principal Software Engineer Red Hat

3 3

[Tempest] Devstack to configure tempest on a remote node
by Jakub Libosvar 09 Jan '25

09 Jan '25

Hello stackers, I have been attempting to deploy a “spine & leaf” like topology for upstream testing job of the ovn-bgp-agent [1]. The idea is to have an all-in-one devstack VM where all openstack services are running. This VM is plugged to two leaf switches respectively with its two interfaces, and each switch is connected to the “spine” switch. With the leaf switches behaving as route reflected we can validate routes are advertised to the spine. Now in order to validate that, I need to execute the tempest tests on the spine. That means I want to use devstack on the spine VM that calls OpenStack services API on the devstack all-in-one VM. The spine VM should not have any openstack service running. The problem I’m struggling with that the tempest is configured based on the ENABLED_SERVICES for the current box. I tried to hack the spine VM portion by calling configure_tempest with custom ENABLED_SERVICES from a local.sh [2] which seems to configure the tempest.conf correctly but the stack.sh script calls test-config [3] phase after local.sh is invoked, which re-configures tempest.conf back to the state as if no OpenStack services were enabled. I kinda hit the wall and ran out of ideas for hacks/workarounds with the current devstack codebase. I thought one way of solving this would be to decouple ENABLED_SERVICES into two separate variables, one being used to configure services and one for tempest purposes but before I dive into that, I wanted to ask if anybody has any experience doing something similar (configuring only tempest with devstack) or maybe there is already some variable I am not aware of. Thank you for any inputs, ideas or suggestions :) Kuba [1] https://review.opendev.org/c/openstack/ovn-bgp-agent/+/936968 [2] https://review.opendev.org/c/openstack/ovn-bgp-agent/+/936968/54/playbooks/… [3] https://opendev.org/openstack/devstack/src/branch/master/stack.sh#L1482-L15…

2 1

RDO User Survey
by Amy Marrich 09 Jan '25

09 Jan '25

Hi everyone, During today's RDO meeting we had a few topics that we need further information on in order to make decisions. As a result, we have created a 2 question survey[0] to collect the needed information. This survey will remain open for 2 weeks so we can make plans for the Epoxy release with the gathered information. The first question is in regards to CentOS Stream version(s) for the Epoxy release, CentOS Stream 9 only, 10 only, or both versions. The second question is to determine whether future RDO releases will continue to be all releases or only SLURP releases, if SLURP are the only releases folks are using. Thanks, Amy 0 - https://forms.gle/44atURvazYPXVvwZ8

1 0

[Neutron] Bug report last 2 weeks
by Rodolfo Alonso Hernandez 08 Jan '25

08 Jan '25

Hello Neutrinos: This is the bug report of the last 2 weeks. HIGH * Packet Logging Framework (OVN/ML2) ( https://bugs.launchpad.net/neutron/+bug/2092474) *Not assigned*. * with pyroute2 master dvr functional tests failing with route mismatch as returned routes are not from specific table ( https://bugs.launchpad.net/neutron/+bug/2092567) There is an open bug for pyroute2 (https://github.com/svinota/pyroute2/issues/1225) that should fix this problem. * with pyroute2 master dvr functional tests failing as pyroute.NetNS not honoring flags (https://bugs.launchpad.net/neutron/+bug/2092722) There is an open bug for pyroute2 (https://github.com/svinota/pyroute2/issues/1226) that should fix this problem. * Potential GPL violation in master branch due to GPL script embedded into Apache2 python code (https://bugs.launchpad.net/neutron/+bug/2092907) *Not assigned*. * Metadata service does not work when port security is disabled ( https://bugs.launchpad.net/neutron/+bug/2092971) *Not assigned*. MEDIUM * DeprecationWarning: create_subnetpool ( https://bugs.launchpad.net/neutron/+bug/2092657) *Not assigned*. * NoneType' object has no attribute 'registered_rules' ( https://bugs.launchpad.net/neutron/+bug/2092659) *Not assigned*. * OVN: Ports not activated on time when DPDK is used ( https://bugs.launchpad.net/neutron/+bug/2092407) Assigned to Ihar, patch proposed: https://review.opendev.org/c/openstack/neutron/+/938197. INCOMPLETE * deadlock： delete the network ( https://bugs.launchpad.net/neutron/+bug/2092983) Regards.

3 5

check Nova keyring file
by Michel Niyoyita 08 Jan '25

08 Jan '25

Hello Team, I am integrating Openstack 2024.1 with ceph cluster reef running on ubuntu 22.05, I have configured openstack global as follow: ceph_nova_keyring: "ceph.client.nova.keyring" ceph_nova_user: "nova" ceph_nova_pool_name: "vms" I have copied nova keyring too from /etc/ceph to /etc/kolla/config/nova ; /etc/kolla/config/nova$ ls -l total 16 -rw-r--r-- 1 stack stack 64 Jan 7 13:12 ceph.client.cinder.keyring -rw-r--r-- 1 stack stack 62 Jan 7 15:11 ceph.client.nova.keyring -rw-r--r-- 1 stack stack 283 Jan 7 14:51 ceph.conf -rw-rw-r-- 1 stack stack 182 Jan 7 15:14 nova.conf (kolla-ansible) stack@open-deployment:/etc/kolla/config/nova$ I have created nova user in ceph : sudo ceph auth get-or-create client.nova mon 'profile rbd' osd 'profile rbd pool=vms, profile rbd pool=images' mgr 'profile rbd pool=vms' -o /etc/ceph/ceph.client.nova.keyring But while deploying I am facing the following issue : TASK [nova-cell : Check nova keyring file] ******************************************************************************************** fatal: [compute1 -> localhost]: FAILED! => {"msg": "No file was found when using first_found."} fatal: [compute2 -> localhost]: FAILED! => {"msg": "No file was found when using first_found."} fatal: [compute3 -> localhost]: FAILED! => {"msg": "No file was found when using first_found."} the same error is occuring also when deploying allinone . I tried two different release : 2023.2 and 2024.1 , this is the ansible version installed : pip install 'ansible>=8,<9' tree /etc/kolla/config/ /etc/kolla/config/ ├── cinder │ ├── ceph.conf │ ├── cinder-backup │ │ ├── ceph.client.cinder-backup.keyring │ │ └── ceph.client.cinder.keyring │ └── cinder-volume │ └── ceph.client.cinder.keyring ├── glance │ ├── ceph.client.cinder.keyring │ ├── ceph.client.glance.keyring │ └── ceph.conf └── nova ├── ceph.client.cinder.keyring ├── ceph.client.nova.keyring └── ceph.conf Kindly help to solve the issue , it takes long time without solution. previously was not occured on 2023.1 Best Regards Michel

2 2

Nova create vm with special ipv4 and ipv6
by Alex Song (宋文平) 08 Jan '25

08 Jan '25

Hi Nova team, does nova support create vm with special ipv4 and ipv6? As port create with ipv4 and ipv6 works by: openstack port create 111 --network 10851e96-8995-4988-b819-0514b38d7353 --fixed-ip subnet=a564ac90-3ff3-47d7-8ae5-20e2666eaa3c,ip-address=172.24.4.54 --fixed-ip subnet=afa3c858-c3d3-4784-8964-2c46840b714d,ip-address=2001:db8::183 but vm create with ipv4 and ipv6 with the cmd openstack server create --flavor 1 --image 62254c92-a579-4c8a-a2f4-f2624cef8ec1 --nic net-id=10851e96-8995-4988-b819-0514b38d7353,v4-fixed-ip=172.24.4.53,v6-fixed-ip=2001:db8::188 test raise exception: openstack server create: error: argument --nic: Invalid argument net-id=10851e96-8995-4988-b819-0514b38d7353,v4-fixed-ip=172.24.4.53,v6-fixed-ip=2001:db8::188; either 'v4-fixed-ip' or 'v6-fixed-ip' should be specified but not both. So why nova doesnot support?

2 3

[ops][nova] How to make sure that on a hypervisor there is room for a VM with a specific flavor
by Massimo Sgaravatto 07 Jan '25

07 Jan '25

Dear all I have the following use case: I defined a flavor F for VMS with 10 VCPUs, 20 GB of RAM, 30 GB of root disk, 1 GPU XYZ. I use PCI passthrough to manage GPUs. I have only a XYZ GPU on my cluster (on a H hypervisor), so a VM using this flavor F must be "placed" to the compute node H (and this is not a problem: I know how to do that). H is very large: it has a lot of RAM and cores, so it can host other VMs that do not use GPUs. How can I make it so that VMs without GPU can go on H, but there must be room for a VM with flavor F (if it doesn't already exist)? Thanks, and happy new year ! Cheers, Massimo

3 3

[openstack][octavia]Resize a Load Balancer
by Nguyễn Hữu Khôi 07 Jan '25

07 Jan '25

Hello. I see that patch https://review.opendev.org/c/openstack/octavia/+/890215 Could I ask why we cannot merge it? Nguyen Huu Khoi

2 2

ERROR dpif_netlink|INFO|dpif_netlink_meter_transact OVS_METER_CMD_SET failed
by alibasafa18＠gmail.com 07 Jan '25

07 Jan '25

I have set up OpenStack using Kolla Ansible with OVS and OVN. On one compute node, meters are functioning correctly, but on another, I am encountering an issue. Both nodes use the system datapath, have the same kernel version, and run the same version of OVS. Despite this, the problem persists. Has anyone experienced a similar issue? What could be the root cause? For example, I have two compute nodes. One of them does not have any issues, but the other meter did not work . following compute meter does not working: > sudo docker exec -it openvswitch_vswitchd bash (openvswitch-vswitchd)[root@compute1 /]# lsmod | grep open openvswitch 172032 13 nsh 16384 1 openvswitch nf_conncount 24576 1 openvswitch nf_nat 49152 1 openvswitch nf_conntrack 172032 6 nf_nat,nfnetlink_cttimeout,openvswitch,nf_conntrack_netlink,nf_conncount,ip_vs nf_defrag_ipv6 24576 3 nf_conntrack,openvswitch,ip_vs libcrc32c 16384 6 nf_conntrack,nf_nat,openvswitch,btrfs,raid456,ip_vs (openvswitch-vswitchd)[root@compute1 /]# ovs-appctl dpif/show-dp-features br-int Masked set action: Yes Tunnel push pop: No Ufid: Yes Truncate action: Yes Clone action: Yes Sample nesting: 10 Conntrack eventmask: Yes Conntrack clear: Yes Max dp_hash algorithm: 0 Check pkt length action: Yes Conntrack timeout policy: Yes Explicit Drop action: No Optimized Balance TCP mode: No Conntrack all-zero IP SNAT: Yes MPLS Label add: Yes Max VLAN headers: 2 Max MPLS depth: 3 Recirc: Yes CT state: Yes CT zone: Yes CT mark: Yes CT label: Yes CT state NAT: Yes CT orig tuple: Yes CT orig tuple for IPv6: Yes IPv6 ND Extension: No (openvswitch-vswitchd)[root@compute1 /]# ovs-ofctl -OOpenFlow15 meter-features br-int OFPST_METER_FEATURES reply (OF1.5) (xid=0x2): max_meter:0 max_bands:0 max_color:0 band_types: 0 capabilities: 0 (openvswitch-vswitchd)[root@compute1 /]# ovs-vsctl --version ovs-vsctl (Open vSwitch) 3.3.2 DB Schema 8.5.0 (openvswitch-vswitchd)[root@compute1 /]# uname -r 5.15.0-86-generic (openvswitch-vswitchd)[root@compute1 /]# uptime 08:27:39 up 87 days, 19:51, 0 users, load average: 3.36, 4.58, 4.19 ====================== following compute working without issue: > sudo docker exec -it openvswitch_vswitchd bash (openvswitch-vswitchd)[root@compute2 /]# lsmod | grep open openvswitch 172032 23 nsh 16384 1 openvswitch nf_conncount 24576 1 openvswitch nf_nat 49152 1 openvswitch nf_conntrack 172032 6 nf_nat,nfnetlink_cttimeout,openvswitch,nf_conntrack_netlink,nf_conncount,ip_vs nf_defrag_ipv6 24576 3 nf_conntrack,openvswitch,ip_vs libcrc32c 16384 6 nf_conntrack,nf_nat,openvswitch,btrfs,raid456,ip_vs (openvswitch-vswitchd)[root@compute2 /]# ovs-appctl dpif/show-dp-features br-int Masked set action: Yes Tunnel push pop: No Ufid: Yes Truncate action: Yes Clone action: Yes Sample nesting: 10 Conntrack eventmask: Yes Conntrack clear: Yes Max dp_hash algorithm: 0 Check pkt length action: Yes Conntrack timeout policy: Yes Explicit Drop action: No Optimized Balance TCP mode: No Conntrack all-zero IP SNAT: Yes MPLS Label add: Yes Max VLAN headers: 2 Max MPLS depth: 3 Recirc: Yes CT state: Yes CT zone: Yes CT mark: Yes CT label: Yes CT state NAT: Yes CT orig tuple: Yes CT orig tuple for IPv6: Yes IPv6 ND Extension: No (openvswitch-vswitchd)[root@compute27 /]# ovs-ofctl -OOpenFlow15 meter-features br-int OFPST_METER_FEATURES reply (OF1.5) (xid=0x2): max_meter:200000 max_bands:1 max_color:0 band_types: drop capabilities: kbps pktps burst stats (openvswitch-vswitchd)[root@compute2 /]# ovs-vsctl --version ovs-vsctl (Open vSwitch) 3.3.2 DB Schema 8.5.0 (openvswitch-vswitchd)[root@compute2 /]# uname -r 5.15.0-86-generic (openvswitch-vswitchd)[root@compute2 /]# uptime 08:29:47 up 87 days, 19:53, 0 users, load average: 11.23, 13.13, 13.29

1 0

ERROR dpif_netlink|INFO|dpif_netlink_meter_transact OVS_METER_CMD_SET failed
by alibasafa18＠gmail.com 07 Jan '25

07 Jan '25

I have set up OpenStack using Kolla Ansible with OVS and OVN. On one compute node, meters are functioning correctly, but on another, I am encountering an issue. Both nodes use the system datapath, have the same kernel version, and run the same version of OVS. Despite this, the problem persists. Has anyone experienced a similar issue? What could be the root cause? i have both compute for example in the following. following compute node i have issue: > sudo docker exec -it openvswitch_vswitchd bash (openvswitch-vswitchd)[root@si1-compute1 /]# lsmod | grep open openvswitch 172032 13 nsh 16384 1 openvswitch nf_conncount 24576 1 openvswitch nf_nat 49152 1 openvswitch nf_conntrack 172032 6 nf_nat,nfnetlink_cttimeout,openvswitch,nf_conntrack_netlink,nf_conncount,ip_vs nf_defrag_ipv6 24576 3 nf_conntrack,openvswitch,ip_vs libcrc32c 16384 6 nf_conntrack,nf_nat,openvswitch,btrfs,raid456,ip_vs (openvswitch-vswitchd)[root@si1-compute4174 /]# ovs-appctl dpif/show-dp-features br-int Masked set action: Yes Tunnel push pop: No Ufid: Yes Truncate action: Yes Clone action: Yes Sample nesting: 10 Conntrack eventmask: Yes Conntrack clear: Yes Max dp_hash algorithm: 0 Check pkt length action: Yes Conntrack timeout policy: Yes Explicit Drop action: No Optimized Balance TCP mode: No Conntrack all-zero IP SNAT: Yes MPLS Label add: Yes Max VLAN headers: 2 Max MPLS depth: 3 Recirc: Yes CT state: Yes CT zone: Yes CT mark: Yes CT label: Yes CT state NAT: Yes CT orig tuple: Yes CT orig tuple for IPv6: Yes IPv6 ND Extension: No (openvswitch-vswitchd)[root@si1-compute1 /]# ovs-ofctl -OOpenFlow15 meter-features br-int OFPST_METER_FEATURES reply (OF1.5) (xid=0x2): max_meter:0 max_bands:0 max_color:0 band_types: 0 capabilities: 0 (openvswitch-vswitchd)[root@si1-compute1 /]# ovs-vsctl --version ovs-vsctl (Open vSwitch) 3.3.2 DB Schema 8.5.0 (openvswitch-vswitchd)[root@si1-compute1 /]# uname -r 5.15.0-86-generic (openvswitch-vswitchd)[root@si1-compute1 /]# uptime 08:27:39 up 87 days, 19:51, 0 users, load average: 3.36, 4.58, 4.19 ================================= following compute node working correctly: (openvswitch-vswitchd)[root@si1-compute2 /]# lsmod | grep open openvswitch 172032 23 nsh 16384 1 openvswitch nf_conncount 24576 1 openvswitch nf_nat 49152 1 openvswitch nf_conntrack 172032 6 nf_nat,nfnetlink_cttimeout,openvswitch,nf_conntrack_netlink,nf_conncount,ip_vs nf_defrag_ipv6 24576 3 nf_conntrack,openvswitch,ip_vs libcrc32c 16384 6 nf_conntrack,nf_nat,openvswitch,btrfs,raid456,ip_vs (openvswitch-vswitchd)[root@si1-compute2 /]# ovs-appctl dpif/show-dp-features br-int Masked set action: Yes Tunnel push pop: No Ufid: Yes Truncate action: Yes Clone action: Yes Sample nesting: 10 Conntrack eventmask: Yes Conntrack clear: Yes Max dp_hash algorithm: 0 Check pkt length action: Yes Conntrack timeout policy: Yes Explicit Drop action: No Optimized Balance TCP mode: No Conntrack all-zero IP SNAT: Yes MPLS Label add: Yes Max VLAN headers: 2 Max MPLS depth: 3 Recirc: Yes CT state: Yes CT zone: Yes CT mark: Yes CT label: Yes CT state NAT: Yes CT orig tuple: Yes CT orig tuple for IPv6: Yes IPv6 ND Extension: No (openvswitch-vswitchd)[root@si1-compute2 /]# ovs-ofctl -OOpenFlow15 meter-features br-int OFPST_METER_FEATURES reply (OF1.5) (xid=0x2): max_meter:200000 max_bands:1 max_color:0 band_types: drop capabilities: kbps pktps burst stats (openvswitch-vswitchd)[root@si1-compute2 /]# ovs-vsctl --version ovs-vsctl (Open vSwitch) 3.3.2 DB Schema 8.5.0 (openvswitch-vswitchd)[root@si1-compute2 /]# uname -r 5.15.0-86-generic (openvswitch-vswitchd)[root@si1-compute2 /]# uptime 08:29:47 up 87 days, 19:53, 0 users, load average: 11.23, 13.13, 13.29

1 0

[tc][all] OpenStack Technical Committee Weekly Meeting Agenda (2025.1/R-12)
by Goutham Pacha Ravi 07 Jan '25

07 Jan '25

Hello Stackers, Happy New Year! The OpenStack Technical Committee will resume its regular meeting schedule this week. The next weekly meeting is scheduled to take place simultaneously on Video (Zoom) and IRC (OFTC) on January 7, 2024, at 1800 UTC. Please find the meeting agenda and details for joining the meeting on the meeting wiki [1]. On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] TC Meeting Agenda, 2025-01-07: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

custom playbook for nova
by engineer2024 06 Jan '25

06 Jan '25

can anyone guide me what are the tasks required for configuring nova service in OSA, so that I want to execute only those tasks and db steps , skipping the package installation steps. So I need to run nova playbook just to configure the nova like nova.conf, populating it with relevant options, do nova db migrate and then just restart the services. Is there a way to do it ? Thanks elinux

2 2

yoga to antelope upgrade path
by Satish Patel 06 Jan '25

06 Jan '25

Folks, I am running yoga in production with ovn driver and planning to upgrade to the 2023.1 release. Are there any issues to directly go to 2023.1 before intermediate release? Or anything critical I should be aware of?

3 3

[neutron][ovn] ovn database backup and restore
by Satish Patel 05 Jan '25

05 Jan '25

Folks, I have OVN running on 3 node controllers with a RAFT clustering. What is the best way to take backup of all the databases of nb_db and sb_db and how to restore them during a full disaster situation? Is the following the correct way to take backup? ovsdb-client backup unix:///run/ovn/ovnnb_db.sock OVN_Northbound > nb-snap-$(date +%s).txt ovsdb-client backup unix:///run/ovn/ovnsb_db.sock OVN_Southbound > sb-snap-$(date +%s).txt During restore on what node we should restore and does restore work on fresh cluster or existing cluster? How do other folks handle disaster scenarios where all 3 nodes are down or required migration?

3 6

Failing at TASK [glance : Copy over multiple ceph configs for Glance]
by Karera Tony 05 Jan '25

05 Jan '25

Hello Team, I am deploying Openstack release 2024.02 with External Ceph. However I am encountering the error below. But it is failing here TASK [glance : Copy over multiple ceph configs for Glance] Please assist Regards Tony Karera

2 1

[release] Release countdown for week R-12
by Thierry Carrez 03 Jan '25

03 Jan '25

Development Focus ----------------- The epoxy-2 milestone is next week, on January 9th, 2025! Epoxy-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/epoxy/schedule.html for details. General Information ------------------- Libraries need to be released at least once per milestone period. Next week, the release team will propose releases for any library that has not been otherwise released since milestone 1. PTL's and release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well. A +1 would be appreciated, but if we do not hear anything at all by the end of the week, we will assume things are OK to proceed. Remember that non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. Next week is also the deadline to freeze the contents of the final release. All new 2025.1 Epoxy deliverables need to have a deliverable file in https://opendev.org/openstack/releases/src/branch/master/deliverables and need to have done a release by milestone-2. Changes proposing those deliverables for inclusion in 2025.1 Epoxy have been posted, please update them with an actual release request before the milestone-2 deadline if you plan on including that deliverable in 2025.1 Epoxy, or -1 if you need one more cycle to be ready. Upcoming Deadlines & Dates -------------------------- Epoxy-2 Milestone: January 9th, 2025 2025.1 Epoxy final release: April 2nd, 2025 -- Thierry Carrez (ttx)

1 0