Hi Team

I have deployed TripleO Train Undercloud and Overcloud in HA Mode. After which I Upgraded TripleO Undercloud to Ussuri Release.

While upgrading the Overcloud Nodes, there were some minor hiccups initially, but after enabling internet and enabling Centos repositories, we resolved all the issues.

After executing openstack overcloud upgrade run command for only the controllers, we get some puppet issues for pacemaker service for Controller-0 (Active DC)

The same check passed for Controller-1 and Controller-2. All the overcloud nodes have been spawned with the same image, so there must not be any issue.

Attaching the logs below: 2022-03-09 15:22:02.510128 | 525400f8-4aa7-5981-5af0-000000006a82 | TASK | Gather podman infos 2022-03-09 15:22:02.869577 | 525400f8-4aa7-5981-5af0-000000006a0f | CHANGED | Enable and start netns-placeholder | overcloud-controller-1 2022-03-09 15:22:02.873413 | 525400f8-4aa7-5981-5af0-000000006a0f | TIMING | tripleo_container_manage : Enable and start netns-placeholder | overcloud-controller-1 | 0:14:39.816795 | 0.97s 2022-03-09 15:22:02.901485 | 525400f8-4aa7-5981-5af0-000000006940 | TASK | Update container configs with new config hashes 2022-03-09 15:22:02.943620 | 525400f8-4aa7-5981-5af0-000000006940 | TIMING | tripleo_container_manage : Update container configs with new config hashes | overcloud-controller-1 | 0:14:39.887003 | 0.04s 2022-03-09 15:22:02.960739 | 1fecaf23-8a62-488a-87ab-a2ee3c307ae5 | INCLUDED | /usr/share/ansible/roles/tripleo_container_manage/tasks/puppet_config.yml | overcloud-controller-1 2022-03-09 15:22:02.988374 | 525400f8-4aa7-5981-5af0-000000006ab5 | TASK | Update config hashes for container startup configs 2022-03-09 15:22:03.380251 | 525400f8-4aa7-5981-5af0-000000006ab5 | OK | Update config hashes for container startup configs | overcloud-controller-1 2022-03-09 15:22:03.381768 | 525400f8-4aa7-5981-5af0-000000006ab5 | TIMING | tripleo_container_manage : Update config hashes for container startup configs | overcloud-controller-1 | 0:14:40.325159 | 0.39s 2022-03-09 15:22:03.409190 | 525400f8-4aa7-5981-5af0-000000006941 | TASK | Delete orphan containers from /var/lib/tripleo-config/container-startup-config/step_3 2022-03-09 15:22:03.453834 | 525400f8-4aa7-5981-5af0-000000006941 | TIMING | tripleo_container_manage : Delete orphan containers from /var/lib/tripleo-config/container-startup-config/step_3 | overcloud-controller-1 | 0:14:40.397197 | 0.04s 2022-03-09 15:22:03.470413 | 8e8040a9-b1ed-498c-becb-4a25381f487f | INCLUDED | /usr/share/ansible/roles/tripleo_container_manage/tasks/delete_orphan.yml | overcloud-controller-1 2022-03-09 15:22:03.498978 | 525400f8-4aa7-5981-5af0-000000006ad8 | TASK | Gather podman infos instead.\nTraceback (most recent call last):\n File "/usr/lib/python3.6/site-packages/pcs/resource.py", line 1039, in resource_update\n resource.getAttribute("type"),\n File "/usr/lib/python3.6/site-packages/pcs/lib/resource_agent/facade.py", line 200, in facade_from_parsed_name\n parse_metadata(name, load_metadata(self._runner, name))\n File "/usr/lib/python3.6/site-packages/pcs/lib/resource_agent/xml.py", line 114, in load_metadata\n return _metadata_xml_to_dom(_load_metadata_xml(runner, agent_name))\n File "/usr/lib/python3.6/site-packages/pcs/lib/resource_agent/xml.py", line 52, in _load_metadata_xml\n raise UnableToGetAgentMetadata(agent_name.full_name, stderr.strip())\npcs.lib.resource_agent.error.UnableToGetAgentMetadata\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/sbin/pcs", line 11, in <module>\n load_entry_point('pcs===0.10.11.22-9d83b', 'console_scripts', 'pcs')()\n File "/usr/lib/python3.6/site-packages/pcs/app.py", line 278, in main\n utils.get_library_wrapper(), argv, utils.get_input_modifiers()\n File "/usr/lib/python3.6/site-packages/pcs/cli/common/routing.py", line 33, in _router\n return cmd_map[sub_cmd](lib, argv_next, modifiers)\n File "/usr/lib/python3.6/site-packages/pcs/cli/common/routing.py", line 33, in _router\n return cmd_map[sub_cmd](lib, argv_next, modifiers)\n File "/usr/lib/python3.6/site-packages/pcs/resource.py", line 1058, in resource_update\n [lib_ra.resource_agent_error_to_report_item(e, severity)]\n File "/usr/lib/python3.6/site-packages/pcs/cli/reports/output.py", line 38, in process_library_reports\n report_dto = report_item.to_dto()\n File "/usr/lib/python3.6/site-packages/pcs/common/reports/item.py", line 182, in to_dto\n severity=self.severity.to_dto(),\nAttributeError: 'str' object has no attribute 'to_dto'\nError: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Resource::Ocf[ovndb_servers]/Pcmk_resource[ovndb_servers]: Could not evaluate: pcmk_resource_has_changed? --force resource update ovndb_servers ocf:ovn:ovndb-servers master_ip=[fd00:fd00:fd00:2000::53] nb_master_port=6641 sb_master_port=6642 manage_northd=yes inactive_probe_interval=180000 listen_on_master_ip_only=yes meta notify=true container-attribute-target=host op start timeout=200s stop timeout=200s monitor interval=10s role=Master timeout=60s monitor interval=30s role=Slave timeout=60s bundle ovn-dbs-bundle returned error ovndb_servers. This should never happen.\nWarning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.", "stderr_lines": ["Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5", " (file: /etc/puppet/hiera.yaml)", "Warning: Undefined variable '::deploy_config_name'; ", " (file & line not available)", "Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/6.14/deprecated_language.html", " (file & line not available)", "*Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ipv6 instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-ex... https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/etc/puppet/modules/tripleo/manifests/profile/pacemaker/ovn_dbs_bundle.pp", 336]:["unknown", 1]", "* (location: /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:34:in `deprecation')", "Warning: This command is deprecated and will be removed. Please use 'pcs property config' instead.", "Warning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.", "Warning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.", "Traceback (most recent call last):", " File "/usr/lib/python3.6/site-packages/pcs/resource.py", line 1039, in resource_update", " resource.getAttribute("type"),", " File "/usr/lib/python3.6/site-packages/pcs/lib/resource_agent/facade.py", line 200, in facade_from_parsed_name", " parse_metadata(name, load_metadata(self._runner, name))", " File "/usr/lib/python3.6/site-packages/pcs/lib/resource_agent/xml.py", line 114, in load_metadata", " return _metadata_xml_to_dom(_load_metadata_xml(runner, agent_name))", " File "/usr/lib/python3.6/site-packages/pcs/lib/resource_agent/xml.py", line 52, in _load_metadata_xml", " raise UnableToGetAgentMetadata(agent_name.full_name, stderr.strip())", "pcs.lib.resource_agent.error.UnableToGetAgentMetadata", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", " File "/sbin/pcs", line 11, in <module>", " load_entry_point('pcs===0.10.11.22-9d83b', 'console_scripts', 'pcs')()", " File "/usr/lib/python3.6/site-packages/pcs/app.py", line 278, in main", " utils.get_library_wrapper(), argv, utils.get_input_modifiers()", " File "/usr/lib/python3.6/site-packages/pcs/cli/common/routing.py", line 33, in _router", " return cmd_map[sub_cmd](lib, argv_next, modifiers)", " File "/usr/lib/python3.6/site-packages/pcs/cli/common/routing.py", line 33, in _router", " return cmd_map[sub_cmd](lib, argv_next, modifiers)", " File "/usr/lib/python3.6/site-packages/pcs/resource.py", line 1058, in resource_update", " [lib_ra.resource_agent_error_to_report_item(e, severity)]", " File "/usr/lib/python3.6/site-packages/pcs/cli/reports/output.py", line 38, in process_library_reports", " report_dto = report_item.to_dto()", " File "/usr/lib/python3.6/site-packages/pcs/common/reports/item.py", line 182, in to_dto", " severity=self.severity.to_dto(),", "AttributeError: 'str' object has no attribute 'to_dto'", "Error: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Resource::Ocf[ovndb_servers]/Pcmk_resource[ovndb_servers]: Could not evaluate: pcmk_resource_has_changed? --force resource update ovndb_servers ocf:ovn:ovndb-servers master_ip=[fd00:fd00:fd00:2000::53] nb_master_port=6641 sb_master_port=6642 manage_northd=yes inactive_probe_interval=180000 listen_on_master_ip_only=yes meta notify=true container-attribute-target=host op start timeout=200s stop timeout=200s monitor interval=10s role=Master timeout=60s monitor interval=30s role=Slave timeout=60s bundle ovn-dbs-bundle returned error ovndb_servers. This should never happen.", "Warning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead."], "stdout": "Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 0.75 seconds\nNotice: Applied catalog in 30.15 seconds\nChanges:\nEvents:\n Failure: 1\n Total: 1\nResources:\n Failed: 1\n Out of sync: 1\n Skipped: 26\n Total: 35\nTime:\n Config retrieval: 0.84\n Pcmk resource: 11.39\n Last run: 1646819523\n Pcmk constraint: 2.86\n Transaction evaluation: 30.09\n Catalog application: 30.15\n Pcmk bundle: 7.09\n Pcmk property: 8.43\n Total: 30.15\nVersion:\n Config: 1646819492\n Puppet: 6.14.0", "stdout_lines": ["Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 0.75 seconds", "Notice: Applied catalog in 30.15 seconds", "Changes:", "Events:", " Failure: 1", " Total: 1", "Resources:", " Failed: 1", " Out of sync: 1", " Skipped: 26", " Total: 35", "Time:", " Config retrieval: 0.84", " Pcmk resource: 11.39", " Last run: 1646819523", " Pcmk constraint: 2.86", " Transaction evaluation: 30.09", " Catalog application: 30.15", " Pcmk bundle: 7.09", " Pcmk property: 8.43", " Total: 30.15", "Version:", " Config: 1646819492", " Puppet: 6.14.0"]} 2022-03-09 15:22:03.686083 | 525400f8-4aa7-5981-5af0-0000000067ae | FATAL | Run init bundle puppet on the host for ovn_dbs | overcloud-controller-0 | error={"changed": false, "cmd": "puppet apply --detailed-exitcodes --summarize --color=false --modulepath '/etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules' --tags 'pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ip,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::colocation' -e 'include ::tripleo::profile::base::pacemaker; include ::tripleo::profile::pacemaker::ovn_dbs_bundle'\n", "delta": "0:00:36.507726", "end": "2022-03-09 15:22:03.635619", "failed_when_result": true, "msg": "non-zero return code", "rc": 4, "start": "2022-03-09 15:21:27.127893", "stderr": "Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5\n (file: /etc/puppet/hiera.yaml)\nWarning: Undefined variable '::deploy_config_name'; \n (file & line not available)\nWarning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/6.14/deprecated_language.html%5Cn (file & line not available)\nWarning: This method is deprecated, please use match expressions with Stdlib::Compat::Ipv6 instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-ex.... at ["/etc/puppet/modules/tripleo/manifests/profile/pacemaker/ovn_dbs_bundle.pp", 336]:["unknown", 1]\n (location: /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:34:in `deprecation')\nWarning: This command is deprecated and will be removed. Please use 'pcs property config' instead.\nWarning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.\nWarning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.\nTraceback (most recent call last):\n File "/usr/lib/python3.6/site-packages/pcs/resource.py", line 1039, in resource_update\n resource.getAttribute("type"),\n File "/usr/lib/python3.6/site-packages/pcs/lib/resource_agent/facade.py", line 200, in facade_from_parsed_name\n parse_metadata(name, load_metadata(self._runner, name))\n File "/usr/lib/python3.6/site-packages/pcs/lib/resource_agent/xml.py", line 114, in load_metadata\n return _metadata_xml_to_dom(_load_metadata_xml(runner, agent_name))\n File "/usr/lib/python3.6/site-packages/pcs/lib/resource_agent/xml.py", line 52, in _load_metadata_xml\n raise UnableToGetAgentMetadata(agent_name.full_name, stderr.strip())\npcs.lib.resource_agent.error.UnableToGetAgentMetadata\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/sbin/pcs", line 11, in <module>\n load_entry_point('pcs===0.10.11.22-9d83b', 'console_scripts', 'pcs')()\n File "/usr/lib/python3.6/site-packages/pcs/app.py", line 278, in main\n utils.get_library_wrapper(), argv, utils.get_input_modifiers()\n File "/usr/lib/python3.6/site-packages/pcs/cli/common/routing.py", line 33, in _router\n return cmd_map[sub_cmd](lib, argv_next, modifiers)\n File "/usr/lib/python3.6/site-packages/pcs/cli/common/routing.py", line 33, in _router\n return cmd_map[sub_cmd](lib, argv_next, modifiers)\n File "/usr/lib/python3.6/site-packages/pcs/resource.py", line 1058, in resource_update\n [lib_ra.resource_agent_error_to_report_item(e, severity)]\n File "/usr/lib/python3.6/site-packages/pcs/cli/reports/output.py", line 38, in process_library_reports\n report_dto = report_item.to_dto()\n File "/usr/lib/python3.6/site-packages/pcs/common/reports/item.py", line 182, in to_dto\n severity=self.severity.to_dto(),\nAttributeError: 'str' object has no attribute 'to_dto'\nError: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Resource::Ocf[ovndb_servers]/Pcmk_resource[ovndb_servers]: Could not evaluate: pcmk_resource_has_changed? --force resource update ovndb_servers ocf:ovn:ovndb-servers master_ip=[fd00:fd00:fd00:2000::53] nb_master_port=6641 sb_master_port=6642 manage_northd=yes inactive_probe_interval=180000 listen_on_master_ip_only=yes meta notify=true container-attribute-target=host op start timeout=200s stop timeout=200s monitor interval=10s role=Master timeout=60s monitor interval=30s role=Slave timeout=60s bundle ovn-dbs-bundle returned error ovndb_servers. This should never happen.\nWarning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.", "stderr_lines": ["Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5", " (file: /etc/puppet/hiera.yaml)", "Warning: Undefined variable '::deploy_config_name'; ", " (file & line not available)", "Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/6.14/deprecated_language.html", " (file & line not available)", "Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ipv6 instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-ex.... at ["/etc/puppet/modules/tripleo/manifests/profile/pacemaker/ovn_dbs_bundle.pp", 336]:["unknown", 1]", " (location: /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:34:in `deprecation')", "Warning: This command is deprecated and will be removed. Please use 'pcs property config' instead.", "Warning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.", "Warning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.", "Traceback (most recent call last):", " File "/usr/lib/python3.6/site-packages/pcs/resource.py", line 1039, in resource_update", " resource.getAttribute("type"),", " File "/usr/lib/python3.6/site-packages/pcs/lib/resource_agent/facade.py", line 200, in facade_from_parsed_name", " parse_metadata(name, load_metadata(self._runner, name))", " File "/usr/lib/python3.6/site-packages/pcs/lib/resource_agent/xml.py", line 114, in load_metadata", " return _metadata_xml_to_dom(_load_metadata_xml(runner, agent_name))", " File "/usr/lib/python3.6/site-packages/pcs/lib/resource_agent/xml.py", line 52, in _load_metadata_xml", " raise UnableToGetAgentMetadata(agent_name.full_name, stderr.strip())", "pcs.lib.resource_agent.error.UnableToGetAgentMetadata", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", " File "/sbin/pcs", line 11, in <module>", " load_entry_point('pcs===0.10.11.22-9d83b', 'console_scripts', 'pcs')()", " File "/usr/lib/python3.6/site-packages/pcs/app.py", line 278, in main", " utils.get_library_wrapper(), argv, utils.get_input_modifiers()", " File "/usr/lib/python3.6/site-packages/pcs/cli/common/routing.py", line 33, in _router", " return cmd_map[sub_cmd](lib, argv_next, modifiers)", " File "/usr/lib/python3.6/site-packages/pcs/cli/common/routing.py", line 33, in _router", " return cmd_map[sub_cmd](lib, argv_next, modifiers)", " File "/usr/lib/python3.6/site-packages/pcs/resource.py", line 1058, in resource_update", " [lib_ra.resource_agent_error_to_report_item(e, severity)]", " File "/usr/lib/python3.6/site-packages/pcs/cli/reports/output.py", line 38, in process_library_reports", " report_dto = report_item.to_dto()", " File "/usr/lib/python3.6/site-packages/pcs/common/reports/item.py", line 182, in to_dto", " severity=self.severity.to_dto(),", "AttributeError: 'str' object has no attribute 'to_dto'", "Error: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Resource::Ocf[ovndb_servers]/Pcmk_resource[ovndb_servers]: Could not evaluate: pcmk_resource_has_changed? --force resource update ovndb_servers ocf:ovn:ovndb-servers master_ip=[fd00:fd00:fd00:2000::53] nb_master_port=6641 sb_master_port=6642 manage_northd=yes inactive_probe_interval=180000 listen_on_master_ip_only=yes meta notify=true container-attribute-target=host op start timeout=200s stop timeout=200s monitor interval=10s role=Master timeout=60s monitor interval=30s role=Slave timeout=60s bundle ovn-dbs-bundle returned error ovndb_servers. This should never happen.", *"Warning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead."], "stdout":* "Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 0.75 seconds\nNotice: Applied catalog in 30.15 seconds\nChanges:\nEvents:\n Failure: 1\n Total: 1\nResources:\n Failed: 1\n Out of sync: 1\n Skipped: 26\n Total: 35\nTime:\n Config retrieval: 0.84\n Pcmk resource: 11.39\n Last run: 1646819523\n Pcmk constraint: 2.86\n Transaction evaluation: 30.09\n Catalog application: 30.15\n Pcmk bundle: 7.09\n Pcmk property: 8.43\n Total: 30.15\nVersion:\n Config: 1646819492\n Puppet: 6.14.0", "stdout_lines": ["Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 0.75 seconds", "Notice: Applied catalog in 30.15 seconds", "Changes:", "Events:", " Failure: 1", " Total: 1", "Resources:", " Failed: 1", " Out of sync: 1", " Skipped: 26", " Total: 35", "Time:", " Config retrieval: 0.84", " Pcmk resource: 11.39", " Last run: 1646819523", " Pcmk constraint: 2.86", " Transaction evaluation: 30.09", " Catalog application: 30.15", " Pcmk bundle: 7.09", " Pcmk property: 8.43", " Total: 30.15", "Version:", " Config: 1646819492", " Puppet: 6.14.0"]} 2022-03-09 15:22:03.690015 | 525400f8-4aa7-5981-5af0-0000000067ae | TIMING | tripleo_ha_wrapper : Run init bundle puppet on the host for ovn_dbs | overcloud-controller-0 | 0:14:40.633394 | 36.87s 2022-03-09 15:22:05.269139 | 525400f8-4aa7-5981-5af0-000000006a82 | OK | Gather podman infos | overcloud-controller-2 2022-03-09 15:22:05.270057 | 525400f8-4aa7-5981-5af0-000000006a82 | TIMING | tripleo_container_manage : Gather podman infos | overcloud-controller-2 | 0:14:42.213453 | 2.76s 2022-03-09 15:22:05.726794 | 525400f8-4aa7-5981-5af0-000000006a83 | TASK | Delete orphan containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3 2022-03-09 15:22:05.786601 | 525400f8-4aa7-5981-5af0-000000006a83 | TIMING | Delete orphan containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3 | overcloud-controller-2 | 0:14:42.729983 | 0.06s 2022-03-09 15:22:06.038548 | 525400f8-4aa7-5981-5af0-000000006b27 | TIMING | tripleo_container_rm : include_tasks | overcloud-controller-2 | 0:14:42.981908 | 0.19s 2022-03-09 15:22:06.068012 | 525400f8-4aa7-5981-5af0-000000006942 | TASK | Create containers from /var/lib/tripleo-config/container-startup-config/step_3 2022-03-09 15:22:06.119081 | 525400f8-4aa7-5981-5af0-000000006942 | TIMING | tripleo_container_manage : Create containers from /var/lib/tripleo-config/container-startup-config/step_3 | overcloud-controller-2 | 0:14:43.062462 | 0.05s 2022-03-09 15:22:06.136762 | f2c26285-bf89-4ee3-a43f-cdf6e8722aaa | INCLUDED | /usr/share/ansible/roles/tripleo_container_manage/tasks/create.yml | overcloud-controller-2

Regards Anirudh Gupta