Greetings,
In all likelihood, the credentials are wrong for the baremetal node and the lock is being held by the conductor who is still trying to record the power state. The lock is an intentional behavior clients should retry if they encounter the lock. This is because BMC's often cannot handle concurrent requests.
I would first manually verify:
* That the nodes are not in maintenance state (openstack baremetal node show). The node last_error field may have a hint or indication to the actual error, but visit the next two bullet points. * That a power state of on or off has been recorded. If it has not been recorded, the supplied credentials or or access is correct. * If you're sure about the credentials, verify basic connectivity to the BMC address. Some BMCs are very particular about *how* the networking is configured, specifically to help limit attacks from the network itself.
-Julia
On Wed, Apr 21, 2021 at 7:25 PM Vinesh N vineshnellaiappan@gmail.com wrote:
hi, i am facing an issue while introspect the bare metal nodes,
error message "4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf | 2021-04-22T01:41:32 | 2021-04-22T01:41:35 | Failed to set boot device to PXE: Failed to set boot device for node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf: Client Error for url: http://10.0.1.202:6385/v1/nodes/4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf/managem..., Node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf is locked by host undercloud.localdomain, please retry after the current operation is completed"
(undercloud) [stack@undercloud ~]$ cat /etc/*release CentOS Linux release 8.3.2011
ussuri version
(undercloud) [stack@undercloud ~]$ openstack image list /usr/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) +--------------------------------------+------------------------+--------+ | ID | Name | Status | +--------------------------------------+------------------------+--------+ | 8ddcd168-cc18-4ce2-97c5-c3502ac471a4 | overcloud-full | active | | 8d9cfac9-400b-4570-b0b1-baeb175b16c4 | overcloud-full-initrd | active | | c561f1d5-41ae-4599-81ea-de2c1e74eae7 | overcloud-full-vmlinuz | active | +--------------------------------------+------------------------+--------+
Using the command to introspect the node, it was able to discover the node and I could provision the node boot via pxe, and load the image on the node. I could see the login prompt on the server, after some time of provision shut the node down.
openstack overcloud node discover --range 10.0.40.5 --credentials admin:XXXX --introspect --provide
/usr/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) Successfully probed node IP 10.0.40.5 Successfully registered node UUID 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf /usr/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning)
PLAY [Baremetal Introspection for multiple Ironic Nodes] *********************** 2021-04-22 07:04:28.978299 | 002590fe-0d22-76eb-1a70-000000000008 | TASK | Check for required inputs 2021-04-22 07:04:29.002729 | 002590fe-0d22-76eb-1a70-000000000008 | SKIPPED | Check for required inputs | localhost | item=node_uuids 2021-04-22 07:04:29.004468 | 002590fe-0d22-76eb-1a70-000000000008 | TIMING | Check for required inputs | localhost | 0:00:00.069134 | 0.0 .... .... ....
2021-04-22 07:11:43.261714 | 002590fe-0d22-76eb-1a70-000000000016 | TASK | Nodes that failed introspection 2021-04-22 07:11:43.296417 | 002590fe-0d22-76eb-1a70-000000000016 | FATAL | Nodes that failed introspection | localhost | error={ "msg": " 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf" } 2021-04-22 07:11:43.297359 | 002590fe-0d22-76eb-1a70-000000000016 | TIMING | Nodes that failed introspection | localhost | 0:07:14.362025 | 0.03s
NO MORE HOSTS LEFT *************************************************************
PLAY RECAP ********************************************************************* localhost : ok=4 changed=1 unreachable=0 failed=1 skipped=5 rescued=0 ignored=0 2021-04-22 07:11:43.301553 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2021-04-22 07:11:43.302101 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 10 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2021-04-22 07:11:43.302609 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:07:14.367265 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2021-04-22 07:11:43.303162 | UUID | Info | Host | Task Name | Run Time 2021-04-22 07:11:43.303740 | 002590fe-0d22-76eb-1a70-000000000014 | SUMMARY | localhost | Start baremetal introspection | 434.03s 2021-04-22 07:11:43.304248 | 002590fe-0d22-76eb-1a70-000000000015 | SUMMARY | localhost | Nodes that passed introspection | 0.04s 2021-04-22 07:11:43.304814 | 002590fe-0d22-76eb-1a70-000000000016 | SUMMARY | localhost | Nodes that failed introspection | 0.03s 2021-04-22 07:11:43.305341 | 002590fe-0d22-76eb-1a70-000000000008 | SUMMARY | localhost | Check for required inputs | 0.03s 2021-04-22 07:11:43.305854 | 002590fe-0d22-76eb-1a70-00000000000a | SUMMARY | localhost | Set node_uuids_intro fact | 0.02s 2021-04-22 07:11:43.306397 | 002590fe-0d22-76eb-1a70-000000000010 | SUMMARY | localhost | Check if validation enabled | 0.02s 2021-04-22 07:11:43.306904 | 002590fe-0d22-76eb-1a70-000000000012 | SUMMARY | localhost | Fail if validations are disabled | 0.02s 2021-04-22 07:11:43.307379 | 002590fe-0d22-76eb-1a70-00000000000e | SUMMARY | localhost | Set concurrency fact | 0.02s 2021-04-22 07:11:43.307913 | 002590fe-0d22-76eb-1a70-00000000000c | SUMMARY | localhost | Notice | 0.02s 2021-04-22 07:11:43.308417 | 002590fe-0d22-76eb-1a70-000000000011 | SUMMARY | localhost | Run Validations | 0.02s 2021-04-22 07:11:43.308926 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2021-04-22 07:11:43.309423 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2021-04-22 07:11:43.310021 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2021-04-22 07:11:43.310545 | The following node(s) had failures: localhost 2021-04-22 07:11:43.311080 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2 Exception occured while running the command Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 187, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_node.py", line 462, in take_action retry_timeout=parsed_args.retry_timeout, File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/baremetal.py", line 193, in introspect "retry_timeout": retry_timeout, File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 728, in run_ansible_playbook raise RuntimeError(err_msg) RuntimeError: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2 Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
(undercloud) [stack@undercloud ~]$ openstack baremetal introspection list /usr/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) +--------------------------------------+---------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | UUID | Started at | Finished at | Error | +--------------------------------------+---------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf | 2021-04-22T01:41:32 | 2021-04-22T01:41:35 | Failed to set boot device to PXE: Failed to set boot device for node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf: Client Error for url: http://10.0.1.202:6385/v1/nodes/4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf/managem..., Node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf is locked by host undercloud.localdomain, please retry after the current operation is completed. | | 3d091348-e9c7-4e99-80e3-df72d332d935 | 2021-04-21T12:36:30 | 2021-04-21T12:36:32 | Failed to set boot device to PXE: Failed to set boot device for node 3d091348-e9c7-4e99-80e3-df72d332d935: Client Error for url: http://10.0.1.202:6385/v1/nodes/3d091348-e9c7-4e99-80e3-df72d332d935/managem..., Node 3d091348-e9c7-4e99-80e3-df72d332d935 is locked by host undercloud.localdomain, please retry after the current operation is completed. | +--------------------------------------+---------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+