[TripleO] overcloud node introspect failed
Julia Kreger
juliaashleykreger at gmail.com
Mon Apr 26 13:29:41 UTC 2021
Greetings,
In all likelihood, the credentials are wrong for the baremetal node
and the lock is being held by the conductor who is still trying to
record the power state. The lock is an intentional behavior clients
should retry if they encounter the lock. This is because BMC's often
cannot handle concurrent requests.
I would first manually verify:
* That the nodes are not in maintenance state (openstack baremetal
node show). The node last_error field may have a hint or indication to
the actual error, but visit the next two bullet points.
* That a power state of on or off has been recorded. If it has not
been recorded, the supplied credentials or or access is correct.
* If you're sure about the credentials, verify basic connectivity to
the BMC address. Some BMCs are very particular about *how* the
networking is configured, specifically to help limit attacks from the
network itself.
-Julia
On Wed, Apr 21, 2021 at 7:25 PM Vinesh N <vineshnellaiappan at gmail.com> wrote:
>
> hi,
> i am facing an issue while introspect the bare metal nodes,
>
> error message
> "4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf | 2021-04-22T01:41:32 | 2021-04-22T01:41:35 | Failed to set boot device to PXE: Failed to set boot device for node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf: Client Error for url: http://10.0.1.202:6385/v1/nodes/4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf/management/boot_device, Node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf is locked by host undercloud.localdomain, please retry after the current operation is completed"
>
>
> (undercloud) [stack at undercloud ~]$ cat /etc/*release
> CentOS Linux release 8.3.2011
>
> ussuri version
>
> (undercloud) [stack at undercloud ~]$ openstack image list
> /usr/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't match a supported version!
> RequestsDependencyWarning)
> +--------------------------------------+------------------------+--------+
> | ID | Name | Status |
> +--------------------------------------+------------------------+--------+
> | 8ddcd168-cc18-4ce2-97c5-c3502ac471a4 | overcloud-full | active |
> | 8d9cfac9-400b-4570-b0b1-baeb175b16c4 | overcloud-full-initrd | active |
> | c561f1d5-41ae-4599-81ea-de2c1e74eae7 | overcloud-full-vmlinuz | active |
> +--------------------------------------+------------------------+--------+
>
> Using the command to introspect the node, it was able to discover the node and I could provision the node boot via pxe, and load the image on the node. I could see the login prompt on the server, after some time of provision shut the node down.
>
> openstack overcloud node discover --range 10.0.40.5 --credentials admin:XXXX --introspect --provide
>
> /usr/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't match a supported version!
> RequestsDependencyWarning)
> Successfully probed node IP 10.0.40.5
> Successfully registered node UUID 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf
> /usr/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't match a supported version!
> RequestsDependencyWarning)
>
> PLAY [Baremetal Introspection for multiple Ironic Nodes] ***********************
> 2021-04-22 07:04:28.978299 | 002590fe-0d22-76eb-1a70-000000000008 | TASK | Check for required inputs
> 2021-04-22 07:04:29.002729 | 002590fe-0d22-76eb-1a70-000000000008 | SKIPPED | Check for required inputs | localhost | item=node_uuids
> 2021-04-22 07:04:29.004468 | 002590fe-0d22-76eb-1a70-000000000008 | TIMING | Check for required inputs | localhost | 0:00:00.069134 | 0.0
> ....
> ....
> ....
>
> 2021-04-22 07:11:43.261714 | 002590fe-0d22-76eb-1a70-000000000016 | TASK | Nodes that failed introspection
> 2021-04-22 07:11:43.296417 | 002590fe-0d22-76eb-1a70-000000000016 | FATAL | Nodes that failed introspection | localhost | error={
> "msg": " 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf"
> }
> 2021-04-22 07:11:43.297359 | 002590fe-0d22-76eb-1a70-000000000016 | TIMING | Nodes that failed introspection | localhost | 0:07:14.362025 | 0.03s
>
> NO MORE HOSTS LEFT *************************************************************
>
> PLAY RECAP *********************************************************************
> localhost : ok=4 changed=1 unreachable=0 failed=1 skipped=5 rescued=0 ignored=0
> 2021-04-22 07:11:43.301553 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 2021-04-22 07:11:43.302101 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 10 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 2021-04-22 07:11:43.302609 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:07:14.367265 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 2021-04-22 07:11:43.303162 | UUID | Info | Host | Task Name | Run Time
> 2021-04-22 07:11:43.303740 | 002590fe-0d22-76eb-1a70-000000000014 | SUMMARY | localhost | Start baremetal introspection | 434.03s
> 2021-04-22 07:11:43.304248 | 002590fe-0d22-76eb-1a70-000000000015 | SUMMARY | localhost | Nodes that passed introspection | 0.04s
> 2021-04-22 07:11:43.304814 | 002590fe-0d22-76eb-1a70-000000000016 | SUMMARY | localhost | Nodes that failed introspection | 0.03s
> 2021-04-22 07:11:43.305341 | 002590fe-0d22-76eb-1a70-000000000008 | SUMMARY | localhost | Check for required inputs | 0.03s
> 2021-04-22 07:11:43.305854 | 002590fe-0d22-76eb-1a70-00000000000a | SUMMARY | localhost | Set node_uuids_intro fact | 0.02s
> 2021-04-22 07:11:43.306397 | 002590fe-0d22-76eb-1a70-000000000010 | SUMMARY | localhost | Check if validation enabled | 0.02s
> 2021-04-22 07:11:43.306904 | 002590fe-0d22-76eb-1a70-000000000012 | SUMMARY | localhost | Fail if validations are disabled | 0.02s
> 2021-04-22 07:11:43.307379 | 002590fe-0d22-76eb-1a70-00000000000e | SUMMARY | localhost | Set concurrency fact | 0.02s
> 2021-04-22 07:11:43.307913 | 002590fe-0d22-76eb-1a70-00000000000c | SUMMARY | localhost | Notice | 0.02s
> 2021-04-22 07:11:43.308417 | 002590fe-0d22-76eb-1a70-000000000011 | SUMMARY | localhost | Run Validations | 0.02s
> 2021-04-22 07:11:43.308926 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 2021-04-22 07:11:43.309423 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 2021-04-22 07:11:43.310021 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~
> 2021-04-22 07:11:43.310545 | The following node(s) had failures: localhost
> 2021-04-22 07:11:43.311080 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
> Exception occured while running the command
> Traceback (most recent call last):
> File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
> super(Command, self).run(parsed_args)
> File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
> return super(Command, self).run(parsed_args)
> File "/usr/lib/python3.6/site-packages/cliff/command.py", line 187, in run
> return_code = self.take_action(parsed_args) or 0
> File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_node.py", line 462, in take_action
> retry_timeout=parsed_args.retry_timeout,
> File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/baremetal.py", line 193, in introspect
> "retry_timeout": retry_timeout,
> File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 728, in run_ansible_playbook
> raise RuntimeError(err_msg)
> RuntimeError: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
> Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
>
>
> (undercloud) [stack at undercloud ~]$ openstack baremetal introspection list
> /usr/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't match a supported version!
> RequestsDependencyWarning)
> +--------------------------------------+---------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> | UUID | Started at | Finished at | Error |
> +--------------------------------------+---------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> | 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf | 2021-04-22T01:41:32 | 2021-04-22T01:41:35 | Failed to set boot device to PXE: Failed to set boot device for node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf: Client Error for url: http://10.0.1.202:6385/v1/nodes/4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf/management/boot_device, Node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf is locked by host undercloud.localdomain, please retry after the current operation is completed. |
> | 3d091348-e9c7-4e99-80e3-df72d332d935 | 2021-04-21T12:36:30 | 2021-04-21T12:36:32 | Failed to set boot device to PXE: Failed to set boot device for node 3d091348-e9c7-4e99-80e3-df72d332d935: Client Error for url: http://10.0.1.202:6385/v1/nodes/3d091348-e9c7-4e99-80e3-df72d332d935/management/boot_device, Node 3d091348-e9c7-4e99-80e3-df72d332d935 is locked by host undercloud.localdomain, please retry after the current operation is completed. |
> +--------------------------------------+---------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
More information about the openstack-discuss
mailing list