[TripleO] overcloud node introspect failed

Vinesh N vineshnellaiappan at gmail.com
Mon Apr 26 14:53:11 UTC 2021


Thanks for your suggestion,

here are the details, 1) ipmitool with password,
                                 2) node show details
                                 3) node validate
                                 4) driver list

*(undercloud) [stack at n001 ~]$ ipmitool -I lanplus -H 10.0.40.6 -L
ADMINISTRATOR -p 623 -U admin -R 1 -N 5  -P *** lan print 1*
Set in Progress         : Set Complete
Auth Type Support       : NONE MD2 MD5 PASSWORD
Auth Type Enable        : Callback : MD2 MD5 PASSWORD
                        : User     : MD2 MD5 PASSWORD
                        : Operator : MD2 MD5 PASSWORD
                        : Admin    : MD2 MD5 PASSWORD
                        : OEM      : MD2 MD5 PASSWORD
IP Address Source       : Static Address
IP Address              : 10.0.40.6
Subnet Mask             : 255.255.0.0
MAC Address             : 0c:c4:7a:3c:c0:b9
SNMP Community String   : public
IP Header               : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled
Default Gateway IP      : 0.0.0.0
Default Gateway MAC     : 00:00:00:00:00:00
Backup Gateway IP       : 0.0.0.0
Backup Gateway MAC      : 00:00:00:00:00:00
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 1,2,3,6,7,8,11,12
Cipher Suite Priv Max   : XaaaXXaaaXXaaXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM
Bad Password Threshold  : Not Available



*(undercloud) [stack at n001 ~]$ openstack baremetal node show
23a17f0f-d683-4f90-8696-f947485900f9*
/usr/lib/python3.6/site-packages/requests/__init__.py:91:
RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't
match a supported version!
  RequestsDependencyWarning)
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                  | Value




             |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| allocation_uuid        | None




            |
| automated_clean        | None




            |
| bios_interface         | no-bios




             |
| boot_interface         | ipxe




            |
| chassis_uuid           | None




            |
| clean_step             | {}




            |
| conductor              | n001.localdomain




            |
| conductor_group        |




             |
| console_enabled        | False




             |
| console_interface      | ipmitool-socat




            |
| created_at             | 2021-04-26T08:22:49+00:00




             |
| deploy_interface       | direct




            |
| deploy_step            | {}




            |
| description            | None




            |
| driver                 | ipmi




            |
| driver_info            | {'deploy_kernel':
'file:///var/lib/ironic/httpboot/agent.kernel', 'rescue_kernel':
'file:///var/lib/ironic/httpboot/agent.kernel', 'deploy_ramdisk':
'file:///var/lib/ironic/httpboot/agent.ramdisk', 'rescue_ramdisk':
'file:///var/lib/ironic/httpboot/agent.ramdisk', 'ipmi_address':
'10.0.1.11', 'ipmi_username': 'admin', 'ipmi_password': '******',
'ipmi_port': 623} |
| driver_internal_info   | {}




            |
| extra                  | {}




            |
| fault                  | None




            |
| inspect_interface      | inspector




             |
| inspection_finished_at | None




            |
| inspection_started_at  | None




            |
| instance_info          | {}




            |
| instance_uuid          | None




            |
| last_error             | None




            |
| lessee                 | None




            |
| maintenance            | False




             |
| maintenance_reason     | None




            |
| management_interface   | ipmitool




            |
| name                   | None




            |
| network_data           | {}




            |
| network_interface      | flat




            |
| owner                  | None




            |
| power_interface        | ipmitool




            |
| power_state            | power off




             |
| properties             | {'capabilities': 'boot_option:local', 'vendor':
'supermicro'}



             |
| protected              | False




             |
| protected_reason       | None




            |
| provision_state        | manageable




            |
| provision_updated_at   | 2021-04-26T08:22:55+00:00




             |
| raid_config            | {}




            |
| raid_interface         | no-raid




             |
| rescue_interface       | agent




             |
| reservation            | None




            |
| resource_class         | baremetal




             |
| retired                | False




             |
| retired_reason         | None




            |
| storage_interface      | noop




            |
| target_power_state     | None




            |
| target_provision_state | None




            |
| target_raid_config     | {}




            |
| traits                 | []




            |
| updated_at             | 2021-04-26T08:35:27+00:00




             |
| uuid                   | 23a17f0f-d683-4f90-8696-f947485900f9




            |
| vendor_interface       | ipmitool




            |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
and here is the validate output, i could see the pxe assign ipaddress to
node and loaded the operating system.

*(undercloud) [stack at n001 ~]$ openstack baremetal node validate
23a17f0f-d683-4f90-8696-f947485900f9*
/usr/lib/python3.6/site-packages/requests/__init__.py:91:
RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't
match a supported version!
  RequestsDependencyWarning)
+------------+--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Interface  | Result | Reason


                                                               |
+------------+--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| bios       | False  | Driver ipmi does not support bios (disabled or not
implemented).

                                                               |
| boot       | False  | Cannot validate image information for node
23a17f0f-d683-4f90-8696-f947485900f9 because one or more parameters are
missing from its instance_info and insufficent information is present to
boot from a remote volume. Missing are: ['image_source', 'kernel',
'ramdisk'] |
| console    | False  | Either missing 'ipmi_terminal_port' parameter in
node's driver_info or [console]port_range is not configured

                                                                 |
| deploy     | False  | Cannot validate image information for node
23a17f0f-d683-4f90-8696-f947485900f9 because one or more parameters are
missing from its instance_info and insufficent information is present to
boot from a remote volume. Missing are: ['image_source', 'kernel',
'ramdisk'] |
| inspect    | True   |


                                                              |
| management | True   |


                                                              |
| network    | True   |


                                                              |
| power      | True   |


                                                              |
| raid       | False  | Driver ipmi does not support raid (disabled or not
implemented).

                                                               |
| rescue     | False  | Cannot validate image information for node
23a17f0f-d683-4f90-8696-f947485900f9 because one or more parameters are
missing from its instance_info and insufficent information is present to
boot from a remote volume. Missing are: ['image_source', 'kernel',
'ramdisk'] |
| storage    | True   |


                                                              |
+------------+--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

*(undercloud) [stack at n001 ~]$ openstack baremetal driver list*
/usr/lib/python3.6/site-packages/requests/__init__.py:91:
RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't
match a supported version!
  RequestsDependencyWarning)
+---------------------+------------------+
| Supported driver(s) | Active host(s)   |
+---------------------+------------------+
| idrac               | n001.localdomain |
| ilo                 | n001.localdomain |
| ipmi                | n001.localdomain |
| redfish             | n001.localdomain |
+---------------------+------------------+

-Vinesh

On Mon, Apr 26, 2021 at 6:59 PM Julia Kreger <juliaashleykreger at gmail.com>
wrote:

> Greetings,
>
> In all likelihood, the credentials are wrong for the baremetal node
> and the lock is being held by the conductor who is still trying to
> record the power state. The lock is an intentional behavior clients
> should retry if they encounter the lock. This is because BMC's often
> cannot handle concurrent requests.
>
> I would first manually verify:
>
> * That the nodes are not in maintenance state (openstack baremetal
> node show). The node last_error field may have a hint or indication to
> the actual error, but visit the next two bullet points.
> * That a power state of on or off has been recorded. If it has not
> been recorded, the supplied credentials or or access is correct.
> * If you're sure about the credentials, verify basic connectivity to
> the BMC address. Some BMCs are very particular about *how* the
> networking is configured, specifically to help limit attacks from the
> network itself.
>
> -Julia
>
>
> On Wed, Apr 21, 2021 at 7:25 PM Vinesh N <vineshnellaiappan at gmail.com>
> wrote:
> >
> > hi,
> >    i am facing an issue while introspect the bare metal nodes,
> >
> > error message
> > "4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf | 2021-04-22T01:41:32 |
> 2021-04-22T01:41:35 | Failed to set boot device to PXE: Failed to set boot
> device for node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf: Client Error for url:
> http://10.0.1.202:6385/v1/nodes/4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf/management/boot_device,
> Node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf is locked by host
> undercloud.localdomain, please retry after the current operation is
> completed"
> >
> >
> > (undercloud) [stack at undercloud ~]$ cat /etc/*release
> > CentOS Linux release 8.3.2011
> >
> > ussuri version
> >
> > (undercloud) [stack at undercloud ~]$ openstack image list
> > /usr/lib/python3.6/site-packages/requests/__init__.py:91:
> RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't
> match a supported version!
> >   RequestsDependencyWarning)
> >
> +--------------------------------------+------------------------+--------+
> > | ID                                   | Name                   | Status
> |
> >
> +--------------------------------------+------------------------+--------+
> > | 8ddcd168-cc18-4ce2-97c5-c3502ac471a4 | overcloud-full         | active
> |
> > | 8d9cfac9-400b-4570-b0b1-baeb175b16c4 | overcloud-full-initrd  | active
> |
> > | c561f1d5-41ae-4599-81ea-de2c1e74eae7 | overcloud-full-vmlinuz | active
> |
> >
> +--------------------------------------+------------------------+--------+
> >
> > Using the command to introspect the node, it was able to discover the
> node and I could provision the node boot via pxe, and load the image on the
> node. I could see the login prompt on the server, after some time of
> provision shut the node down.
> >
> > openstack overcloud node discover --range 10.0.40.5 --credentials
> admin:XXXX --introspect --provide
> >
> > /usr/lib/python3.6/site-packages/requests/__init__.py:91:
> RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't
> match a supported version!
> >   RequestsDependencyWarning)
> > Successfully probed node IP 10.0.40.5
> > Successfully registered node UUID 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf
> > /usr/lib/python3.6/site-packages/requests/__init__.py:91:
> RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't
> match a supported version!
> >   RequestsDependencyWarning)
> >
> > PLAY [Baremetal Introspection for multiple Ironic Nodes]
> ***********************
> > 2021-04-22 07:04:28.978299 | 002590fe-0d22-76eb-1a70-000000000008 |
>  TASK | Check for required inputs
> > 2021-04-22 07:04:29.002729 | 002590fe-0d22-76eb-1a70-000000000008 |
> SKIPPED | Check for required inputs | localhost | item=node_uuids
> > 2021-04-22 07:04:29.004468 | 002590fe-0d22-76eb-1a70-000000000008 |
>  TIMING | Check for required inputs | localhost | 0:00:00.069134 | 0.0
> > ....
> > ....
> > ....
> >
> > 2021-04-22 07:11:43.261714 | 002590fe-0d22-76eb-1a70-000000000016 |
>  TASK | Nodes that failed introspection
> > 2021-04-22 07:11:43.296417 | 002590fe-0d22-76eb-1a70-000000000016 |
> FATAL | Nodes that failed introspection | localhost | error={
> >     "msg": " 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf"
> > }
> > 2021-04-22 07:11:43.297359 | 002590fe-0d22-76eb-1a70-000000000016 |
>  TIMING | Nodes that failed introspection | localhost | 0:07:14.362025 |
> 0.03s
> >
> > NO MORE HOSTS LEFT
> *************************************************************
> >
> > PLAY RECAP
> *********************************************************************
> > localhost                  : ok=4    changed=1    unreachable=0
> failed=1    skipped=5    rescued=0    ignored=0
> > 2021-04-22 07:11:43.301553 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary
> Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 2021-04-22 07:11:43.302101 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total
> Tasks: 10         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 2021-04-22 07:11:43.302609 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time:
> 0:07:14.367265 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 2021-04-22 07:11:43.303162 |                                 UUID |
>  Info |       Host |   Task Name |   Run Time
> > 2021-04-22 07:11:43.303740 | 002590fe-0d22-76eb-1a70-000000000014 |
> SUMMARY |  localhost | Start baremetal introspection | 434.03s
> > 2021-04-22 07:11:43.304248 | 002590fe-0d22-76eb-1a70-000000000015 |
> SUMMARY |  localhost | Nodes that passed introspection | 0.04s
> > 2021-04-22 07:11:43.304814 | 002590fe-0d22-76eb-1a70-000000000016 |
> SUMMARY |  localhost | Nodes that failed introspection | 0.03s
> > 2021-04-22 07:11:43.305341 | 002590fe-0d22-76eb-1a70-000000000008 |
> SUMMARY |  localhost | Check for required inputs | 0.03s
> > 2021-04-22 07:11:43.305854 | 002590fe-0d22-76eb-1a70-00000000000a |
> SUMMARY |  localhost | Set node_uuids_intro fact | 0.02s
> > 2021-04-22 07:11:43.306397 | 002590fe-0d22-76eb-1a70-000000000010 |
> SUMMARY |  localhost | Check if validation enabled | 0.02s
> > 2021-04-22 07:11:43.306904 | 002590fe-0d22-76eb-1a70-000000000012 |
> SUMMARY |  localhost | Fail if validations are disabled | 0.02s
> > 2021-04-22 07:11:43.307379 | 002590fe-0d22-76eb-1a70-00000000000e |
> SUMMARY |  localhost | Set concurrency fact | 0.02s
> > 2021-04-22 07:11:43.307913 | 002590fe-0d22-76eb-1a70-00000000000c |
> SUMMARY |  localhost | Notice | 0.02s
> > 2021-04-22 07:11:43.308417 | 002590fe-0d22-76eb-1a70-000000000011 |
> SUMMARY |  localhost | Run Validations | 0.02s
> > 2021-04-22 07:11:43.308926 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End
> Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 2021-04-22 07:11:43.309423 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State
> Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 2021-04-22 07:11:43.310021 | ~~~~~~~~~~~~~~~~~~ Number of nodes which
> did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~
> > 2021-04-22 07:11:43.310545 |  The following node(s) had failures:
> localhost
> > 2021-04-22 07:11:43.311080 |
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > Ansible execution failed. playbook:
> /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run
> Status: failed, Return Code: 2
> > Exception occured while running the command
> > Traceback (most recent call last):
> >   File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line
> 34, in run
> >     super(Command, self).run(parsed_args)
> >   File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py",
> line 41, in run
> >     return super(Command, self).run(parsed_args)
> >   File "/usr/lib/python3.6/site-packages/cliff/command.py", line 187, in
> run
> >     return_code = self.take_action(parsed_args) or 0
> >   File
> "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_node.py", line
> 462, in take_action
> >     retry_timeout=parsed_args.retry_timeout,
> >   File
> "/usr/lib/python3.6/site-packages/tripleoclient/workflows/baremetal.py",
> line 193, in introspect
> >     "retry_timeout": retry_timeout,
> >   File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line
> 728, in run_ansible_playbook
> >     raise RuntimeError(err_msg)
> > RuntimeError: Ansible execution failed. playbook:
> /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run
> Status: failed, Return Code: 2
> > Ansible execution failed. playbook:
> /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run
> Status: failed, Return Code: 2
> >
> >
> > (undercloud) [stack at undercloud ~]$ openstack baremetal introspection
> list
> > /usr/lib/python3.6/site-packages/requests/__init__.py:91:
> RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't
> match a supported version!
> >   RequestsDependencyWarning)
> >
> +--------------------------------------+---------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> > | UUID                                 | Started at          | Finished
> at         | Error
>
>
>
>                                                                       |
> >
> +--------------------------------------+---------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> > | 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf | 2021-04-22T01:41:32 |
> 2021-04-22T01:41:35 | Failed to set boot device to PXE: Failed to set boot
> device for node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf: Client Error for url:
> http://10.0.1.202:6385/v1/nodes/4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf/management/boot_device,
> Node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf is locked by host
> undercloud.localdomain, please retry after the current operation is
> completed. |
> > | 3d091348-e9c7-4e99-80e3-df72d332d935 | 2021-04-21T12:36:30 |
> 2021-04-21T12:36:32 | Failed to set boot device to PXE: Failed to set boot
> device for node 3d091348-e9c7-4e99-80e3-df72d332d935: Client Error for url:
> http://10.0.1.202:6385/v1/nodes/3d091348-e9c7-4e99-80e3-df72d332d935/management/boot_device,
> Node 3d091348-e9c7-4e99-80e3-df72d332d935 is locked by host
> undercloud.localdomain, please retry after the current operation is
> completed. |
> >
> +--------------------------------------+---------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210426/d723eb77/attachment-0001.html>


More information about the openstack-discuss mailing list