[TripleO] overcloud node introspect failed
Vinesh N
vineshnellaiappan at gmail.com
Mon Apr 26 14:53:11 UTC 2021
Thanks for your suggestion,
here are the details, 1) ipmitool with password,
2) node show details
3) node validate
4) driver list
*(undercloud) [stack at n001 ~]$ ipmitool -I lanplus -H 10.0.40.6 -L
ADMINISTRATOR -p 623 -U admin -R 1 -N 5 -P *** lan print 1*
Set in Progress : Set Complete
Auth Type Support : NONE MD2 MD5 PASSWORD
Auth Type Enable : Callback : MD2 MD5 PASSWORD
: User : MD2 MD5 PASSWORD
: Operator : MD2 MD5 PASSWORD
: Admin : MD2 MD5 PASSWORD
: OEM : MD2 MD5 PASSWORD
IP Address Source : Static Address
IP Address : 10.0.40.6
Subnet Mask : 255.255.0.0
MAC Address : 0c:c4:7a:3c:c0:b9
SNMP Community String : public
IP Header : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
Default Gateway IP : 0.0.0.0
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 1,2,3,6,7,8,11,12
Cipher Suite Priv Max : XaaaXXaaaXXaaXX
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
Bad Password Threshold : Not Available
*(undercloud) [stack at n001 ~]$ openstack baremetal node show
23a17f0f-d683-4f90-8696-f947485900f9*
/usr/lib/python3.6/site-packages/requests/__init__.py:91:
RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't
match a supported version!
RequestsDependencyWarning)
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value
|
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| allocation_uuid | None
|
| automated_clean | None
|
| bios_interface | no-bios
|
| boot_interface | ipxe
|
| chassis_uuid | None
|
| clean_step | {}
|
| conductor | n001.localdomain
|
| conductor_group |
|
| console_enabled | False
|
| console_interface | ipmitool-socat
|
| created_at | 2021-04-26T08:22:49+00:00
|
| deploy_interface | direct
|
| deploy_step | {}
|
| description | None
|
| driver | ipmi
|
| driver_info | {'deploy_kernel':
'file:///var/lib/ironic/httpboot/agent.kernel', 'rescue_kernel':
'file:///var/lib/ironic/httpboot/agent.kernel', 'deploy_ramdisk':
'file:///var/lib/ironic/httpboot/agent.ramdisk', 'rescue_ramdisk':
'file:///var/lib/ironic/httpboot/agent.ramdisk', 'ipmi_address':
'10.0.1.11', 'ipmi_username': 'admin', 'ipmi_password': '******',
'ipmi_port': 623} |
| driver_internal_info | {}
|
| extra | {}
|
| fault | None
|
| inspect_interface | inspector
|
| inspection_finished_at | None
|
| inspection_started_at | None
|
| instance_info | {}
|
| instance_uuid | None
|
| last_error | None
|
| lessee | None
|
| maintenance | False
|
| maintenance_reason | None
|
| management_interface | ipmitool
|
| name | None
|
| network_data | {}
|
| network_interface | flat
|
| owner | None
|
| power_interface | ipmitool
|
| power_state | power off
|
| properties | {'capabilities': 'boot_option:local', 'vendor':
'supermicro'}
|
| protected | False
|
| protected_reason | None
|
| provision_state | manageable
|
| provision_updated_at | 2021-04-26T08:22:55+00:00
|
| raid_config | {}
|
| raid_interface | no-raid
|
| rescue_interface | agent
|
| reservation | None
|
| resource_class | baremetal
|
| retired | False
|
| retired_reason | None
|
| storage_interface | noop
|
| target_power_state | None
|
| target_provision_state | None
|
| target_raid_config | {}
|
| traits | []
|
| updated_at | 2021-04-26T08:35:27+00:00
|
| uuid | 23a17f0f-d683-4f90-8696-f947485900f9
|
| vendor_interface | ipmitool
|
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
and here is the validate output, i could see the pxe assign ipaddress to
node and loaded the operating system.
*(undercloud) [stack at n001 ~]$ openstack baremetal node validate
23a17f0f-d683-4f90-8696-f947485900f9*
/usr/lib/python3.6/site-packages/requests/__init__.py:91:
RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't
match a supported version!
RequestsDependencyWarning)
+------------+--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Interface | Result | Reason
|
+------------+--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| bios | False | Driver ipmi does not support bios (disabled or not
implemented).
|
| boot | False | Cannot validate image information for node
23a17f0f-d683-4f90-8696-f947485900f9 because one or more parameters are
missing from its instance_info and insufficent information is present to
boot from a remote volume. Missing are: ['image_source', 'kernel',
'ramdisk'] |
| console | False | Either missing 'ipmi_terminal_port' parameter in
node's driver_info or [console]port_range is not configured
|
| deploy | False | Cannot validate image information for node
23a17f0f-d683-4f90-8696-f947485900f9 because one or more parameters are
missing from its instance_info and insufficent information is present to
boot from a remote volume. Missing are: ['image_source', 'kernel',
'ramdisk'] |
| inspect | True |
|
| management | True |
|
| network | True |
|
| power | True |
|
| raid | False | Driver ipmi does not support raid (disabled or not
implemented).
|
| rescue | False | Cannot validate image information for node
23a17f0f-d683-4f90-8696-f947485900f9 because one or more parameters are
missing from its instance_info and insufficent information is present to
boot from a remote volume. Missing are: ['image_source', 'kernel',
'ramdisk'] |
| storage | True |
|
+------------+--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
*(undercloud) [stack at n001 ~]$ openstack baremetal driver list*
/usr/lib/python3.6/site-packages/requests/__init__.py:91:
RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't
match a supported version!
RequestsDependencyWarning)
+---------------------+------------------+
| Supported driver(s) | Active host(s) |
+---------------------+------------------+
| idrac | n001.localdomain |
| ilo | n001.localdomain |
| ipmi | n001.localdomain |
| redfish | n001.localdomain |
+---------------------+------------------+
-Vinesh
On Mon, Apr 26, 2021 at 6:59 PM Julia Kreger <juliaashleykreger at gmail.com>
wrote:
> Greetings,
>
> In all likelihood, the credentials are wrong for the baremetal node
> and the lock is being held by the conductor who is still trying to
> record the power state. The lock is an intentional behavior clients
> should retry if they encounter the lock. This is because BMC's often
> cannot handle concurrent requests.
>
> I would first manually verify:
>
> * That the nodes are not in maintenance state (openstack baremetal
> node show). The node last_error field may have a hint or indication to
> the actual error, but visit the next two bullet points.
> * That a power state of on or off has been recorded. If it has not
> been recorded, the supplied credentials or or access is correct.
> * If you're sure about the credentials, verify basic connectivity to
> the BMC address. Some BMCs are very particular about *how* the
> networking is configured, specifically to help limit attacks from the
> network itself.
>
> -Julia
>
>
> On Wed, Apr 21, 2021 at 7:25 PM Vinesh N <vineshnellaiappan at gmail.com>
> wrote:
> >
> > hi,
> > i am facing an issue while introspect the bare metal nodes,
> >
> > error message
> > "4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf | 2021-04-22T01:41:32 |
> 2021-04-22T01:41:35 | Failed to set boot device to PXE: Failed to set boot
> device for node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf: Client Error for url:
> http://10.0.1.202:6385/v1/nodes/4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf/management/boot_device,
> Node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf is locked by host
> undercloud.localdomain, please retry after the current operation is
> completed"
> >
> >
> > (undercloud) [stack at undercloud ~]$ cat /etc/*release
> > CentOS Linux release 8.3.2011
> >
> > ussuri version
> >
> > (undercloud) [stack at undercloud ~]$ openstack image list
> > /usr/lib/python3.6/site-packages/requests/__init__.py:91:
> RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't
> match a supported version!
> > RequestsDependencyWarning)
> >
> +--------------------------------------+------------------------+--------+
> > | ID | Name | Status
> |
> >
> +--------------------------------------+------------------------+--------+
> > | 8ddcd168-cc18-4ce2-97c5-c3502ac471a4 | overcloud-full | active
> |
> > | 8d9cfac9-400b-4570-b0b1-baeb175b16c4 | overcloud-full-initrd | active
> |
> > | c561f1d5-41ae-4599-81ea-de2c1e74eae7 | overcloud-full-vmlinuz | active
> |
> >
> +--------------------------------------+------------------------+--------+
> >
> > Using the command to introspect the node, it was able to discover the
> node and I could provision the node boot via pxe, and load the image on the
> node. I could see the login prompt on the server, after some time of
> provision shut the node down.
> >
> > openstack overcloud node discover --range 10.0.40.5 --credentials
> admin:XXXX --introspect --provide
> >
> > /usr/lib/python3.6/site-packages/requests/__init__.py:91:
> RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't
> match a supported version!
> > RequestsDependencyWarning)
> > Successfully probed node IP 10.0.40.5
> > Successfully registered node UUID 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf
> > /usr/lib/python3.6/site-packages/requests/__init__.py:91:
> RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't
> match a supported version!
> > RequestsDependencyWarning)
> >
> > PLAY [Baremetal Introspection for multiple Ironic Nodes]
> ***********************
> > 2021-04-22 07:04:28.978299 | 002590fe-0d22-76eb-1a70-000000000008 |
> TASK | Check for required inputs
> > 2021-04-22 07:04:29.002729 | 002590fe-0d22-76eb-1a70-000000000008 |
> SKIPPED | Check for required inputs | localhost | item=node_uuids
> > 2021-04-22 07:04:29.004468 | 002590fe-0d22-76eb-1a70-000000000008 |
> TIMING | Check for required inputs | localhost | 0:00:00.069134 | 0.0
> > ....
> > ....
> > ....
> >
> > 2021-04-22 07:11:43.261714 | 002590fe-0d22-76eb-1a70-000000000016 |
> TASK | Nodes that failed introspection
> > 2021-04-22 07:11:43.296417 | 002590fe-0d22-76eb-1a70-000000000016 |
> FATAL | Nodes that failed introspection | localhost | error={
> > "msg": " 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf"
> > }
> > 2021-04-22 07:11:43.297359 | 002590fe-0d22-76eb-1a70-000000000016 |
> TIMING | Nodes that failed introspection | localhost | 0:07:14.362025 |
> 0.03s
> >
> > NO MORE HOSTS LEFT
> *************************************************************
> >
> > PLAY RECAP
> *********************************************************************
> > localhost : ok=4 changed=1 unreachable=0
> failed=1 skipped=5 rescued=0 ignored=0
> > 2021-04-22 07:11:43.301553 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary
> Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 2021-04-22 07:11:43.302101 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total
> Tasks: 10 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 2021-04-22 07:11:43.302609 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time:
> 0:07:14.367265 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 2021-04-22 07:11:43.303162 | UUID |
> Info | Host | Task Name | Run Time
> > 2021-04-22 07:11:43.303740 | 002590fe-0d22-76eb-1a70-000000000014 |
> SUMMARY | localhost | Start baremetal introspection | 434.03s
> > 2021-04-22 07:11:43.304248 | 002590fe-0d22-76eb-1a70-000000000015 |
> SUMMARY | localhost | Nodes that passed introspection | 0.04s
> > 2021-04-22 07:11:43.304814 | 002590fe-0d22-76eb-1a70-000000000016 |
> SUMMARY | localhost | Nodes that failed introspection | 0.03s
> > 2021-04-22 07:11:43.305341 | 002590fe-0d22-76eb-1a70-000000000008 |
> SUMMARY | localhost | Check for required inputs | 0.03s
> > 2021-04-22 07:11:43.305854 | 002590fe-0d22-76eb-1a70-00000000000a |
> SUMMARY | localhost | Set node_uuids_intro fact | 0.02s
> > 2021-04-22 07:11:43.306397 | 002590fe-0d22-76eb-1a70-000000000010 |
> SUMMARY | localhost | Check if validation enabled | 0.02s
> > 2021-04-22 07:11:43.306904 | 002590fe-0d22-76eb-1a70-000000000012 |
> SUMMARY | localhost | Fail if validations are disabled | 0.02s
> > 2021-04-22 07:11:43.307379 | 002590fe-0d22-76eb-1a70-00000000000e |
> SUMMARY | localhost | Set concurrency fact | 0.02s
> > 2021-04-22 07:11:43.307913 | 002590fe-0d22-76eb-1a70-00000000000c |
> SUMMARY | localhost | Notice | 0.02s
> > 2021-04-22 07:11:43.308417 | 002590fe-0d22-76eb-1a70-000000000011 |
> SUMMARY | localhost | Run Validations | 0.02s
> > 2021-04-22 07:11:43.308926 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End
> Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 2021-04-22 07:11:43.309423 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State
> Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 2021-04-22 07:11:43.310021 | ~~~~~~~~~~~~~~~~~~ Number of nodes which
> did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~
> > 2021-04-22 07:11:43.310545 | The following node(s) had failures:
> localhost
> > 2021-04-22 07:11:43.311080 |
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > Ansible execution failed. playbook:
> /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run
> Status: failed, Return Code: 2
> > Exception occured while running the command
> > Traceback (most recent call last):
> > File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line
> 34, in run
> > super(Command, self).run(parsed_args)
> > File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py",
> line 41, in run
> > return super(Command, self).run(parsed_args)
> > File "/usr/lib/python3.6/site-packages/cliff/command.py", line 187, in
> run
> > return_code = self.take_action(parsed_args) or 0
> > File
> "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_node.py", line
> 462, in take_action
> > retry_timeout=parsed_args.retry_timeout,
> > File
> "/usr/lib/python3.6/site-packages/tripleoclient/workflows/baremetal.py",
> line 193, in introspect
> > "retry_timeout": retry_timeout,
> > File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line
> 728, in run_ansible_playbook
> > raise RuntimeError(err_msg)
> > RuntimeError: Ansible execution failed. playbook:
> /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run
> Status: failed, Return Code: 2
> > Ansible execution failed. playbook:
> /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run
> Status: failed, Return Code: 2
> >
> >
> > (undercloud) [stack at undercloud ~]$ openstack baremetal introspection
> list
> > /usr/lib/python3.6/site-packages/requests/__init__.py:91:
> RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't
> match a supported version!
> > RequestsDependencyWarning)
> >
> +--------------------------------------+---------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> > | UUID | Started at | Finished
> at | Error
>
>
>
> |
> >
> +--------------------------------------+---------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> > | 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf | 2021-04-22T01:41:32 |
> 2021-04-22T01:41:35 | Failed to set boot device to PXE: Failed to set boot
> device for node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf: Client Error for url:
> http://10.0.1.202:6385/v1/nodes/4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf/management/boot_device,
> Node 4bdd55bd-c4d9-4af0-b619-2e0b7b0107cf is locked by host
> undercloud.localdomain, please retry after the current operation is
> completed. |
> > | 3d091348-e9c7-4e99-80e3-df72d332d935 | 2021-04-21T12:36:30 |
> 2021-04-21T12:36:32 | Failed to set boot device to PXE: Failed to set boot
> device for node 3d091348-e9c7-4e99-80e3-df72d332d935: Client Error for url:
> http://10.0.1.202:6385/v1/nodes/3d091348-e9c7-4e99-80e3-df72d332d935/management/boot_device,
> Node 3d091348-e9c7-4e99-80e3-df72d332d935 is locked by host
> undercloud.localdomain, please retry after the current operation is
> completed. |
> >
> +--------------------------------------+---------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210426/d723eb77/attachment-0001.html>
More information about the openstack-discuss
mailing list