TripleO | OS-net config logs | openstack overcloud node provision
Hi Team, Scenario: We were performing overcloud deployment. During this, we were continuously facing the following errors on two of the nodes out of the four nodes: PLAY [Overcloud Node Grow Volumes] ********************************************* 2023-06-02 19:07:33.483504 | 52540049-4965-0177-e681-00000000000a | TASK | Wait for provisioned nodes to boot [WARNING]: Unhandled error in Python interpreter discovery for host overcloud- controller-0: Failed to connect to the host via ssh: ssh: connect to host 30.30.30.219 port 22: No route to host [WARNING]: Unhandled error in Python interpreter discovery for host overcloud- controller-0: Failed to connect to the host via ssh: ssh: connect to host 30.30.30.219 port 22: Connection refused We faced this error when we were performing overcloud node provisioning. To resolve this issue, we had to update the image and create a user to access the nodes. We were able to access the nodes via this user through the BMC console. This implied that OS was getting correctly installed on the node. Once we accessed the logs on the failed node we came across the following logs: 2023-06-13 09:02:16.120 INFO os_net_config.utils._ordered_nics Finding active nics 2023-06-13 09:02:16.120 INFO os_net_config.utils._ordered_nics eno1 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens2f0 is an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics eno4 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics eno2 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens1f0 is not an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics lo is not an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens2f1 is an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics eno3 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens1f1 is an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml) 2023-06-13 09:02:16.122 INFO os_net_config.utils._ordered_nics Active nics are ['eno1', 'eno2', 'eno3', 'eno4', 'ens1f1', 'ens2f0', 'ens2f1'] 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic2 mapped to: eno2 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic6 mapped to: ens2f0 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic5 mapped to: ens1f1 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic3 mapped to: eno3 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic7 mapped to: ens2f1 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic1 mapped to: eno1 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic4 mapped to: eno4 As we can see in the above logs, only 7 nics are active. But as per our bond config file, we required 8 interfaces: a. nic-1 for management IP b. nic-3 and nic-4 for the br-tenant-2 network. c. nic-5 and nic-6 for the br-tenant network. d. nic-7 and nic 8 for the br-ex network. The wire was faulty in this case and hence only 7 interfaces were available Query: The logs which are visible while we run "openstack overcloud node provision" are not very informative. Is there any way where we could get these logs on the undercloud node itself? This could save some time for future references -- ~ Lokendra skype: lokendrarathour
The --verbose flag is passed through to ansible runs in the provision command, so this will provide more verbose logging: openstack -vvv overcloud node provision On 6/07/23 22:13, Lokendra Rathour wrote:
Hi Team,
Scenario: We were performing overcloud deployment. During this, we were continuously facing the following errors on two of the nodes out of the four nodes:
PLAY [Overcloud Node Grow Volumes] ********************************************* 2023-06-02 19:07:33.483504 | 52540049-4965-0177-e681-00000000000a | TASK | Wait for provisioned nodes to boot [WARNING]: Unhandled error in Python interpreter discovery for host overcloud- controller-0: Failed to connect to the host via ssh: ssh: connect to host 30.30.30.219 port 22: No route to host [WARNING]: Unhandled error in Python interpreter discovery for host overcloud- controller-0: Failed to connect to the host via ssh: ssh: connect to host 30.30.30.219 port 22: Connection refused
We faced this error when we were performing overcloud node provisioning.
To resolve this issue, we had to update the image and create a user to access the nodes. We were able to access the nodes via this user through the BMC console.
This implied that OS was getting correctly installed on the node. Once we accessed the logs on the failed node we came across the following logs:
2023-06-13 09:02:16.120 INFO os_net_config.utils._ordered_nics Finding active nics 2023-06-13 09:02:16.120 INFO os_net_config.utils._ordered_nics eno1 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens2f0 is an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics eno4 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics eno2 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens1f0 is not an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics lo is not an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens2f1 is an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics eno3 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens1f1 is an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml) 2023-06-13 09:02:16.122 INFO os_net_config.utils._ordered_nics Active nics are ['eno1', 'eno2', 'eno3', 'eno4', 'ens1f1', 'ens2f0', 'ens2f1'] 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic2 mapped to: eno2 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic6 mapped to: ens2f0 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic5 mapped to: ens1f1 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic3 mapped to: eno3 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic7 mapped to: ens2f1 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic1 mapped to: eno1 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic4 mapped to: eno4
As we can see in the above logs, only 7 nics are active. But as per our bond config file, we required 8 interfaces: a. nic-1 for management IP b. nic-3 and nic-4 for the br-tenant-2 network. c. nic-5 and nic-6 for the br-tenant network. d. nic-7 and nic 8 for the br-ex network.
The wire was faulty in this case and hence only 7 interfaces were available
Query: The logs which are visible while we run "openstack overcloud node provision" are not very informative. Is there any way where we could get these logs on the undercloud node itself? This could save some time for future references
-- ~ Lokendra skype: lokendrarathour
participants (2)
-
Lokendra Rathour
-
Steve Baker