Hi, The error in the screenshot, the booting node is not able to dowload the network boot program from the TFTP server on the undercloud. Can you: * Verify that the TFTP server is running? * Check the log file - /var/log/containers/ironic/dnsmasq.log * Try to capture the TFTP traffic on br-ctlplane to a PCAP file to see if we can use that to dignose the problem. * download that file manually? For example: curl -O tftp://<undercloud_ip>/snponly.efi ? Best Regards Harald On 6/16/22 09:05, Lokendra Rathour wrote:
Hi Shephard, Thanks, after changing the details as mentioned, Undercloud got installed successfully. Now as a part to test the introspection we added a single node and initiated the introspection on which we are getting errors. IP as per the inspector range is getting allocated, but soon after the IP allocation the introspection ILO gives the below error: image.png it says, Downloading NBP file.... PXE-E99: Unexpected network error.
Underlcoud.conf:
undercloud_debug = true clean_nodes = true cleanup = false container_cli = podman container_healthcheck_disabled = true container_images_file = /home/stack/containers-prepare-parameter.yaml deployment_user = stack enable_heat = true enable_ironic = true enable_ironic_inspector = true enable_neutron = true generate_service_certificate = false enable_routed_networks = false ipv6_address_mode = dhcpv6-stateful ipxe_enabled = true ironic_default_network_interface = neutron ironic_enabled_network_interfaces = neutron,flat local_interface = enp8s0 local_ip = aaaa:aaaa:aaaa::1/64 subnets = ctlplane-subnet undercloud_admin_host = aaaa:aaaa:aaaa::1 undercloud_public_host = aaaa:aaaa:aaaa::1 undercloud_hostname = undercloud.com <http://undercloud.com> undercloud_ntp_servers = 30.30.30.3 undercloud_timezone = UTC [ctlplane-subnet] cidr = aaaa:aaaa:aaaa::/64 dhcp_end = aaaa:aaaa:aaaa::19 dhcp_start = aaaa:aaaa:aaaa::5 gateway = aaaa:aaaa:aaaa::1 inspection_iprange = aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40
the ironic config in the container:
[root@undercloud /]# vi /etc/ironic-inspector/dnsmasq.conf port=0 interface=br-ctlplane
dhcp-range=set:ctlplane-subnet,aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40,64,10m dhcp-option-force=tag:ctlplane-subnet,option:mtu,1500 dhcp-sequential-ip dhcp-match=ipxe,175 dhcp-match=set:efi,option:client-arch,7 dhcp-match=set:efi,option:client-arch,9 dhcp-match=set:efi,option:client-arch,11 # dhcpv6s for Client System Architecture Type (61) dhcp-match=set:efi6,option6:61,0007 dhcp-match=set:efi6,option6:61,0009 dhcp-match=set:efi6,option6:61,0011 dhcp-userclass=set:ipxe6,iPXE # Client is already running iPXE; move to next stage of chainloading dhcp-boot=tag:ipxe,http://[aaaa:aaaa:aaaa::1]:8088/inspector.ipxe dhcp-option=tag:ipxe6,option6:bootfile-url,http://[aaaa:aaaa:aaaa::1]:8088/inspector.ipxe # Client is PXE booting over EFI without iPXE ROM; send EFI version of iPXE chainloader dhcp-boot=tag:efi,tag:!ipxe,snponly.efi dhcp-option=tag:efi6,tag:!ipxe6,option6:bootfile-url,tftp://[aaaa:aaaa:aaaa::1]/snponly.efi # Client is running PXE over BIOS; send BIOS version of iPXE chainloader dhcp-boot=undionly.kpxe,localhost.localdomain,aaaa:aaaa:aaaa::1
dhcp-hostsdir=/var/lib/ironic-inspector/dhcp-hostsdir
Please check and help me with the possible error and resolution.
Best Regards, Lokendra
On Thu, Jun 16, 2022 at 5:15 AM Brendan Shephard <bshephar@redhat.com <mailto:bshephar@redhat.com>> wrote:
Hey,
Looks like that is the problem. The [ ] around the IP address are causing the issue. If I try to run dnsmasq using exactly the output you get, it gives me the same error: [root@tripleo-director ~]# /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=[aaaa:aaaa:aaaa::1] --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot
dnsmasq: bad command line options: try --help
VS without the [ ] I can see it starts up normally.
The settings in your undercloud.conf file look to be correct I believe. So I think there might be a bug here. I don't think we should be saving that value with the square brackets, or we would need to filter them out when we gather the value in that variable.
I raised a bug for it here so that we can dig into this and find what needs fixing: https://bugs.launchpad.net/tripleo/+bug/1978892 <https://bugs.launchpad.net/tripleo/+bug/1978892>
In the meantime, if you edit that hieradata value, are you able to get that container started?
Change this: [root@tripleo-director ~]# egrep -r 'tftp_bind_host' /etc/puppet/hieradata/ /etc/puppet/hieradata/service_configs.json: "ironic::pxe::tftp_bind_host": "%{lookup('ctlplane_uri')}",
To this: "ironic::pxe::tftp_bind_host": "aaaa:aaaa:aaaa::1"
Then restart the service: sudo systemctl restart tripleo_ironic_pxe_http.service tripleo_ironic_pxe_tftp.service
Does that get the container running without the error? I did the same in my environment and can see that dnsmasq is running properly like that: [root@tripleo-director ~]# ps -ef | grep aaaa root 71180 52675 0 19:24 pts/4 00:00:00 /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=aaaa:aaaa:aaaa::1 --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot
Brendan Shephard
Software Engineer
Red Hat APAC <https://www.redhat.com>
193 N Quay
Brisbane City QLD 4000
@RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>
On Thu, Jun 16, 2022 at 12:12 AM Lokendra Rathour <lokendrarathour@gmail.com <mailto:lokendrarathour@gmail.com>> wrote:
Hi Shephard, I am getting the local_ip (ipv6) of the undercloud :
[root@undercloud stack]# sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml [aaaa:aaaa:aaaa::1]
is this because of some ipv6 reasons?
On Wed, Jun 15, 2022 at 6:08 PM Brendan Shephard <bshephar@redhat.com <mailto:bshephar@redhat.com>> wrote:
Hey,
Ok, that command looks fine. What about that variable there? Do you get anything back when you run: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml
Mine returns: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml 192.168.24.115
Brendan Shephard
Software Engineer
Red Hat APAC <https://www.redhat.com>
193 N Quay
Brisbane City QLD 4000
@RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>
On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour <lokendrarathour@gmail.com <mailto:lokendrarathour@gmail.com>> wrote:
Hi Shephard,
this is the command from my wallaby: [root@undercloud ~]# sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json { "cap_add": [ "NET_ADMIN", "NET_RAW", "SETUID" ], "command": [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ], "environment": { "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "TRIPLEO_CONFIG_HASH": "9fb3e4e0e35ee35fdf74cfccb16a7543" }, "healthcheck": { "test": "/openstack/healthcheck" }, "image": "undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo", "net": "host", "privileged": false, "restart": "always", "start_order": 90, "volumes": [ "/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro",
"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",
"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/etc/puppet:/etc/puppet:ro",
"/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro",
"/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro", "/var/lib/ironic:/var/lib/ironic:shared,z", "/var/log/containers/ironic:/var/log/ironic:z", "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z" ] }[root@undercloud ~]#
Comparing both, they look alike. please check once.
On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard <bshephar@redhat.com <mailto:bshephar@redhat.com>> wrote:
Hi,
Looks like the command was in a different file in Wallaby, can you check: sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json
That one should have the dnsmasq command it's trying to run. For example, here it is from my Wallaby environment: [stack@undercloud-0 ~]$ sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json | jq .command [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ]
Brendan Shephard
Software Engineer
Red Hat APAC <https://www.redhat.com>
193 N Quay
Brisbane City QLD 4000
@RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>
On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour <lokendrarathour@gmail.com <mailto:lokendrarathour@gmail.com>> wrote:
Hi Shephard, Here is the o/p of the file:
[root@undercloud ~]# sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json { "config_files": [ { "dest": "/", "merge": true, "preserve_properties": true, "source": "/var/lib/kolla/config_files/src/*" } ], "permissions": [ { "owner": "ironic:ironic", "path": "/var/log/ironic", "recurse": true }, { "owner": "ironic:ironic", "path": "/var/lib/ironic", "recurse": true } ] }[root@undercloud ~]#
Thanks once agan.
-Lokendra
On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard <bshephar@redhat.com <mailto:bshephar@redhat.com>> wrote:
Looks like something wrong with the dnsmasq command the container is being launched with. What command is it trying to run?
sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json
Brendan Shephard
Software Engineer
Red Hat APAC <https://www.redhat.com>
193 N Quay
Brisbane City QLD 4000
@RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>
On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta <anyrude10@gmail.com <mailto:anyrude10@gmail.com>> wrote:
Hi Brendan,
Thanks for your response.
Please find the log below.
[stack@undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp
dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help
[stack@undercloud t2u2v2w]$ sudo podman ps --filter name=ironic_pxe -a CONTAINER ID IMAGE
COMMAND CREATED STATUS PORTS NAMES 02dacbc74cec undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo /bin/bash -c BIND... 3 hours ago Exited (1) 3 hours ago (unhealthy) ironic_pxe_tftp 1f8ca39fba32 undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo kolla_start 3 hours ago Up 3 hours ago (healthy) ironic_pxe_http
Regards
Anirudh Gupta
On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard <bshephar@redhat.com <mailto:bshephar@redhat.com>> wrote:
Hey Anirudh,
You would need to look at the logs for the ironic_pxe_tftp container to see why it's failing.
I assume the tftp container is not Up when you run this command? [stack@tripleo-director overcloud_playbooks]$ sudo podman ps --filter name=ironic_pxe -a CONTAINER ID IMAGE
COMMAND CREATED STATUS PORTS NAMES 0170be36e291 registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo <http://registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo> kolla_start 12 days ago Up 30 hours ago (healthy) ironic_pxe_tftp e507f722bdf0 registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo <http://registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo> kolla_start 12 days ago Up 30 hours ago (healthy) ironic_pxe_http
Then check the logs to see what the error is: [stack@tripleo-director overcloud_playbooks]$ sudo podman logs ironic_pxe_tftp
Brendan Shephard
Software Engineer
Red Hat APAC <https://www.redhat.com>
193 N Quay
Brisbane City QLD 4000
@RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>
On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta <anyrude10@gmail.com <mailto:anyrude10@gmail.com>> wrote:
Hi Team,
I am trying to deploy Openstack Wallaby Undercloud on IPv6, but facing the below error:
2022-06-14 05:01:23.213708 | 52540083-cfa2-3f20-e9dc-00000000286f | TASK | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 2022-06-14 05:03:22.912816 | 52540083-cfa2-3f20-e9dc-00000000286f | FATAL | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 | undercloud | error={"changed": false, "msg": "Service ironic_pxe_tftp has not started yet"} 2022-06-14 05:03:22.914400 | 52540083-cfa2-3f20-e9dc-00000000286f | TIMING | tripleo_container_manage : Manage container systemd
Sample Undercloud.conf is as follows:
[DEFAULT] clean_nodes = true cleanup = false container_cli = podman container_healthcheck_disabled = true container_images_file = /home/stack/containers-prepare-parameter.yaml deployment_user = stack enable_ironic = true enable_ironic_inspector = true enable_neutron = true enable_routed_networks = false generate_service_certificate = false ipv6_address_mode = dhcpv6-stateful ipxe_enabled = true local_interface = enp8s0 local_ip = aaaa:aaaa:aaaa::1/64 subnets = ctlplane-subnet undercloud_admin_host = aaaa:aaaa:aaaa::1 undercloud_hostname = undercloud.com <http://undercloud.com> undercloud_ntp_servers = 30.30.30.3 undercloud_public_host = aaaa:aaaa:aaaa::1 undercloud_timezone = UTC
[ctlplane-subnet] cidr = aaaa:aaaa:aaaa::/64 dhcp_end = aaaa:aaaa:aaaa::f dhcp_start = aaaa:aaaa:aaaa::a gateway = aaaa:aaaa:aaaa::1 inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9
Can someone please help in this regard.
Anirudh Gupta
-- ~ Lokendra www.inertiaspeaks.com <http://www.inertiaspeaks.com> www.inertiagroups.com <http://www.inertiagroups.com> skype: lokendrarathour
-- ~ Lokendra www.inertiaspeaks.com <http://www.inertiaspeaks.com> www.inertiagroups.com <http://www.inertiagroups.com> skype: lokendrarathour
-- ~ Lokendra www.inertiaspeaks.com <http://www.inertiaspeaks.com> www.inertiagroups.com <http://www.inertiagroups.com> skype: lokendrarathour
-- ~ Lokendra www.inertiaspeaks.com <http://www.inertiaspeaks.com> www.inertiagroups.com <http://www.inertiagroups.com> skype: lokendrarathour