Hey, Looks like your server is booting in Legacy BIOS mode but the PXE file Ironic is offering is for UEFI. Did you try booting in UEFI mode instead of Legacy BIOS? Brendan Shephard Software Engineer Red Hat APAC <https://www.redhat.com> 193 N Quay Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit> On Thu, Jun 16, 2022 at 5:05 PM Lokendra Rathour <lokendrarathour@gmail.com> wrote:
Hi Shephard, Thanks, after changing the details as mentioned, Undercloud got installed successfully. Now as a part to test the introspection we added a single node and initiated the introspection on which we are getting errors. IP as per the inspector range is getting allocated, but soon after the IP allocation the introspection ILO gives the below error: [image: image.png] it says, Downloading NBP file.... PXE-E99: Unexpected network error.
Underlcoud.conf:
undercloud_debug = true clean_nodes = true cleanup = false container_cli = podman container_healthcheck_disabled = true container_images_file = /home/stack/containers-prepare-parameter.yaml deployment_user = stack enable_heat = true enable_ironic = true enable_ironic_inspector = true enable_neutron = true generate_service_certificate = false enable_routed_networks = false ipv6_address_mode = dhcpv6-stateful ipxe_enabled = true ironic_default_network_interface = neutron ironic_enabled_network_interfaces = neutron,flat local_interface = enp8s0 local_ip = aaaa:aaaa:aaaa::1/64 subnets = ctlplane-subnet undercloud_admin_host = aaaa:aaaa:aaaa::1 undercloud_public_host = aaaa:aaaa:aaaa::1 undercloud_hostname = undercloud.com undercloud_ntp_servers = 30.30.30.3 undercloud_timezone = UTC [ctlplane-subnet] cidr = aaaa:aaaa:aaaa::/64 dhcp_end = aaaa:aaaa:aaaa::19 dhcp_start = aaaa:aaaa:aaaa::5 gateway = aaaa:aaaa:aaaa::1 inspection_iprange = aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40
the ironic config in the container:
[root@undercloud /]# vi /etc/ironic-inspector/dnsmasq.conf port=0 interface=br-ctlplane
dhcp-range=set:ctlplane-subnet,aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40,64,10m dhcp-option-force=tag:ctlplane-subnet,option:mtu,1500 dhcp-sequential-ip dhcp-match=ipxe,175 dhcp-match=set:efi,option:client-arch,7 dhcp-match=set:efi,option:client-arch,9 dhcp-match=set:efi,option:client-arch,11 # dhcpv6s for Client System Architecture Type (61) dhcp-match=set:efi6,option6:61,0007 dhcp-match=set:efi6,option6:61,0009 dhcp-match=set:efi6,option6:61,0011 dhcp-userclass=set:ipxe6,iPXE # Client is already running iPXE; move to next stage of chainloading dhcp-boot=tag:ipxe,http://[aaaa:aaaa:aaaa::1]:8088/inspector.ipxe dhcp-option=tag:ipxe6,option6:bootfile-url,http:// [aaaa:aaaa:aaaa::1]:8088/inspector.ipxe # Client is PXE booting over EFI without iPXE ROM; send EFI version of iPXE chainloader dhcp-boot=tag:efi,tag:!ipxe,snponly.efi
dhcp-option=tag:efi6,tag:!ipxe6,option6:bootfile-url,tftp://[aaaa:aaaa:aaaa::1]/snponly.efi # Client is running PXE over BIOS; send BIOS version of iPXE chainloader dhcp-boot=undionly.kpxe,localhost.localdomain,aaaa:aaaa:aaaa::1
dhcp-hostsdir=/var/lib/ironic-inspector/dhcp-hostsdir
Please check and help me with the possible error and resolution.
Best Regards, Lokendra
On Thu, Jun 16, 2022 at 5:15 AM Brendan Shephard <bshephar@redhat.com> wrote:
Hey,
Looks like that is the problem. The [ ] around the IP address are causing the issue. If I try to run dnsmasq using exactly the output you get, it gives me the same error: [root@tripleo-director ~]# /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=[aaaa:aaaa:aaaa::1] --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot
dnsmasq: bad command line options: try --help
VS without the [ ] I can see it starts up normally.
The settings in your undercloud.conf file look to be correct I believe. So I think there might be a bug here. I don't think we should be saving that value with the square brackets, or we would need to filter them out when we gather the value in that variable.
I raised a bug for it here so that we can dig into this and find what needs fixing: https://bugs.launchpad.net/tripleo/+bug/1978892
In the meantime, if you edit that hieradata value, are you able to get that container started?
Change this: [root@tripleo-director ~]# egrep -r 'tftp_bind_host' /etc/puppet/hieradata/ /etc/puppet/hieradata/service_configs.json: "ironic::pxe::tftp_bind_host": "%{lookup('ctlplane_uri')}",
To this: "ironic::pxe::tftp_bind_host": "aaaa:aaaa:aaaa::1"
Then restart the service: sudo systemctl restart tripleo_ironic_pxe_http.service tripleo_ironic_pxe_tftp.service
Does that get the container running without the error? I did the same in my environment and can see that dnsmasq is running properly like that: [root@tripleo-director ~]# ps -ef | grep aaaa root 71180 52675 0 19:24 pts/4 00:00:00 /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=aaaa:aaaa:aaaa::1 --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot
Brendan Shephard
Software Engineer
Red Hat APAC <https://www.redhat.com>
193 N Quay
Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>
On Thu, Jun 16, 2022 at 12:12 AM Lokendra Rathour < lokendrarathour@gmail.com> wrote:
Hi Shephard, I am getting the local_ip (ipv6) of the undercloud :
[root@undercloud stack]# sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml [aaaa:aaaa:aaaa::1]
is this because of some ipv6 reasons?
On Wed, Jun 15, 2022 at 6:08 PM Brendan Shephard <bshephar@redhat.com> wrote:
Hey,
Ok, that command looks fine. What about that variable there? Do you get anything back when you run: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml
Mine returns: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml 192.168.24.115
Brendan Shephard
Software Engineer
Red Hat APAC <https://www.redhat.com>
193 N Quay
Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>
On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour < lokendrarathour@gmail.com> wrote:
Hi Shephard,
this is the command from my wallaby: [root@undercloud ~]# sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json { "cap_add": [ "NET_ADMIN", "NET_RAW", "SETUID" ], "command": [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ], "environment": { "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "TRIPLEO_CONFIG_HASH": "9fb3e4e0e35ee35fdf74cfccb16a7543" }, "healthcheck": { "test": "/openstack/healthcheck" }, "image": "undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo", "net": "host", "privileged": false, "restart": "always", "start_order": 90, "volumes": [ "/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro", "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",
"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/etc/puppet:/etc/puppet:ro",
"/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro",
"/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro", "/var/lib/ironic:/var/lib/ironic:shared,z", "/var/log/containers/ironic:/var/log/ironic:z", "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z" ] }[root@undercloud ~]#
Comparing both, they look alike. please check once.
On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard <bshephar@redhat.com> wrote:
Hi,
Looks like the command was in a different file in Wallaby, can you check: sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json
That one should have the dnsmasq command it's trying to run. For example, here it is from my Wallaby environment: [stack@undercloud-0 ~]$ sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json | jq .command [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ]
Brendan Shephard
Software Engineer
Red Hat APAC <https://www.redhat.com>
193 N Quay
Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>
On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour < lokendrarathour@gmail.com> wrote:
> Hi Shephard, > Here is the o/p of the file: > > [root@undercloud ~]# sudo cat > /var/lib/kolla/config_files/ironic_pxe_tftp.json > { > "config_files": [ > { > "dest": "/", > "merge": true, > "preserve_properties": true, > "source": "/var/lib/kolla/config_files/src/*" > } > ], > "permissions": [ > { > "owner": "ironic:ironic", > "path": "/var/log/ironic", > "recurse": true > }, > { > "owner": "ironic:ironic", > "path": "/var/lib/ironic", > "recurse": true > } > ] > }[root@undercloud ~]# > > > Thanks once agan. > > -Lokendra > > > On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard < > bshephar@redhat.com> wrote: > >> Looks like something wrong with the dnsmasq command the container >> is being launched with. What command is it trying to run? >> >> sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json >> >> Brendan Shephard >> >> Software Engineer >> >> Red Hat APAC <https://www.redhat.com> >> >> 193 N Quay >> >> Brisbane City QLD 4000 >> @RedHat <https://twitter.com/redhat> Red Hat >> <https://www.linkedin.com/company/red-hat> Red Hat >> <https://www.facebook.com/RedHatInc> >> <https://red.ht/sig> >> <https://redhat.com/summit> >> >> >> On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta <anyrude10@gmail.com> >> wrote: >> >>> Hi Brendan, >>> >>> Thanks for your response. >>> >>> Please find the log below. >>> >>> [stack@undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp >>> >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> >>> [stack@undercloud t2u2v2w]$ sudo podman ps --filter >>> name=ironic_pxe -a >>> CONTAINER ID IMAGE >>> COMMAND CREATED >>> STATUS PORTS NAMES >>> 02dacbc74cec >>> undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo >>> /bin/bash -c BIND... 3 hours ago Exited (1) 3 hours ago (unhealthy) >>> ironic_pxe_tftp >>> 1f8ca39fba32 >>> undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo >>> kolla_start 3 hours ago Up 3 hours ago (healthy) >>> ironic_pxe_http >>> >>> >>> Regards >>> >>> Anirudh Gupta >>> >>> On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard < >>> bshephar@redhat.com> wrote: >>> >>>> Hey Anirudh, >>>> >>>> You would need to look at the logs for the ironic_pxe_tftp container >>>> to see why it's failing. >>>> >>>> I assume the tftp container is not Up when you run this command? >>>> [stack@tripleo-director overcloud_playbooks]$ sudo podman ps >>>> --filter name=ironic_pxe -a >>>> CONTAINER ID IMAGE >>>> COMMAND CREATED STATUS >>>> PORTS NAMES >>>> 0170be36e291 >>>> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo >>>> kolla_start 12 days ago Up 30 hours ago (healthy) >>>> ironic_pxe_tftp >>>> e507f722bdf0 >>>> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo >>>> kolla_start 12 days ago Up 30 hours ago (healthy) >>>> ironic_pxe_http >>>> >>>> Then check the logs to see what the error is: >>>> [stack@tripleo-director overcloud_playbooks]$ sudo podman logs >>>> ironic_pxe_tftp >>>> >>>> >>>> >>>> Brendan Shephard >>>> >>>> Software Engineer >>>> >>>> Red Hat APAC <https://www.redhat.com> >>>> >>>> 193 N Quay >>>> >>>> Brisbane City QLD 4000 >>>> @RedHat <https://twitter.com/redhat> Red Hat >>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>> <https://www.facebook.com/RedHatInc> >>>> <https://red.ht/sig> >>>> <https://redhat.com/summit> >>>> >>>> >>>> On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta < >>>> anyrude10@gmail.com> wrote: >>>> >>>>> Hi Team, >>>>> >>>>> I am trying to deploy Openstack Wallaby Undercloud on IPv6, but >>>>> facing the below error: >>>>> >>>>> 2022-06-14 05:01:23.213708 | >>>>> 52540083-cfa2-3f20-e9dc-00000000286f | TASK | Manage container systemd >>>>> services and cleanup old systemd healthchecks for >>>>> /var/lib/tripleo-config/container-startup-config/step_4 >>>>> 2022-06-14 05:03:22.912816 | >>>>> 52540083-cfa2-3f20-e9dc-00000000286f | FATAL | Manage container systemd >>>>> services and cleanup old systemd healthchecks for >>>>> /var/lib/tripleo-config/container-startup-config/step_4 | undercloud | >>>>> error={"changed": false, "msg": "Service ironic_pxe_tftp has not started >>>>> yet"} >>>>> 2022-06-14 05:03:22.914400 | >>>>> 52540083-cfa2-3f20-e9dc-00000000286f | TIMING | tripleo_container_manage : >>>>> Manage container systemd >>>>> >>>>> Sample Undercloud.conf is as follows: >>>>> >>>>> [DEFAULT] >>>>> clean_nodes = true >>>>> cleanup = false >>>>> container_cli = podman >>>>> container_healthcheck_disabled = true >>>>> container_images_file = >>>>> /home/stack/containers-prepare-parameter.yaml >>>>> deployment_user = stack >>>>> enable_ironic = true >>>>> enable_ironic_inspector = true >>>>> enable_neutron = true >>>>> enable_routed_networks = false >>>>> generate_service_certificate = false >>>>> ipv6_address_mode = dhcpv6-stateful >>>>> ipxe_enabled = true >>>>> local_interface = enp8s0 >>>>> local_ip = aaaa:aaaa:aaaa::1/64 >>>>> subnets = ctlplane-subnet >>>>> undercloud_admin_host = aaaa:aaaa:aaaa::1 >>>>> undercloud_hostname = undercloud.com >>>>> undercloud_ntp_servers = 30.30.30.3 >>>>> undercloud_public_host = aaaa:aaaa:aaaa::1 >>>>> undercloud_timezone = UTC >>>>> >>>>> [ctlplane-subnet] >>>>> cidr = aaaa:aaaa:aaaa::/64 >>>>> dhcp_end = aaaa:aaaa:aaaa::f >>>>> dhcp_start = aaaa:aaaa:aaaa::a >>>>> gateway = aaaa:aaaa:aaaa::1 >>>>> inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9 >>>>> >>>>> Can someone please help in this regard. >>>>> >>>>> Anirudh Gupta >>>>> >>>>> > > -- > ~ Lokendra > www.inertiaspeaks.com > www.inertiagroups.com > skype: lokendrarathour > > >
-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour
-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour
-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour
