[Tripleo] - IPv6 Wallaby Undercloud Installation failure
Harald Jensas
hjensas at redhat.com
Fri Jun 17 12:23:09 UTC 2022
Hi,
The error in the screenshot, the booting node is not able to dowload the
network boot program from the TFTP server on the undercloud.
Can you:
* Verify that the TFTP server is running?
* Check the log file - /var/log/containers/ironic/dnsmasq.log
* Try to capture the TFTP traffic on br-ctlplane to a PCAP file to see
if we can use that to dignose the problem.
* download that file manually?
For example: curl -O tftp://<undercloud_ip>/snponly.efi ?
Best Regards
Harald
On 6/16/22 09:05, Lokendra Rathour wrote:
> Hi Shephard,
> Thanks, after changing the details as mentioned, Undercloud got
> installed successfully.
> Now as a part to test the introspection we added a single node and
> initiated the introspection on which we are getting errors.
> IP as per the inspector range is getting allocated, but soon after the
> IP allocation the introspection ILO gives the below error:
> image.png
> it says, Downloading NBP file....
> PXE-E99: Unexpected network error.
>
> Underlcoud.conf:
>
> undercloud_debug = true
> clean_nodes = true
> cleanup = false
> container_cli = podman
> container_healthcheck_disabled = true
> container_images_file = /home/stack/containers-prepare-parameter.yaml
> deployment_user = stack
> enable_heat = true
> enable_ironic = true
> enable_ironic_inspector = true
> enable_neutron = true
> generate_service_certificate = false
> enable_routed_networks = false
> ipv6_address_mode = dhcpv6-stateful
> ipxe_enabled = true
> ironic_default_network_interface = neutron
> ironic_enabled_network_interfaces = neutron,flat
> local_interface = enp8s0
> local_ip = aaaa:aaaa:aaaa::1/64
> subnets = ctlplane-subnet
> undercloud_admin_host = aaaa:aaaa:aaaa::1
> undercloud_public_host = aaaa:aaaa:aaaa::1
> undercloud_hostname = undercloud.com <http://undercloud.com>
> undercloud_ntp_servers = 30.30.30.3
> undercloud_timezone = UTC
> [ctlplane-subnet]
> cidr = aaaa:aaaa:aaaa::/64
> dhcp_end = aaaa:aaaa:aaaa::19
> dhcp_start = aaaa:aaaa:aaaa::5
> gateway = aaaa:aaaa:aaaa::1
> inspection_iprange = aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40
>
>
> the ironic config in the container:
>
> [root at undercloud /]# vi /etc/ironic-inspector/dnsmasq.conf
> port=0
> interface=br-ctlplane
>
> dhcp-range=set:ctlplane-subnet,aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40,64,10m
> dhcp-option-force=tag:ctlplane-subnet,option:mtu,1500
> dhcp-sequential-ip
> dhcp-match=ipxe,175
> dhcp-match=set:efi,option:client-arch,7
> dhcp-match=set:efi,option:client-arch,9
> dhcp-match=set:efi,option:client-arch,11
> # dhcpv6s for Client System Architecture Type (61)
> dhcp-match=set:efi6,option6:61,0007
> dhcp-match=set:efi6,option6:61,0009
> dhcp-match=set:efi6,option6:61,0011
> dhcp-userclass=set:ipxe6,iPXE
> # Client is already running iPXE; move to next stage of chainloading
> dhcp-boot=tag:ipxe,http://[aaaa:aaaa:aaaa::1]:8088/inspector.ipxe
> dhcp-option=tag:ipxe6,option6:bootfile-url,http://[aaaa:aaaa:aaaa::1]:8088/inspector.ipxe
> # Client is PXE booting over EFI without iPXE ROM; send EFI version of
> iPXE chainloader
> dhcp-boot=tag:efi,tag:!ipxe,snponly.efi
> dhcp-option=tag:efi6,tag:!ipxe6,option6:bootfile-url,tftp://[aaaa:aaaa:aaaa::1]/snponly.efi
> # Client is running PXE over BIOS; send BIOS version of iPXE chainloader
> dhcp-boot=undionly.kpxe,localhost.localdomain,aaaa:aaaa:aaaa::1
>
> dhcp-hostsdir=/var/lib/ironic-inspector/dhcp-hostsdir
>
>
> Please check and help me with the possible error and resolution.
>
> Best Regards,
> Lokendra
>
>
>
>
> On Thu, Jun 16, 2022 at 5:15 AM Brendan Shephard <bshephar at redhat.com
> <mailto:bshephar at redhat.com>> wrote:
>
> Hey,
>
> Looks like that is the problem. The [ ] around the IP address are
> causing the issue. If I try to run dnsmasq using exactly the output
> you get, it gives me the same error:
> [root at tripleo-director ~]# /usr/sbin/dnsmasq --keep-in-foreground
> --log-facility=/var/log/ironic/dnsmasq.log --user=root
> --conf-file=/dev/null --listen-address=[aaaa:aaaa:aaaa::1] --port=0
> --enable-tftp --tftp-root=/var/lib/ironic/tftpboot
>
> dnsmasq: bad command line options: try --help
>
> VS without the [ ] I can see it starts up normally.
>
> The settings in your undercloud.conf file look to be correct I
> believe. So I think there might be a bug here. I don't think we
> should be saving that value with the square brackets, or we would
> need to filter them out when we gather the value in that variable.
>
> I raised a bug for it here so that we can dig into this and find
> what needs fixing:
> https://bugs.launchpad.net/tripleo/+bug/1978892
> <https://bugs.launchpad.net/tripleo/+bug/1978892>
>
> In the meantime, if you edit that hieradata value, are you able to
> get that container started?
>
> Change this:
> [root at tripleo-director ~]# egrep -r 'tftp_bind_host'
> /etc/puppet/hieradata/
> /etc/puppet/hieradata/service_configs.json:
> "ironic::pxe::tftp_bind_host": "%{lookup('ctlplane_uri')}",
>
> To this:
> "ironic::pxe::tftp_bind_host": "aaaa:aaaa:aaaa::1"
>
> Then restart the service:
> sudo systemctl restart tripleo_ironic_pxe_http.service
> tripleo_ironic_pxe_tftp.service
>
> Does that get the container running without the error? I did the
> same in my environment and can see that dnsmasq is running properly
> like that:
> [root at tripleo-director ~]# ps -ef | grep aaaa
> root 71180 52675 0 19:24 pts/4 00:00:00
> /usr/sbin/dnsmasq --keep-in-foreground
> --log-facility=/var/log/ironic/dnsmasq.log --user=root
> --conf-file=/dev/null --listen-address=aaaa:aaaa:aaaa::1 --port=0
> --enable-tftp --tftp-root=/var/lib/ironic/tftpboot
>
> Brendan Shephard
>
> Software Engineer
>
> Red Hat APAC <https://www.redhat.com>
>
> 193 N Quay
>
> Brisbane City QLD 4000
>
> @RedHat <https://twitter.com/redhat> Red Hat
> <https://www.linkedin.com/company/red-hat> Red Hat
> <https://www.facebook.com/RedHatInc>
> <https://red.ht/sig>
> <https://redhat.com/summit>
>
>
>
> On Thu, Jun 16, 2022 at 12:12 AM Lokendra Rathour
> <lokendrarathour at gmail.com <mailto:lokendrarathour at gmail.com>> wrote:
>
> Hi Shephard,
> I am getting the local_ip (ipv6) of the undercloud :
>
> [root at undercloud stack]# sudo hiera ironic::pxe::tftp_bind_host
> -c /etc/puppet/hiera.yaml
> [aaaa:aaaa:aaaa::1]
>
> is this because of some ipv6 reasons?
>
>
> On Wed, Jun 15, 2022 at 6:08 PM Brendan Shephard
> <bshephar at redhat.com <mailto:bshephar at redhat.com>> wrote:
>
> Hey,
>
> Ok, that command looks fine. What about that variable there?
> Do you get anything back when you run:
> sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml
>
> Mine returns:
> sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml
> 192.168.24.115
>
> Brendan Shephard
>
> Software Engineer
>
> Red Hat APAC <https://www.redhat.com>
>
> 193 N Quay
>
> Brisbane City QLD 4000
>
> @RedHat <https://twitter.com/redhat> Red Hat
> <https://www.linkedin.com/company/red-hat> Red Hat
> <https://www.facebook.com/RedHatInc>
> <https://red.ht/sig>
> <https://redhat.com/summit>
>
>
>
> On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour
> <lokendrarathour at gmail.com
> <mailto:lokendrarathour at gmail.com>> wrote:
>
> Hi Shephard,
>
> this is the command from my wallaby:
> [root at undercloud ~]# sudo cat
> /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json
> {
> "cap_add": [
> "NET_ADMIN",
> "NET_RAW",
> "SETUID"
> ],
> "command": [
> "/bin/bash",
> "-c",
> "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c
> /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq
> --keep-in-foreground
> --log-facility=/var/log/ironic/dnsmasq.log --user=root
> --conf-file=/dev/null --listen-address=$BIND_HOST
> --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot"
> ],
> "environment": {
> "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS",
> "TRIPLEO_CONFIG_HASH":
> "9fb3e4e0e35ee35fdf74cfccb16a7543"
> },
> "healthcheck": {
> "test": "/openstack/healthcheck"
> },
> "image":
> "undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo",
> "net": "host",
> "privileged": false,
> "restart": "always",
> "start_order": 90,
> "volumes": [
> "/etc/hosts:/etc/hosts:ro",
> "/etc/localtime:/etc/localtime:ro",
>
> "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
>
> "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",
>
> "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
>
> "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro",
> "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro",
> "/dev/log:/dev/log",
> "/etc/puppet:/etc/puppet:ro",
>
> "/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro",
>
> "/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro",
> "/var/lib/ironic:/var/lib/ironic:shared,z",
> "/var/log/containers/ironic:/var/log/ironic:z",
> "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z"
> ]
> }[root at undercloud ~]#
>
> Comparing both, they look alike.
> please check once.
>
> On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard
> <bshephar at redhat.com <mailto:bshephar at redhat.com>> wrote:
>
> Hi,
>
> Looks like the command was in a different file in
> Wallaby, can you check:
> sudo cat
> /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json
>
> That one should have the dnsmasq command it's trying
> to run. For example, here it is from my Wallaby
> environment:
> [stack at undercloud-0 ~]$ sudo cat
> /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json
> | jq .command
> [
> "/bin/bash",
> "-c",
> "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c
> /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq
> --keep-in-foreground
> --log-facility=/var/log/ironic/dnsmasq.log
> --user=root --conf-file=/dev/null
> --listen-address=$BIND_HOST --port=0 --enable-tftp
> --tftp-root=/var/lib/ironic/tftpboot"
> ]
>
>
>
> Brendan Shephard
>
> Software Engineer
>
> Red Hat APAC <https://www.redhat.com>
>
> 193 N Quay
>
> Brisbane City QLD 4000
>
> @RedHat <https://twitter.com/redhat> Red Hat
> <https://www.linkedin.com/company/red-hat> Red Hat
> <https://www.facebook.com/RedHatInc>
> <https://red.ht/sig>
> <https://redhat.com/summit>
>
>
>
> On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour
> <lokendrarathour at gmail.com
> <mailto:lokendrarathour at gmail.com>> wrote:
>
> Hi Shephard,
> Here is the o/p of the file:
>
> [root at undercloud ~]# sudo cat
> /var/lib/kolla/config_files/ironic_pxe_tftp.json
> {
> "config_files": [
> {
> "dest": "/",
> "merge": true,
> "preserve_properties": true,
> "source": "/var/lib/kolla/config_files/src/*"
> }
> ],
> "permissions": [
> {
> "owner": "ironic:ironic",
> "path": "/var/log/ironic",
> "recurse": true
> },
> {
> "owner": "ironic:ironic",
> "path": "/var/lib/ironic",
> "recurse": true
> }
> ]
> }[root at undercloud ~]#
>
>
> Thanks once agan.
>
> -Lokendra
>
>
> On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard
> <bshephar at redhat.com
> <mailto:bshephar at redhat.com>> wrote:
>
> Looks like something wrong with the dnsmasq
> command the container is being launched
> with. What command is it trying to run?
>
> sudo cat
> /var/lib/kolla/config_files/ironic_pxe_tftp.json
>
> Brendan Shephard
>
> Software Engineer
>
> Red Hat APAC <https://www.redhat.com>
>
> 193 N Quay
>
> Brisbane City QLD 4000
>
> @RedHat <https://twitter.com/redhat> Red Hat
> <https://www.linkedin.com/company/red-hat>
> Red Hat <https://www.facebook.com/RedHatInc>
> <https://red.ht/sig>
> <https://redhat.com/summit>
>
>
>
> On Wed, Jun 15, 2022 at 6:22 PM Anirudh
> Gupta <anyrude10 at gmail.com
> <mailto:anyrude10 at gmail.com>> wrote:
>
> Hi Brendan,
>
> Thanks for your response.
>
> Please find the log below.
>
> [stack at undercloud t2u2v2w]$ sudo podman
> logs ironic_pxe_tftp
>
> dnsmasq: bad command line options: try
> --help
> dnsmasq: bad command line options: try
> --help
> dnsmasq: bad command line options: try
> --help
> dnsmasq: bad command line options: try
> --help
> dnsmasq: bad command line options: try
> --help
> dnsmasq: bad command line options: try
> --help
>
> [stack at undercloud t2u2v2w]$ sudo podman
> ps --filter name=ironic_pxe -a
> CONTAINER ID IMAGE
>
> COMMAND
> CREATED STATUS
> PORTS NAMES
> 02dacbc74cec
> undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo /bin/bash -c BIND... 3 hours ago Exited (1) 3 hours ago (unhealthy) ironic_pxe_tftp
> 1f8ca39fba32
> undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo kolla_start 3 hours ago Up 3 hours ago (healthy) ironic_pxe_http
>
>
> Regards
>
> Anirudh Gupta
>
>
> On Wed, Jun 15, 2022 at 11:30 AM Brendan
> Shephard <bshephar at redhat.com
> <mailto:bshephar at redhat.com>> wrote:
>
> Hey Anirudh,
>
> You would need to look at the logs
> for the ironic_pxe_tftp container to
> see why it's failing.
>
> I assume the tftp container is not
> Up when you run this command?
> [stack at tripleo-director
> overcloud_playbooks]$ sudo podman ps
> --filter name=ironic_pxe -a
> CONTAINER ID IMAGE
>
>
> COMMAND CREATED STATUS
> PORTS NAMES
> 0170be36e291
> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo
> <http://registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo>
> kolla_start 12 days ago Up 30
> hours ago (healthy)
> ironic_pxe_tftp
> e507f722bdf0
> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo
> <http://registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo>
> kolla_start 12 days ago Up 30
> hours ago (healthy)
> ironic_pxe_http
>
> Then check the logs to see what the
> error is:
> [stack at tripleo-director
> overcloud_playbooks]$ sudo podman
> logs ironic_pxe_tftp
>
>
>
> Brendan Shephard
>
> Software Engineer
>
> Red Hat APAC <https://www.redhat.com>
>
> 193 N Quay
>
> Brisbane City QLD 4000
>
> @RedHat <https://twitter.com/redhat>
> Red Hat
> <https://www.linkedin.com/company/red-hat>
> Red Hat
> <https://www.facebook.com/RedHatInc>
> <https://red.ht/sig>
> <https://redhat.com/summit>
>
>
>
> On Wed, Jun 15, 2022 at 7:53 AM
> Anirudh Gupta <anyrude10 at gmail.com
> <mailto:anyrude10 at gmail.com>> wrote:
>
> Hi Team,
>
> I am trying to deploy Openstack
> Wallaby Undercloud on IPv6, but
> facing the below error:
>
> 2022-06-14 05:01:23.213708 |
> 52540083-cfa2-3f20-e9dc-00000000286f
> | TASK | Manage container
> systemd services and cleanup old
> systemd healthchecks for
> /var/lib/tripleo-config/container-startup-config/step_4
> 2022-06-14 05:03:22.912816 |
> 52540083-cfa2-3f20-e9dc-00000000286f
> | FATAL | Manage container
> systemd services and cleanup old
> systemd healthchecks for
> /var/lib/tripleo-config/container-startup-config/step_4
> | undercloud | error={"changed":
> false, "msg": "Service
> ironic_pxe_tftp has not started
> yet"}
> 2022-06-14 05:03:22.914400 |
> 52540083-cfa2-3f20-e9dc-00000000286f
> | TIMING |
> tripleo_container_manage :
> Manage container systemd
>
> Sample Undercloud.conf is as
> follows:
>
> [DEFAULT]
> clean_nodes = true
> cleanup = false
> container_cli = podman
> container_healthcheck_disabled =
> true
> container_images_file =
> /home/stack/containers-prepare-parameter.yaml
> deployment_user = stack
> enable_ironic = true
> enable_ironic_inspector = true
> enable_neutron = true
> enable_routed_networks = false
> generate_service_certificate = false
> ipv6_address_mode = dhcpv6-stateful
> ipxe_enabled = true
> local_interface = enp8s0
> local_ip = aaaa:aaaa:aaaa::1/64
> subnets = ctlplane-subnet
> undercloud_admin_host =
> aaaa:aaaa:aaaa::1
> undercloud_hostname =
> undercloud.com
> <http://undercloud.com>
> undercloud_ntp_servers = 30.30.30.3
> undercloud_public_host =
> aaaa:aaaa:aaaa::1
> undercloud_timezone = UTC
>
> [ctlplane-subnet]
> cidr = aaaa:aaaa:aaaa::/64
> dhcp_end = aaaa:aaaa:aaaa::f
> dhcp_start = aaaa:aaaa:aaaa::a
> gateway = aaaa:aaaa:aaaa::1
> inspection_iprange =
> aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9
>
> Can someone please help in this
> regard.
>
> Anirudh Gupta
>
>
>
> --
> ~ Lokendra
> www.inertiaspeaks.com <http://www.inertiaspeaks.com>
> www.inertiagroups.com <http://www.inertiagroups.com>
> skype: lokendrarathour
>
>
>
>
> --
> ~ Lokendra
> www.inertiaspeaks.com <http://www.inertiaspeaks.com>
> www.inertiagroups.com <http://www.inertiagroups.com>
> skype: lokendrarathour
>
>
>
>
> --
> ~ Lokendra
> www.inertiaspeaks.com <http://www.inertiaspeaks.com>
> www.inertiagroups.com <http://www.inertiagroups.com>
> skype: lokendrarathour
>
>
>
>
> --
> ~ Lokendra
> www.inertiaspeaks.com <http://www.inertiaspeaks.com>
> www.inertiagroups.com <http://www.inertiagroups.com>
> skype: lokendrarathour
>
>
More information about the openstack-discuss
mailing list