[Tripleo] - IPv6 Wallaby Undercloud Installation failure

Lokendra Rathour lokendrarathour at gmail.com
Thu Jun 16 07:05:32 UTC 2022


Hi Shephard,
Thanks, after changing the details as mentioned, Undercloud got installed
successfully.
Now as a part to test the introspection we added a single node and
initiated the introspection on which we are getting errors.
IP as per the inspector range is getting allocated, but soon after the IP
allocation the introspection ILO gives the below error:
[image: image.png]
it says, Downloading NBP file....
PXE-E99: Unexpected network error.

Underlcoud.conf:

undercloud_debug = true
clean_nodes = true
cleanup = false
container_cli = podman
container_healthcheck_disabled = true
container_images_file = /home/stack/containers-prepare-parameter.yaml
deployment_user = stack
enable_heat = true
enable_ironic = true
enable_ironic_inspector = true
enable_neutron = true
generate_service_certificate = false
enable_routed_networks = false
ipv6_address_mode = dhcpv6-stateful
ipxe_enabled = true
ironic_default_network_interface = neutron
ironic_enabled_network_interfaces = neutron,flat
local_interface = enp8s0
local_ip = aaaa:aaaa:aaaa::1/64
subnets = ctlplane-subnet
undercloud_admin_host = aaaa:aaaa:aaaa::1
undercloud_public_host = aaaa:aaaa:aaaa::1
undercloud_hostname = undercloud.com
undercloud_ntp_servers = 30.30.30.3
undercloud_timezone = UTC
[ctlplane-subnet]
cidr = aaaa:aaaa:aaaa::/64
dhcp_end = aaaa:aaaa:aaaa::19
dhcp_start = aaaa:aaaa:aaaa::5
gateway = aaaa:aaaa:aaaa::1
inspection_iprange = aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40


the ironic config in the container:

[root at undercloud /]# vi /etc/ironic-inspector/dnsmasq.conf
port=0
interface=br-ctlplane

dhcp-range=set:ctlplane-subnet,aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40,64,10m
dhcp-option-force=tag:ctlplane-subnet,option:mtu,1500
dhcp-sequential-ip
dhcp-match=ipxe,175
dhcp-match=set:efi,option:client-arch,7
dhcp-match=set:efi,option:client-arch,9
dhcp-match=set:efi,option:client-arch,11
# dhcpv6s for Client System Architecture Type (61)
dhcp-match=set:efi6,option6:61,0007
dhcp-match=set:efi6,option6:61,0009
dhcp-match=set:efi6,option6:61,0011
dhcp-userclass=set:ipxe6,iPXE
# Client is already running iPXE; move to next stage of chainloading
dhcp-boot=tag:ipxe,http://[aaaa:aaaa:aaaa::1]:8088/inspector.ipxe
dhcp-option=tag:ipxe6,option6:bootfile-url,http://
[aaaa:aaaa:aaaa::1]:8088/inspector.ipxe
# Client is PXE booting over EFI without iPXE ROM; send EFI version of iPXE
chainloader
dhcp-boot=tag:efi,tag:!ipxe,snponly.efi
dhcp-option=tag:efi6,tag:!ipxe6,option6:bootfile-url,tftp://[aaaa:aaaa:aaaa::1]/snponly.efi
# Client is running PXE over BIOS; send BIOS version of iPXE chainloader
dhcp-boot=undionly.kpxe,localhost.localdomain,aaaa:aaaa:aaaa::1

dhcp-hostsdir=/var/lib/ironic-inspector/dhcp-hostsdir


Please check and help me with the possible error and resolution.

Best Regards,
Lokendra




On Thu, Jun 16, 2022 at 5:15 AM Brendan Shephard <bshephar at redhat.com>
wrote:

> Hey,
>
> Looks like that is the problem. The [ ] around the IP address are causing
> the issue. If I try to run dnsmasq using exactly the output you get, it
> gives me the same error:
> [root at tripleo-director ~]# /usr/sbin/dnsmasq --keep-in-foreground
> --log-facility=/var/log/ironic/dnsmasq.log --user=root
> --conf-file=/dev/null --listen-address=[aaaa:aaaa:aaaa::1] --port=0
> --enable-tftp --tftp-root=/var/lib/ironic/tftpboot
>
> dnsmasq: bad command line options: try --help
>
> VS without the [ ] I can see it starts up normally.
>
> The settings in your undercloud.conf file look to be correct I believe. So
> I think there might be a bug here. I don't think we should be saving that
> value with the square brackets, or we would need to filter them out when we
> gather the value in that variable.
>
> I raised a bug for it here so that we can dig into this and find what
> needs fixing:
> https://bugs.launchpad.net/tripleo/+bug/1978892
>
> In the meantime, if you edit that hieradata value, are you able to get
> that container started?
>
> Change this:
> [root at tripleo-director ~]# egrep -r 'tftp_bind_host'
> /etc/puppet/hieradata/
> /etc/puppet/hieradata/service_configs.json:
>  "ironic::pxe::tftp_bind_host": "%{lookup('ctlplane_uri')}",
>
> To this:
>  "ironic::pxe::tftp_bind_host": "aaaa:aaaa:aaaa::1"
>
> Then restart the service:
> sudo systemctl restart tripleo_ironic_pxe_http.service
> tripleo_ironic_pxe_tftp.service
>
> Does that get the container running without the error? I did the same in
> my environment and can see that dnsmasq is running properly like that:
> [root at tripleo-director ~]# ps -ef | grep aaaa
> root       71180   52675  0 19:24 pts/4    00:00:00 /usr/sbin/dnsmasq
> --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root
> --conf-file=/dev/null --listen-address=aaaa:aaaa:aaaa::1 --port=0
> --enable-tftp --tftp-root=/var/lib/ironic/tftpboot
>
> Brendan Shephard
>
> Software Engineer
>
> Red Hat APAC <https://www.redhat.com>
>
> 193 N Quay
>
> Brisbane City QLD 4000
> @RedHat <https://twitter.com/redhat>   Red Hat
> <https://www.linkedin.com/company/red-hat>  Red Hat
> <https://www.facebook.com/RedHatInc>
> <https://red.ht/sig>
> <https://redhat.com/summit>
>
>
> On Thu, Jun 16, 2022 at 12:12 AM Lokendra Rathour <
> lokendrarathour at gmail.com> wrote:
>
>> Hi Shephard,
>> I am getting the local_ip (ipv6) of the undercloud :
>>
>> [root at undercloud stack]# sudo hiera ironic::pxe::tftp_bind_host -c
>> /etc/puppet/hiera.yaml
>> [aaaa:aaaa:aaaa::1]
>>
>> is this because of some ipv6 reasons?
>>
>>
>> On Wed, Jun 15, 2022 at 6:08 PM Brendan Shephard <bshephar at redhat.com>
>> wrote:
>>
>>> Hey,
>>>
>>> Ok, that command looks fine. What about that variable there? Do you get
>>> anything back when you run:
>>> sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml
>>>
>>> Mine returns:
>>> sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml
>>> 192.168.24.115
>>>
>>> Brendan Shephard
>>>
>>> Software Engineer
>>>
>>> Red Hat APAC <https://www.redhat.com>
>>>
>>> 193 N Quay
>>>
>>> Brisbane City QLD 4000
>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>> <https://www.facebook.com/RedHatInc>
>>> <https://red.ht/sig>
>>> <https://redhat.com/summit>
>>>
>>>
>>> On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour <
>>> lokendrarathour at gmail.com> wrote:
>>>
>>>> Hi Shephard,
>>>>
>>>> this is the command from my wallaby:
>>>> [root at undercloud ~]# sudo cat
>>>> /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json
>>>> {
>>>>   "cap_add": [
>>>>     "NET_ADMIN",
>>>>     "NET_RAW",
>>>>     "SETUID"
>>>>   ],
>>>>   "command": [
>>>>     "/bin/bash",
>>>>     "-c",
>>>>     "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c
>>>> /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground
>>>> --log-facility=/var/log/ironic/dnsmasq.log --user=root
>>>> --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp
>>>> --tftp-root=/var/lib/ironic/tftpboot"
>>>>   ],
>>>>   "environment": {
>>>>     "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS",
>>>>     "TRIPLEO_CONFIG_HASH": "9fb3e4e0e35ee35fdf74cfccb16a7543"
>>>>   },
>>>>   "healthcheck": {
>>>>     "test": "/openstack/healthcheck"
>>>>   },
>>>>   "image":
>>>> "undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo",
>>>>   "net": "host",
>>>>   "privileged": false,
>>>>   "restart": "always",
>>>>   "start_order": 90,
>>>>   "volumes": [
>>>>     "/etc/hosts:/etc/hosts:ro",
>>>>     "/etc/localtime:/etc/localtime:ro",
>>>>     "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
>>>>
>>>> "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",
>>>>
>>>> "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
>>>>
>>>> "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro",
>>>>     "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro",
>>>>     "/dev/log:/dev/log",
>>>>     "/etc/puppet:/etc/puppet:ro",
>>>>
>>>> "/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro",
>>>>
>>>> "/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro",
>>>>     "/var/lib/ironic:/var/lib/ironic:shared,z",
>>>>     "/var/log/containers/ironic:/var/log/ironic:z",
>>>>     "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z"
>>>>   ]
>>>> }[root at undercloud ~]#
>>>>
>>>> Comparing both, they look alike.
>>>> please check once.
>>>>
>>>> On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard <bshephar at redhat.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Looks like the command was in a different file in Wallaby, can you
>>>>> check:
>>>>> sudo cat
>>>>> /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json
>>>>>
>>>>> That one should have the dnsmasq command it's trying to run. For
>>>>> example, here it is from my Wallaby environment:
>>>>> [stack at undercloud-0 ~]$ sudo cat
>>>>> /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json
>>>>> | jq .command
>>>>> [
>>>>>   "/bin/bash",
>>>>>   "-c",
>>>>>   "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c
>>>>> /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground
>>>>> --log-facility=/var/log/ironic/dnsmasq.log --user=root
>>>>> --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp
>>>>> --tftp-root=/var/lib/ironic/tftpboot"
>>>>> ]
>>>>>
>>>>>
>>>>>
>>>>> Brendan Shephard
>>>>>
>>>>> Software Engineer
>>>>>
>>>>> Red Hat APAC <https://www.redhat.com>
>>>>>
>>>>> 193 N Quay
>>>>>
>>>>> Brisbane City QLD 4000
>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>> <https://www.facebook.com/RedHatInc>
>>>>> <https://red.ht/sig>
>>>>> <https://redhat.com/summit>
>>>>>
>>>>>
>>>>> On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour <
>>>>> lokendrarathour at gmail.com> wrote:
>>>>>
>>>>>> Hi Shephard,
>>>>>> Here is the o/p of the file:
>>>>>>
>>>>>> [root at undercloud ~]# sudo cat
>>>>>> /var/lib/kolla/config_files/ironic_pxe_tftp.json
>>>>>> {
>>>>>>   "config_files": [
>>>>>>     {
>>>>>>       "dest": "/",
>>>>>>       "merge": true,
>>>>>>       "preserve_properties": true,
>>>>>>       "source": "/var/lib/kolla/config_files/src/*"
>>>>>>     }
>>>>>>   ],
>>>>>>   "permissions": [
>>>>>>     {
>>>>>>       "owner": "ironic:ironic",
>>>>>>       "path": "/var/log/ironic",
>>>>>>       "recurse": true
>>>>>>     },
>>>>>>     {
>>>>>>       "owner": "ironic:ironic",
>>>>>>       "path": "/var/lib/ironic",
>>>>>>       "recurse": true
>>>>>>     }
>>>>>>   ]
>>>>>> }[root at undercloud ~]#
>>>>>>
>>>>>>
>>>>>> Thanks once agan.
>>>>>>
>>>>>> -Lokendra
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard <bshephar at redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Looks like something wrong with the dnsmasq command the container is
>>>>>>> being launched with. What command is it trying to run?
>>>>>>>
>>>>>>> sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json
>>>>>>>
>>>>>>> Brendan Shephard
>>>>>>>
>>>>>>> Software Engineer
>>>>>>>
>>>>>>> Red Hat APAC <https://www.redhat.com>
>>>>>>>
>>>>>>> 193 N Quay
>>>>>>>
>>>>>>> Brisbane City QLD 4000
>>>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>>>> <https://www.facebook.com/RedHatInc>
>>>>>>> <https://red.ht/sig>
>>>>>>> <https://redhat.com/summit>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta <anyrude10 at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Brendan,
>>>>>>>>
>>>>>>>> Thanks for your response.
>>>>>>>>
>>>>>>>> Please find the log below.
>>>>>>>>
>>>>>>>> [stack at undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp
>>>>>>>>
>>>>>>>> dnsmasq: bad command line options: try --help
>>>>>>>> dnsmasq: bad command line options: try --help
>>>>>>>> dnsmasq: bad command line options: try --help
>>>>>>>> dnsmasq: bad command line options: try --help
>>>>>>>> dnsmasq: bad command line options: try --help
>>>>>>>> dnsmasq: bad command line options: try --help
>>>>>>>>
>>>>>>>> [stack at undercloud t2u2v2w]$  sudo podman ps --filter
>>>>>>>> name=ironic_pxe -a
>>>>>>>> CONTAINER ID  IMAGE
>>>>>>>>                                     COMMAND               CREATED
>>>>>>>>  STATUS                              PORTS       NAMES
>>>>>>>> 02dacbc74cec
>>>>>>>>  undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo
>>>>>>>>  /bin/bash -c BIND...  3 hours ago  Exited (1) 3 hours ago (unhealthy)
>>>>>>>>          ironic_pxe_tftp
>>>>>>>> 1f8ca39fba32
>>>>>>>>  undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo
>>>>>>>>  kolla_start           3 hours ago  Up 3 hours ago (healthy)
>>>>>>>>          ironic_pxe_http
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> Anirudh Gupta
>>>>>>>>
>>>>>>>> On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard <
>>>>>>>> bshephar at redhat.com> wrote:
>>>>>>>>
>>>>>>>>> Hey Anirudh,
>>>>>>>>>
>>>>>>>>> You would need to look at the logs for the ironic_pxe_tftp container
>>>>>>>>> to see why it's failing.
>>>>>>>>>
>>>>>>>>> I assume the tftp container is not Up when you run this command?
>>>>>>>>> [stack at tripleo-director overcloud_playbooks]$ sudo podman ps
>>>>>>>>> --filter name=ironic_pxe -a
>>>>>>>>> CONTAINER ID  IMAGE
>>>>>>>>>                                         COMMAND      CREATED      STATUS
>>>>>>>>>                   PORTS       NAMES
>>>>>>>>> 0170be36e291
>>>>>>>>> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo
>>>>>>>>>  kolla_start  12 days ago  Up 30 hours ago (healthy)
>>>>>>>>>  ironic_pxe_tftp
>>>>>>>>> e507f722bdf0
>>>>>>>>> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo
>>>>>>>>>  kolla_start  12 days ago  Up 30 hours ago (healthy)
>>>>>>>>>  ironic_pxe_http
>>>>>>>>>
>>>>>>>>> Then check the logs to see what the error is:
>>>>>>>>> [stack at tripleo-director overcloud_playbooks]$ sudo podman logs
>>>>>>>>> ironic_pxe_tftp
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Brendan Shephard
>>>>>>>>>
>>>>>>>>> Software Engineer
>>>>>>>>>
>>>>>>>>> Red Hat APAC <https://www.redhat.com>
>>>>>>>>>
>>>>>>>>> 193 N Quay
>>>>>>>>>
>>>>>>>>> Brisbane City QLD 4000
>>>>>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>>>>>> <https://www.facebook.com/RedHatInc>
>>>>>>>>> <https://red.ht/sig>
>>>>>>>>> <https://redhat.com/summit>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta <anyrude10 at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Team,
>>>>>>>>>>
>>>>>>>>>> I am trying to deploy Openstack Wallaby Undercloud on IPv6, but
>>>>>>>>>> facing the below error:
>>>>>>>>>>
>>>>>>>>>> 2022-06-14 05:01:23.213708 | 52540083-cfa2-3f20-e9dc-00000000286f
>>>>>>>>>> | TASK | Manage container systemd services and cleanup old systemd
>>>>>>>>>> healthchecks for /var/lib/tripleo-config/container-startup-config/step_4
>>>>>>>>>> 2022-06-14 05:03:22.912816 | 52540083-cfa2-3f20-e9dc-00000000286f
>>>>>>>>>> | FATAL | Manage container systemd services and cleanup old systemd
>>>>>>>>>> healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 |
>>>>>>>>>> undercloud | error={"changed": false, "msg": "Service ironic_pxe_tftp has
>>>>>>>>>> not started yet"}
>>>>>>>>>> 2022-06-14 05:03:22.914400 | 52540083-cfa2-3f20-e9dc-00000000286f
>>>>>>>>>> | TIMING | tripleo_container_manage : Manage container systemd
>>>>>>>>>>
>>>>>>>>>> Sample Undercloud.conf is as follows:
>>>>>>>>>>
>>>>>>>>>> [DEFAULT]
>>>>>>>>>> clean_nodes = true
>>>>>>>>>> cleanup = false
>>>>>>>>>> container_cli = podman
>>>>>>>>>> container_healthcheck_disabled = true
>>>>>>>>>> container_images_file =
>>>>>>>>>> /home/stack/containers-prepare-parameter.yaml
>>>>>>>>>> deployment_user = stack
>>>>>>>>>> enable_ironic = true
>>>>>>>>>> enable_ironic_inspector = true
>>>>>>>>>> enable_neutron = true
>>>>>>>>>> enable_routed_networks = false
>>>>>>>>>> generate_service_certificate = false
>>>>>>>>>> ipv6_address_mode = dhcpv6-stateful
>>>>>>>>>> ipxe_enabled = true
>>>>>>>>>> local_interface = enp8s0
>>>>>>>>>> local_ip = aaaa:aaaa:aaaa::1/64
>>>>>>>>>> subnets = ctlplane-subnet
>>>>>>>>>> undercloud_admin_host = aaaa:aaaa:aaaa::1
>>>>>>>>>> undercloud_hostname = undercloud.com
>>>>>>>>>> undercloud_ntp_servers = 30.30.30.3
>>>>>>>>>> undercloud_public_host = aaaa:aaaa:aaaa::1
>>>>>>>>>> undercloud_timezone = UTC
>>>>>>>>>>
>>>>>>>>>> [ctlplane-subnet]
>>>>>>>>>> cidr = aaaa:aaaa:aaaa::/64
>>>>>>>>>> dhcp_end = aaaa:aaaa:aaaa::f
>>>>>>>>>> dhcp_start = aaaa:aaaa:aaaa::a
>>>>>>>>>> gateway = aaaa:aaaa:aaaa::1
>>>>>>>>>> inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9
>>>>>>>>>>
>>>>>>>>>> Can someone please help in this regard.
>>>>>>>>>>
>>>>>>>>>> Anirudh Gupta
>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> ~ Lokendra
>>>>>> www.inertiaspeaks.com
>>>>>> www.inertiagroups.com
>>>>>> skype: lokendrarathour
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> ~ Lokendra
>>>> www.inertiaspeaks.com
>>>> www.inertiagroups.com
>>>> skype: lokendrarathour
>>>>
>>>>
>>>>
>>
>> --
>> ~ Lokendra
>> www.inertiaspeaks.com
>> www.inertiagroups.com
>> skype: lokendrarathour
>>
>>
>>

-- 
~ Lokendra
www.inertiaspeaks.com
www.inertiagroups.com
skype: lokendrarathour
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220616/912addf5/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 201716 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220616/912addf5/attachment-0001.png>


More information about the openstack-discuss mailing list