[Tripleo] - IPv6 Wallaby Undercloud Installation failure
Harald Jensas
hjensas at redhat.com
Tue Jun 21 12:01:42 UTC 2022
On 6/21/22 09:19, Lokendra Rathour wrote:
> Hi Team,
> So, I am still not able to provision overcloud baremetal nodes using
> IPv6. Based on the previous discussion i tried changing some settings in
> terms of IPV6 Notation, but no luck.
>
> Has anyone of us tried ipv6 based overcloud deployment on
> Wallaby? Please suggest some way forward.
>
We run regressions downstream on Wallaby with IPv6 provisioning.
Looking at the PCAP file, the TFTP requests arrives and the server is
responding.
Not sure why dnsmasq (the tftp-server) is not able to complete the
transfer. It is possible adding 'log-debug' to the TFTP server
configuration file[1] and restart the service
'tripleo_ironic_pxe_tftp.service' will surface more details?
Is selinux enabled? Try disabling it with 'setenforce permissive'?
Check the /var/log/audit/audit.log for denials?
[1]
/var/lib/config-data/puppet-generated/ironic/etc/ironic/dnsmasq-tftp-server.conf
> Thanks once again for your support.
>
> -Lokendra
>
> On Sat, Jun 18, 2022 at 10:54 PM Dan Sneddon <dsneddon at redhat.com
> <mailto:dsneddon at redhat.com>> wrote:
>
> It is arguable that dnsmasq should accept the IP address in
> brackets, since that is a standard way of representing IPv6
> addresses. It gets confusing, see these ServerFault answers for more
> detail:
>
> https://serverfault.com/questions/444554/what-does-mean-as-an-ip-address-bracket-colon-colon-bracket
> <https://serverfault.com/questions/444554/what-does-mean-as-an-ip-address-bracket-colon-colon-bracket>
>
> http://serverfault.com/questions/1026466/ddg#1026469
> <http://serverfault.com/questions/1026466/ddg#1026469>
>
> We originally had a problem where IPv6 addresses needed to be in
> brackets to support URL representation. In TripleO at one time we
> even had two versions of hiera values, with and without the
> brackets. I don’t know if that still exists or not.
>
> I wouldn’t hold my breath about getting such a change into dnsmasq.
> It is notoriously difficult to get patches accepted into dnsmasq,
> especially where UI is concerned.
>
> -Dan Sneddon
>
> On Thu, Jun 16, 2022 at 5:32 AM Brendan Shephard
> <bshephar at redhat.com <mailto:bshephar at redhat.com>> wrote:
>
> Hey,
>
> Looks like that is the problem. The [ ] around the IP address
> are causing the issue. If I try to run dnsmasq using exactly the
> output you get, it gives me the same error:
> [root at tripleo-director ~]# /usr/sbin/dnsmasq
> --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log
> --user=root --conf-file=/dev/null
> --listen-address=[aaaa:aaaa:aaaa::1] --port=0 --enable-tftp
> --tftp-root=/var/lib/ironic/tftpboot
>
> dnsmasq: bad command line options: try --help
>
> VS without the [ ] I can see it starts up normally.
>
> The settings in your undercloud.conf file look to be correct I
> believe. So I think there might be a bug here. I don't think we
> should be saving that value with the square brackets, or we
> would need to filter them out when we gather the value in that
> variable.
>
> I raised a bug for it here so that we can dig into this and find
> what needs fixing:
> https://bugs.launchpad.net/tripleo/+bug/1978892
> <https://bugs.launchpad.net/tripleo/+bug/1978892>
>
> In the meantime, if you edit that hieradata value, are you able
> to get that container started?
>
> Change this:
> [root at tripleo-director ~]# egrep -r 'tftp_bind_host'
> /etc/puppet/hieradata/
> /etc/puppet/hieradata/service_configs.json:
> "ironic::pxe::tftp_bind_host": "%{lookup('ctlplane_uri')}",
>
> To this:
> "ironic::pxe::tftp_bind_host": "aaaa:aaaa:aaaa::1"
>
> Then restart the service:
> sudo systemctl restart tripleo_ironic_pxe_http.service
> tripleo_ironic_pxe_tftp.service
>
> Does that get the container running without the error? I did the
> same in my environment and can see that dnsmasq is running
> properly like that:
> [root at tripleo-director ~]# ps -ef | grep aaaa
> root 71180 52675 0 19:24 pts/4 00:00:00
> /usr/sbin/dnsmasq --keep-in-foreground
> --log-facility=/var/log/ironic/dnsmasq.log --user=root
> --conf-file=/dev/null --listen-address=aaaa:aaaa:aaaa::1
> --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot
>
> Brendan Shephard
>
> Software Engineer
>
> Red Hat APAC <https://www.redhat.com>
>
> 193 N Quay
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
>
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g><https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g><https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
>
> Brisbane City QLD
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
> 4000
>
> @RedHat <https://twitter.com/redhat> Red Hat
> <https://www.linkedin.com/company/red-hat> Red Hat
> <https://www.facebook.com/RedHatInc>
> <https://red.ht/sig>
> <https://redhat.com/summit>
>
>
>
> On Thu, Jun 16, 2022 at 12:12 AM Lokendra Rathour
> <lokendrarathour at gmail.com <mailto:lokendrarathour at gmail.com>>
> wrote:
>
> Hi Shephard,
> I am getting the local_ip (ipv6) of the undercloud :
>
> [root at undercloud stack]# sudo hiera
> ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml
> [aaaa:aaaa:aaaa::1]
>
> is this because of some ipv6 reasons?
>
>
> On Wed, Jun 15, 2022 at 6:08 PM Brendan Shephard
> <bshephar at redhat.com <mailto:bshephar at redhat.com>> wrote:
>
> Hey,
>
> Ok, that command looks fine. What about that variable
> there? Do you get anything back when you run:
> sudo hiera ironic::pxe::tftp_bind_host -c
> /etc/puppet/hiera.yaml
>
> Mine returns:
> sudo hiera ironic::pxe::tftp_bind_host -c
> /etc/puppet/hiera.yaml
> 192.168.24.115
>
> Brendan Shephard
>
> Software Engineer
>
> Red Hat APAC <https://www.redhat.com>
>
> 193 N Quay
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
>
> Brisbane City QLD
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
> 4000
>
> @RedHat <https://twitter.com/redhat>
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
> Red Hat <https://www.linkedin.com/company/red-hat>
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
> Red Hat <https://www.facebook.com/RedHatInc>
> <https://red.ht/sig>
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
>
> <https://redhat.com/summit>
>
>
>
> On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour
> <lokendrarathour at gmail.com
> <mailto:lokendrarathour at gmail.com>> wrote:
>
> Hi Shephard,
>
> this is the command from my wallaby:
> [root at undercloud ~]# sudo cat
> /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json
> {
> "cap_add": [
> "NET_ADMIN",
> "NET_RAW",
> "SETUID"
> ],
> "command": [
> "/bin/bash",
> "-c",
> "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host
> -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq
> --keep-in-foreground
> --log-facility=/var/log/ironic/dnsmasq.log
> --user=root --conf-file=/dev/null
> --listen-address=$BIND_HOST --port=0 --enable-tftp
> --tftp-root=/var/lib/ironic/tftpboot"
> ],
> "environment": {
> "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS",
> "TRIPLEO_CONFIG_HASH":
> "9fb3e4e0e35ee35fdf74cfccb16a7543"
> },
> "healthcheck": {
> "test": "/openstack/healthcheck"
> },
> "image":
> "undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo",
> "net": "host",
> "privileged": false,
> "restart": "always",
> "start_order": 90,
> "volumes": [
> "/etc/hosts:/etc/hosts:ro",
> "/etc/localtime:/etc/localtime:ro",
>
> "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
>
> "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",
>
> "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
>
> "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro",
> "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro",
> "/dev/log:/dev/log",
> "/etc/puppet:/etc/puppet:ro",
>
> "/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro",
>
> "/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro",
> "/var/lib/ironic:/var/lib/ironic:shared,z",
> "/var/log/containers/ironic:/var/log/ironic:z",
>
> "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z"
> ]
> }[root at undercloud ~]#
>
> Comparing both, they look alike.
> please check once.
>
> On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard
> <bshephar at redhat.com <mailto:bshephar at redhat.com>>
> wrote:
>
> Hi,
>
> Looks like the command was in a different file
> in Wallaby, can you check:
> sudo cat
> /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json
>
> That one should have the dnsmasq command it's
> trying to run. For example, here it is from my
> Wallaby environment:
> [stack at undercloud-0 ~]$ sudo cat
> /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json
> | jq .command
> [
> "/bin/bash",
> "-c",
> "BIND_HOST=$(hiera
> ironic::pxe::tftp_bind_host -c
> /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq
> --keep-in-foreground
> --log-facility=/var/log/ironic/dnsmasq.log
> --user=root --conf-file=/dev/null
> --listen-address=$BIND_HOST --port=0
> --enable-tftp --tftp-root=/var/lib/ironic/tftpboot"
> ]
>
>
>
> Brendan Shephard
>
> Software Engineer
>
> Red Hat APAC <https://www.redhat.com>
>
> 193 N Quay
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
>
> Brisbane City QLD
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
> 4000
>
> @RedHat <https://twitter.com/redhat> Red Hat
> <https://www.linkedin.com/company/red-hat> Red
> Hat <https://www.facebook.com/RedHatInc>
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
> <https://red.ht/sig>
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
>
> <https://redhat.com/summit>
>
>
>
> On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour
> <lokendrarathour at gmail.com
> <mailto:lokendrarathour at gmail.com>> wrote:
>
> Hi Shephard,
> Here is the o/p of the file:
>
> [root at undercloud ~]# sudo cat
> /var/lib/kolla/config_files/ironic_pxe_tftp.json
> {
> "config_files": [
> {
> "dest": "/",
> "merge": true,
> "preserve_properties": true,
> "source":
> "/var/lib/kolla/config_files/src/*"
> }
> ],
> "permissions": [
> {
> "owner": "ironic:ironic",
> "path": "/var/log/ironic",
> "recurse": true
> },
> {
> "owner": "ironic:ironic",
> "path": "/var/lib/ironic",
> "recurse": true
> }
> ]
> }[root at undercloud ~]#
>
>
> Thanks once agan.
>
> -Lokendra
>
>
> On Wed, Jun 15, 2022 at 2:38 PM Brendan
> Shephard <bshephar at redhat.com
> <mailto:bshephar at redhat.com>> wrote:
>
> Looks like something wrong with the
> dnsmasq command the container is being
> launched with. What command is it trying
> to run?
>
> sudo cat
> /var/lib/kolla/config_files/ironic_pxe_tftp.json
>
> Brendan
> Shephard<https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
>
> Software Engineer
>
> Red Hat APAC <https://www.redhat.com>
>
> 193 N Quay
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
>
> Brisbane City QLD
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
> 4000
>
> @RedHat <https://twitter.com/redhat> Red
> Hat
> <https://www.linkedin.com/company/red-hat>
> Red Hat
> <https://www.facebook.com/RedHatInc>
> <https://red.ht/sig>
> <https://redhat.com/summit>
>
>
>
> On Wed, Jun 15, 2022 at 6:22 PM Anirudh
> Gupta <anyrude10 at gmail.com
> <mailto:anyrude10 at gmail.com>> wrote:
>
> Hi Brendan,
>
> Thanks for your response.
>
> Please find the log below.
>
> [stack at undercloud t2u2v2w]$ sudo
> podman logs ironic_pxe_tftp
>
> dnsmasq: bad command line options:
> try --help
> dnsmasq: bad command line options:
> try --help
> dnsmasq: bad command line options:
> try --help
> dnsmasq: bad command line options:
> try --help
> dnsmasq: bad command line options:
> try --help
> dnsmasq: bad command line options:
> try --help
>
> [stack at undercloud t2u2v2w]$ sudo
> podman ps --filter name=ironic_pxe -a
> CONTAINER ID IMAGE
>
>
> COMMAND CREATED
> STATUS
> PORTS NAMES
> 02dacbc74cec
> undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo /bin/bash -c BIND... 3 hours ago Exited (1) 3 hours ago (unhealthy) ironic_pxe_tftp
> 1f8ca39fba32
> undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo kolla_start 3 hours ago Up 3 hours ago (healthy) ironic_pxe_http
>
>
> Regards
>
> Anirudh Gupta
>
>
> On Wed, Jun 15, 2022 at 11:30 AM
> Brendan Shephard
> <bshephar at redhat.com
> <mailto:bshephar at redhat.com>> wrote:
>
> Hey Anirudh,
>
> You would need to look at the
> logs for the ironic_pxe_tftp
> container to see why it's failing.
>
> I assume the tftp container is
> not Up when you run this command?
> [stack at tripleo-director
> overcloud_playbooks]$ sudo
> podman ps --filter
> name=ironic_pxe -a
> CONTAINER ID IMAGE
>
>
> COMMAND CREATED
> STATUS
> PORTS NAMES
> 0170be36e291
> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo
> <http://registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo>
> kolla_start 12 days ago Up
> 30 hours ago (healthy)
> ironic_pxe_tftp
> e507f722bdf0
> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo
> <http://registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo>
> kolla_start 12 days ago Up
> 30 hours ago (healthy)
> ironic_pxe_http
>
> Then check the logs to see what
> the error is:
> [stack at tripleo-director
> overcloud_playbooks]$ sudo
> podman logs ironic_pxe_tftp
>
>
>
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
>
>
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
> Brendan Shephard
>
> Software Engineer
>
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
> Red Hat APAC
> <https://www.redhat.com>
>
> 193 N Quay
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
>
> Brisbane City QLD
> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
> 4000
>
> @RedHat
> <https://twitter.com/redhat> Red
> Hat
> <https://www.linkedin.com/company/red-hat>
> Red Hat
> <https://www.facebook.com/RedHatInc>
>
> <https://red.ht/sig>
> <https://redhat.com/summit>
>
>
>
> On Wed, Jun 15, 2022 at 7:53 AM
> Anirudh Gupta
> <anyrude10 at gmail.com
> <mailto:anyrude10 at gmail.com>> wrote:
>
> Hi Team,
>
> I am trying to deploy
> Openstack Wallaby Undercloud
> on IPv6, but facing the
> below error:
>
> 2022-06-14 05:01:23.213708 |
> 52540083-cfa2-3f20-e9dc-00000000286f
> | TASK | Manage container
> systemd services and cleanup
> old systemd healthchecks for
> /var/lib/tripleo-config/container-startup-config/step_4
> 2022-06-14 05:03:22.912816 |
> 52540083-cfa2-3f20-e9dc-00000000286f
> | FATAL | Manage container
> systemd services and cleanup
> old systemd healthchecks for
> /var/lib/tripleo-config/container-startup-config/step_4
> | undercloud |
> error={"changed": false,
> "msg": "Service
> ironic_pxe_tftp has not
> started yet"}
> 2022-06-14 05:03:22.914400 |
> 52540083-cfa2-3f20-e9dc-00000000286f
> | TIMING |
> tripleo_container_manage :
> Manage container systemd
>
> Sample Undercloud.conf is as
> follows:
>
> [DEFAULT]
> clean_nodes = true
> cleanup = false
> container_cli = podman
> container_healthcheck_disabled
> = true
> container_images_file =
> /home/stack/containers-prepare-parameter.yaml
> deployment_user = stack
> enable_ironic = true
> enable_ironic_inspector = true
> enable_neutron = true
> enable_routed_networks = false
> generate_service_certificate
> = false
> ipv6_address_mode =
> dhcpv6-stateful
> ipxe_enabled = true
> local_interface = enp8s0
> local_ip = aaaa:aaaa:aaaa::1/64
> subnets = ctlplane-subnet
> undercloud_admin_host =
> aaaa:aaaa:aaaa::1
> undercloud_hostname =
> undercloud.com
> <http://undercloud.com>
> undercloud_ntp_servers =
> 30.30.30.3
> undercloud_public_host =
> aaaa:aaaa:aaaa::1
> undercloud_timezone = UTC
>
> [ctlplane-subnet]
> cidr = aaaa:aaaa:aaaa::/64
> dhcp_end = aaaa:aaaa:aaaa::f
> dhcp_start = aaaa:aaaa:aaaa::a
> gateway = aaaa:aaaa:aaaa::1
> inspection_iprange =
> aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9
>
> Can someone please help in
> this regard.
>
> Anirudh Gupta
>
>
>
> --
> ~ Lokendra
> www.inertiaspeaks.com
> <http://www.inertiaspeaks.com>
> www.inertiagroups.com
> <http://www.inertiagroups.com>
> skype: lokendrarathour
>
>
>
>
> --
> ~ Lokendra
> www.inertiaspeaks.com <http://www.inertiaspeaks.com>
> www.inertiagroups.com <http://www.inertiagroups.com>
> skype: lokendrarathour
>
>
>
>
> --
> ~ Lokendra
> www.inertiaspeaks.com <http://www.inertiaspeaks.com>
> www.inertiagroups.com <http://www.inertiagroups.com>
> skype: lokendrarathour
>
>
> --
> Dan Sneddon | Senior Principal Software Engineer
> dsneddon at redhat.com <mailto:dsneddon at redhat.com> | redhat.com/cloud
> <http://redhat.com/cloud>
> dsneddon:irc | @dxs:twitter
>
>
>
> --
> ~ Lokendra
> www.inertiaspeaks.com <http://www.inertiaspeaks.com>
> www.inertiagroups.com <http://www.inertiagroups.com>
> skype: lokendrarathour
>
>
More information about the openstack-discuss
mailing list