Hey,

Looks like that is the problem. The [ ] around the IP address are causing the issue. If I try to run dnsmasq using exactly the output you get, it gives me the same error:
[root@tripleo-director ~]# /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=[aaaa:aaaa:aaaa::1] --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot

dnsmasq: bad command line options: try --help

VS without the [ ] I can see it starts up normally.

The settings in your undercloud.conf file look to be correct I believe. So I think there might be a bug here. I don't think we should be saving that value with the square brackets, or we would need to filter them out when we gather the value in that variable.

I raised a bug for it here so that we can dig into this and find what needs fixing:
https://bugs.launchpad.net/tripleo/+bug/1978892

In the meantime, if you edit that hieradata value, are you able to get that container started?

Change this:
[root@tripleo-director ~]# egrep -r 'tftp_bind_host' /etc/puppet/hieradata/
/etc/puppet/hieradata/service_configs.json:    "ironic::pxe::tftp_bind_host": "%{lookup('ctlplane_uri')}",

To this:
 "ironic::pxe::tftp_bind_host": "aaaa:aaaa:aaaa::1"

Then restart the service:
sudo systemctl restart tripleo_ironic_pxe_http.service tripleo_ironic_pxe_tftp.service

Does that get the container running without the error? I did the same in my environment and can see that dnsmasq is running properly like that:
[root@tripleo-director ~]# ps -ef | grep aaaa
root       71180   52675  0 19:24 pts/4    00:00:00 /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=aaaa:aaaa:aaaa::1 --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot

Brendan Shephard

Software Engineer

Red Hat APAC

193 N Quay

Brisbane City QLD 4000

@RedHat   Red Hat   Red Hat

On Thu, Jun 16, 2022 at 12:12 AM Lokendra Rathour <lokendrarathour@gmail.com> wrote:

Hi Shephard,

I am getting the local_ip (ipv6) of the undercloud :

[root@undercloud stack]# sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml
[aaaa:aaaa:aaaa::1]

is this because of some ipv6 reasons?

On Wed, Jun 15, 2022 at 6:08 PM Brendan Shephard <bshephar@redhat.com> wrote:

Hey,

Ok, that command looks fine. What about that variable there? Do you get anything back when you run:

sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml

Mine returns:

sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml
192.168.24.115

Brendan Shephard

Software Engineer

Red Hat APAC

193 N Quay

Brisbane City QLD 4000

@RedHat   Red Hat   Red Hat

On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour <lokendrarathour@gmail.com> wrote:

Hi Shephard,

this is the command from my wallaby:

[root@undercloud ~]# sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json
{
  "cap_add": [
    "NET_ADMIN",
    "NET_RAW",
    "SETUID"
  ],
  "command": [
    "/bin/bash",
    "-c",
    "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot"
  ],
  "environment": {
    "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS",
    "TRIPLEO_CONFIG_HASH": "9fb3e4e0e35ee35fdf74cfccb16a7543"
  },
  "healthcheck": {
    "test": "/openstack/healthcheck"
  },
  "image": "undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo",
  "net": "host",
  "privileged": false,
  "restart": "always",
  "start_order": 90,
  "volumes": [
    "/etc/hosts:/etc/hosts:ro",
    "/etc/localtime:/etc/localtime:ro",
    "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
    "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",
    "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
    "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro",
    "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro",
    "/dev/log:/dev/log",
    "/etc/puppet:/etc/puppet:ro",
    "/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro",
    "/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro",
    "/var/lib/ironic:/var/lib/ironic:shared,z",
    "/var/log/containers/ironic:/var/log/ironic:z",
    "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z"
  ]
}[root@undercloud ~]#

Comparing both, they look alike.

please check once.

On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard <bshephar@redhat.com> wrote:

Hi,

Looks like the command was in a different file in Wallaby, can you check:

sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json

That one should have the dnsmasq command it's trying to run. For example, here it is from my Wallaby environment:

[stack@undercloud-0 ~]$ sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json | jq .command
[
  "/bin/bash",
  "-c",
  "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot"
]

Brendan Shephard

Software Engineer

Red Hat APAC

193 N Quay

Brisbane City QLD 4000

@RedHat   Red Hat   Red Hat

On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour <lokendrarathour@gmail.com> wrote:

Hi Shephard,

Here is the o/p of the file:

[root@undercloud ~]# sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json
{
  "config_files": [
    {
      "dest": "/",
      "merge": true,
      "preserve_properties": true,
      "source": "/var/lib/kolla/config_files/src/*"
    }
  ],
  "permissions": [
    {
      "owner": "ironic:ironic",
      "path": "/var/log/ironic",
      "recurse": true
    },
    {
      "owner": "ironic:ironic",
      "path": "/var/lib/ironic",
      "recurse": true
    }
  ]
}[root@undercloud ~]#

Thanks once agan. 

-Lokendra

On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard <bshephar@redhat.com> wrote:

Looks like something wrong with the dnsmasq command the container is being launched with. What command is it trying to run?

sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json

Brendan Shephard

Software Engineer

Red Hat APAC

193 N Quay

Brisbane City QLD 4000

@RedHat   Red Hat   Red Hat

On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta <anyrude10@gmail.com> wrote:

Hi Brendan,

Thanks for your response.

Please find the log below.

[stack@undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp

dnsmasq: bad command line options: try --help
dnsmasq: bad command line options: try --help
dnsmasq: bad command line options: try --help
dnsmasq: bad command line options: try --help
dnsmasq: bad command line options: try --help
dnsmasq: bad command line options: try --help

[stack@undercloud t2u2v2w]$  sudo podman ps --filter name=ironic_pxe -a
CONTAINER ID  IMAGE                                                                                     COMMAND               CREATED      STATUS                              PORTS       NAMES
02dacbc74cec  undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo  /bin/bash -c BIND...  3 hours ago  Exited (1) 3 hours ago (unhealthy)              ironic_pxe_tftp
1f8ca39fba32  undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo  kolla_start           3 hours ago  Up 3 hours ago (healthy)                        ironic_pxe_http

Regards

Anirudh Gupta

On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard <bshephar@redhat.com> wrote:

Hey Anirudh,

You would need to look at the logs for the ironic_pxe_tftp container to see why it's failing.

I assume the tftp container is not Up when you run this command?
[stack@tripleo-director overcloud_playbooks]$ sudo podman ps --filter name=ironic_pxe -a
CONTAINER ID  IMAGE                                                                                       COMMAND      CREATED      STATUS                     PORTS       NAMES
0170be36e291  registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo  kolla_start  12 days ago  Up 30 hours ago (healthy)              ironic_pxe_tftp
e507f722bdf0  registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo  kolla_start  12 days ago  Up 30 hours ago (healthy)              ironic_pxe_http

Then check the logs to see what the error is:
[stack@tripleo-director overcloud_playbooks]$ sudo podman logs ironic_pxe_tftp

Brendan Shephard

Software Engineer

Red Hat APAC

193 N Quay

Brisbane City QLD 4000

@RedHat   Red Hat   Red Hat

On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta <anyrude10@gmail.com> wrote:

Hi Team,

I am trying to deploy Openstack Wallaby Undercloud on IPv6, but facing the below error:

2022-06-14 05:01:23.213708 | 52540083-cfa2-3f20-e9dc-00000000286f | TASK | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4
2022-06-14 05:03:22.912816 | 52540083-cfa2-3f20-e9dc-00000000286f | FATAL | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 | undercloud | error={"changed": false, "msg": "Service ironic_pxe_tftp has not started yet"}
2022-06-14 05:03:22.914400 | 52540083-cfa2-3f20-e9dc-00000000286f | TIMING | tripleo_container_manage : Manage container systemd

Sample Undercloud.conf is as follows:

[DEFAULT]
clean_nodes = true
cleanup = false
container_cli = podman
container_healthcheck_disabled = true
container_images_file = /home/stack/containers-prepare-parameter.yaml
deployment_user = stack
enable_ironic = true
enable_ironic_inspector = true
enable_neutron = true
enable_routed_networks = false
generate_service_certificate = false
ipv6_address_mode = dhcpv6-stateful
ipxe_enabled = true
local_interface = enp8s0
local_ip = aaaa:aaaa:aaaa::1/64
subnets = ctlplane-subnet
undercloud_admin_host = aaaa:aaaa:aaaa::1
undercloud_hostname = undercloud.com
undercloud_ntp_servers = 30.30.30.3
undercloud_public_host = aaaa:aaaa:aaaa::1
undercloud_timezone = UTC

[ctlplane-subnet]
cidr = aaaa:aaaa:aaaa::/64
dhcp_end = aaaa:aaaa:aaaa::f
dhcp_start = aaaa:aaaa:aaaa::a
gateway = aaaa:aaaa:aaaa::1
inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9

Can someone please help in this regard.

Anirudh Gupta

--

~ Lokendra

www.inertiaspeaks.com

www.inertiagroups.com

skype: lokendrarathour

--

~ Lokendra

www.inertiaspeaks.com

www.inertiagroups.com

skype: lokendrarathour

--

~ Lokendra

www.inertiaspeaks.com

www.inertiagroups.com

skype: lokendrarathour