Looks like something wrong with the dnsmasq command the container is being launched with. What command is it trying to run? sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json Brendan Shephard Software Engineer Red Hat APAC <https://www.redhat.com> 193 N Quay Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit> On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta <anyrude10@gmail.com> wrote:

Hi Brendan,

Thanks for your response.

Please find the log below.

[stack@undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp

dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help

[stack@undercloud t2u2v2w]$ sudo podman ps --filter name=ironic_pxe -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 02dacbc74cec undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo /bin/bash -c BIND... 3 hours ago Exited (1) 3 hours ago (unhealthy) ironic_pxe_tftp 1f8ca39fba32 undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo kolla_start 3 hours ago Up 3 hours ago (healthy) ironic_pxe_http

Regards

Anirudh Gupta

On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hey Anirudh,

You would need to look at the logs for the ironic_pxe_tftp container to see why it's failing.

I assume the tftp container is not Up when you run this command? [stack@tripleo-director overcloud_playbooks]$ sudo podman ps --filter name=ironic_pxe -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0170be36e291 registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo kolla_start 12 days ago Up 30 hours ago (healthy) ironic_pxe_tftp e507f722bdf0 registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo kolla_start 12 days ago Up 30 hours ago (healthy) ironic_pxe_http

Then check the logs to see what the error is: [stack@tripleo-director overcloud_playbooks]$ sudo podman logs ironic_pxe_tftp

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta <anyrude10@gmail.com> wrote:

...

Hi Team,

I am trying to deploy Openstack Wallaby Undercloud on IPv6, but facing the below error:

2022-06-14 05:01:23.213708 | 52540083-cfa2-3f20-e9dc-00000000286f | TASK | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 2022-06-14 05:03:22.912816 | 52540083-cfa2-3f20-e9dc-00000000286f | FATAL | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 | undercloud | error={"changed": false, "msg": "Service ironic_pxe_tftp has not started yet"} 2022-06-14 05:03:22.914400 | 52540083-cfa2-3f20-e9dc-00000000286f | TIMING | tripleo_container_manage : Manage container systemd

Sample Undercloud.conf is as follows:

[DEFAULT] clean_nodes = true cleanup = false container_cli = podman container_healthcheck_disabled = true container_images_file = /home/stack/containers-prepare-parameter.yaml deployment_user = stack enable_ironic = true enable_ironic_inspector = true enable_neutron = true enable_routed_networks = false generate_service_certificate = false ipv6_address_mode = dhcpv6-stateful ipxe_enabled = true local_interface = enp8s0 local_ip = aaaa:aaaa:aaaa::1/64 subnets = ctlplane-subnet undercloud_admin_host = aaaa:aaaa:aaaa::1 undercloud_hostname = undercloud.com undercloud_ntp_servers = 30.30.30.3 undercloud_public_host = aaaa:aaaa:aaaa::1 undercloud_timezone = UTC

[ctlplane-subnet] cidr = aaaa:aaaa:aaaa::/64 dhcp_end = aaaa:aaaa:aaaa::f dhcp_start = aaaa:aaaa:aaaa::a gateway = aaaa:aaaa:aaaa::1 inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9

Can someone please help in this regard.

Anirudh Gupta

Hi, Looks like the command was in a different file in Wallaby, can you check: sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json That one should have the dnsmasq command it's trying to run. For example, here it is from my Wallaby environment: [stack@undercloud-0 ~]$ sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json | jq .command [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ] Brendan Shephard Software Engineer Red Hat APAC <https://www.redhat.com> 193 N Quay Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit> On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour <lokendrarathour@gmail.com> wrote:

Hi Shephard, Here is the o/p of the file:

[root@undercloud ~]# sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json { "config_files": [ { "dest": "/", "merge": true, "preserve_properties": true, "source": "/var/lib/kolla/config_files/src/*" } ], "permissions": [ { "owner": "ironic:ironic", "path": "/var/log/ironic", "recurse": true }, { "owner": "ironic:ironic", "path": "/var/lib/ironic", "recurse": true } ] }[root@undercloud ~]#

Thanks once agan.

-Lokendra

On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard <bshephar@redhat.com> wrote:

...

Looks like something wrong with the dnsmasq command the container is being launched with. What command is it trying to run?

sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta <anyrude10@gmail.com> wrote:

...

Hi Brendan,

Thanks for your response.

Please find the log below.

[stack@undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp

dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help

[stack@undercloud t2u2v2w]$ sudo podman ps --filter name=ironic_pxe -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 02dacbc74cec undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo /bin/bash -c BIND... 3 hours ago Exited (1) 3 hours ago (unhealthy) ironic_pxe_tftp 1f8ca39fba32 undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo kolla_start 3 hours ago Up 3 hours ago (healthy) ironic_pxe_http

Regards

Anirudh Gupta

On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hey Anirudh,

You would need to look at the logs for the ironic_pxe_tftp container to see why it's failing.

I assume the tftp container is not Up when you run this command? [stack@tripleo-director overcloud_playbooks]$ sudo podman ps --filter name=ironic_pxe -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0170be36e291 registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo kolla_start 12 days ago Up 30 hours ago (healthy) ironic_pxe_tftp e507f722bdf0 registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo kolla_start 12 days ago Up 30 hours ago (healthy) ironic_pxe_http

Then check the logs to see what the error is: [stack@tripleo-director overcloud_playbooks]$ sudo podman logs ironic_pxe_tftp

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta <anyrude10@gmail.com> wrote:

...

Hi Team,

I am trying to deploy Openstack Wallaby Undercloud on IPv6, but facing the below error:

2022-06-14 05:01:23.213708 | 52540083-cfa2-3f20-e9dc-00000000286f | TASK | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 2022-06-14 05:03:22.912816 | 52540083-cfa2-3f20-e9dc-00000000286f | FATAL | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 | undercloud | error={"changed": false, "msg": "Service ironic_pxe_tftp has not started yet"} 2022-06-14 05:03:22.914400 | 52540083-cfa2-3f20-e9dc-00000000286f | TIMING | tripleo_container_manage : Manage container systemd

Sample Undercloud.conf is as follows:

[DEFAULT] clean_nodes = true cleanup = false container_cli = podman container_healthcheck_disabled = true container_images_file = /home/stack/containers-prepare-parameter.yaml deployment_user = stack enable_ironic = true enable_ironic_inspector = true enable_neutron = true enable_routed_networks = false generate_service_certificate = false ipv6_address_mode = dhcpv6-stateful ipxe_enabled = true local_interface = enp8s0 local_ip = aaaa:aaaa:aaaa::1/64 subnets = ctlplane-subnet undercloud_admin_host = aaaa:aaaa:aaaa::1 undercloud_hostname = undercloud.com undercloud_ntp_servers = 30.30.30.3 undercloud_public_host = aaaa:aaaa:aaaa::1 undercloud_timezone = UTC

[ctlplane-subnet] cidr = aaaa:aaaa:aaaa::/64 dhcp_end = aaaa:aaaa:aaaa::f dhcp_start = aaaa:aaaa:aaaa::a gateway = aaaa:aaaa:aaaa::1 inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9

Can someone please help in this regard.

Anirudh Gupta

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

Hey, Ok, that command looks fine. What about that variable there? Do you get anything back when you run: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml Mine returns: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml 192.168.24.115 Brendan Shephard Software Engineer Red Hat APAC <https://www.redhat.com> 193 N Quay Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit> On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour <lokendrarathour@gmail.com> wrote:

Hi Shephard,

this is the command from my wallaby: [root@undercloud ~]# sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json { "cap_add": [ "NET_ADMIN", "NET_RAW", "SETUID" ], "command": [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ], "environment": { "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "TRIPLEO_CONFIG_HASH": "9fb3e4e0e35ee35fdf74cfccb16a7543" }, "healthcheck": { "test": "/openstack/healthcheck" }, "image": "undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo", "net": "host", "privileged": false, "restart": "always", "start_order": 90, "volumes": [ "/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro", "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro", "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro", "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",

"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/etc/puppet:/etc/puppet:ro",

"/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro",

"/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro", "/var/lib/ironic:/var/lib/ironic:shared,z", "/var/log/containers/ironic:/var/log/ironic:z", "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z" ] }[root@undercloud ~]#

Comparing both, they look alike. please check once.

On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hi,

Looks like the command was in a different file in Wallaby, can you check: sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json

That one should have the dnsmasq command it's trying to run. For example, here it is from my Wallaby environment: [stack@undercloud-0 ~]$ sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json | jq .command [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ]

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour < lokendrarathour@gmail.com> wrote:

...

Hi Shephard, Here is the o/p of the file:

[root@undercloud ~]# sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json { "config_files": [ { "dest": "/", "merge": true, "preserve_properties": true, "source": "/var/lib/kolla/config_files/src/*" } ], "permissions": [ { "owner": "ironic:ironic", "path": "/var/log/ironic", "recurse": true }, { "owner": "ironic:ironic", "path": "/var/lib/ironic", "recurse": true } ] }[root@undercloud ~]#

Thanks once agan.

-Lokendra

On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard <bshephar@redhat.com> wrote:

...

Looks like something wrong with the dnsmasq command the container is being launched with. What command is it trying to run?

sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta <anyrude10@gmail.com> wrote:

...

Hi Brendan,

Thanks for your response.

Please find the log below.

[stack@undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp

dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help

[stack@undercloud t2u2v2w]$ sudo podman ps --filter name=ironic_pxe -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 02dacbc74cec undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo /bin/bash -c BIND... 3 hours ago Exited (1) 3 hours ago (unhealthy) ironic_pxe_tftp 1f8ca39fba32 undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo kolla_start 3 hours ago Up 3 hours ago (healthy) ironic_pxe_http

Regards

Anirudh Gupta

On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hey Anirudh,

You would need to look at the logs for the ironic_pxe_tftp container to see why it's failing.

I assume the tftp container is not Up when you run this command? [stack@tripleo-director overcloud_playbooks]$ sudo podman ps --filter name=ironic_pxe -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0170be36e291 registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo kolla_start 12 days ago Up 30 hours ago (healthy) ironic_pxe_tftp e507f722bdf0 registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo kolla_start 12 days ago Up 30 hours ago (healthy) ironic_pxe_http

Then check the logs to see what the error is: [stack@tripleo-director overcloud_playbooks]$ sudo podman logs ironic_pxe_tftp

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta <anyrude10@gmail.com> wrote:

> Hi Team, > > I am trying to deploy Openstack Wallaby Undercloud on IPv6, but > facing the below error: > > 2022-06-14 05:01:23.213708 | 52540083-cfa2-3f20-e9dc-00000000286f | > TASK | Manage container systemd services and cleanup old systemd > healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 > 2022-06-14 05:03:22.912816 | 52540083-cfa2-3f20-e9dc-00000000286f | > FATAL | Manage container systemd services and cleanup old systemd > healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 | > undercloud | error={"changed": false, "msg": "Service ironic_pxe_tftp has > not started yet"} > 2022-06-14 05:03:22.914400 | 52540083-cfa2-3f20-e9dc-00000000286f | > TIMING | tripleo_container_manage : Manage container systemd > > Sample Undercloud.conf is as follows: > > [DEFAULT] > clean_nodes = true > cleanup = false > container_cli = podman > container_healthcheck_disabled = true > container_images_file = /home/stack/containers-prepare-parameter.yaml > deployment_user = stack > enable_ironic = true > enable_ironic_inspector = true > enable_neutron = true > enable_routed_networks = false > generate_service_certificate = false > ipv6_address_mode = dhcpv6-stateful > ipxe_enabled = true > local_interface = enp8s0 > local_ip = aaaa:aaaa:aaaa::1/64 > subnets = ctlplane-subnet > undercloud_admin_host = aaaa:aaaa:aaaa::1 > undercloud_hostname = undercloud.com > undercloud_ntp_servers = 30.30.30.3 > undercloud_public_host = aaaa:aaaa:aaaa::1 > undercloud_timezone = UTC > > [ctlplane-subnet] > cidr = aaaa:aaaa:aaaa::/64 > dhcp_end = aaaa:aaaa:aaaa::f > dhcp_start = aaaa:aaaa:aaaa::a > gateway = aaaa:aaaa:aaaa::1 > inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9 > > Can someone please help in this regard. > > Anirudh Gupta > >

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

Hey, Looks like that is the problem. The [ ] around the IP address are causing the issue. If I try to run dnsmasq using exactly the output you get, it gives me the same error: [root@tripleo-director ~]# /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=[aaaa:aaaa:aaaa::1] --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot dnsmasq: bad command line options: try --help VS without the [ ] I can see it starts up normally. The settings in your undercloud.conf file look to be correct I believe. So I think there might be a bug here. I don't think we should be saving that value with the square brackets, or we would need to filter them out when we gather the value in that variable. I raised a bug for it here so that we can dig into this and find what needs fixing: https://bugs.launchpad.net/tripleo/+bug/1978892 In the meantime, if you edit that hieradata value, are you able to get that container started? Change this: [root@tripleo-director ~]# egrep -r 'tftp_bind_host' /etc/puppet/hieradata/ /etc/puppet/hieradata/service_configs.json: "ironic::pxe::tftp_bind_host": "%{lookup('ctlplane_uri')}", To this: "ironic::pxe::tftp_bind_host": "aaaa:aaaa:aaaa::1" Then restart the service: sudo systemctl restart tripleo_ironic_pxe_http.service tripleo_ironic_pxe_tftp.service Does that get the container running without the error? I did the same in my environment and can see that dnsmasq is running properly like that: [root@tripleo-director ~]# ps -ef | grep aaaa root 71180 52675 0 19:24 pts/4 00:00:00 /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=aaaa:aaaa:aaaa::1 --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot Brendan Shephard Software Engineer Red Hat APAC <https://www.redhat.com> 193 N Quay Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit> On Thu, Jun 16, 2022 at 12:12 AM Lokendra Rathour <lokendrarathour@gmail.com> wrote:

Hi Shephard, I am getting the local_ip (ipv6) of the undercloud :

[root@undercloud stack]# sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml [aaaa:aaaa:aaaa::1]

is this because of some ipv6 reasons?

On Wed, Jun 15, 2022 at 6:08 PM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hey,

Ok, that command looks fine. What about that variable there? Do you get anything back when you run: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml

Mine returns: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml 192.168.24.115

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour < lokendrarathour@gmail.com> wrote:

...

Hi Shephard,

this is the command from my wallaby: [root@undercloud ~]# sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json { "cap_add": [ "NET_ADMIN", "NET_RAW", "SETUID" ], "command": [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ], "environment": { "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "TRIPLEO_CONFIG_HASH": "9fb3e4e0e35ee35fdf74cfccb16a7543" }, "healthcheck": { "test": "/openstack/healthcheck" }, "image": "undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo", "net": "host", "privileged": false, "restart": "always", "start_order": 90, "volumes": [ "/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro", "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",

"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",

"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",

"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/etc/puppet:/etc/puppet:ro",

"/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro",

"/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro", "/var/lib/ironic:/var/lib/ironic:shared,z", "/var/log/containers/ironic:/var/log/ironic:z", "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z" ] }[root@undercloud ~]#

Comparing both, they look alike. please check once.

On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hi,

Looks like the command was in a different file in Wallaby, can you check: sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json

That one should have the dnsmasq command it's trying to run. For example, here it is from my Wallaby environment: [stack@undercloud-0 ~]$ sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json | jq .command [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ]

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour < lokendrarathour@gmail.com> wrote:

...

Hi Shephard, Here is the o/p of the file:

[root@undercloud ~]# sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json { "config_files": [ { "dest": "/", "merge": true, "preserve_properties": true, "source": "/var/lib/kolla/config_files/src/*" } ], "permissions": [ { "owner": "ironic:ironic", "path": "/var/log/ironic", "recurse": true }, { "owner": "ironic:ironic", "path": "/var/lib/ironic", "recurse": true } ] }[root@undercloud ~]#

Thanks once agan.

-Lokendra

On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard <bshephar@redhat.com> wrote:

...

Looks like something wrong with the dnsmasq command the container is being launched with. What command is it trying to run?

sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta <anyrude10@gmail.com> wrote:

> Hi Brendan, > > Thanks for your response. > > Please find the log below. > > [stack@undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp > > dnsmasq: bad command line options: try --help > dnsmasq: bad command line options: try --help > dnsmasq: bad command line options: try --help > dnsmasq: bad command line options: try --help > dnsmasq: bad command line options: try --help > dnsmasq: bad command line options: try --help > > [stack@undercloud t2u2v2w]$ sudo podman ps --filter > name=ironic_pxe -a > CONTAINER ID IMAGE > COMMAND CREATED > STATUS PORTS NAMES > 02dacbc74cec > undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo > /bin/bash -c BIND... 3 hours ago Exited (1) 3 hours ago (unhealthy) > ironic_pxe_tftp > 1f8ca39fba32 > undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo > kolla_start 3 hours ago Up 3 hours ago (healthy) > ironic_pxe_http > > > Regards > > Anirudh Gupta > > On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard < > bshephar@redhat.com> wrote: > >> Hey Anirudh, >> >> You would need to look at the logs for the ironic_pxe_tftp container >> to see why it's failing. >> >> I assume the tftp container is not Up when you run this command? >> [stack@tripleo-director overcloud_playbooks]$ sudo podman ps >> --filter name=ironic_pxe -a >> CONTAINER ID IMAGE >> COMMAND CREATED STATUS >> PORTS NAMES >> 0170be36e291 >> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo >> kolla_start 12 days ago Up 30 hours ago (healthy) >> ironic_pxe_tftp >> e507f722bdf0 >> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo >> kolla_start 12 days ago Up 30 hours ago (healthy) >> ironic_pxe_http >> >> Then check the logs to see what the error is: >> [stack@tripleo-director overcloud_playbooks]$ sudo podman logs >> ironic_pxe_tftp >> >> >> >> Brendan Shephard >> >> Software Engineer >> >> Red Hat APAC <https://www.redhat.com> >> >> 193 N Quay >> >> Brisbane City QLD 4000 >> @RedHat <https://twitter.com/redhat> Red Hat >> <https://www.linkedin.com/company/red-hat> Red Hat >> <https://www.facebook.com/RedHatInc> >> <https://red.ht/sig> >> <https://redhat.com/summit> >> >> >> On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta <anyrude10@gmail.com> >> wrote: >> >>> Hi Team, >>> >>> I am trying to deploy Openstack Wallaby Undercloud on IPv6, but >>> facing the below error: >>> >>> 2022-06-14 05:01:23.213708 | 52540083-cfa2-3f20-e9dc-00000000286f >>> | TASK | Manage container systemd services and cleanup old systemd >>> healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 >>> 2022-06-14 05:03:22.912816 | 52540083-cfa2-3f20-e9dc-00000000286f >>> | FATAL | Manage container systemd services and cleanup old systemd >>> healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 | >>> undercloud | error={"changed": false, "msg": "Service ironic_pxe_tftp has >>> not started yet"} >>> 2022-06-14 05:03:22.914400 | 52540083-cfa2-3f20-e9dc-00000000286f >>> | TIMING | tripleo_container_manage : Manage container systemd >>> >>> Sample Undercloud.conf is as follows: >>> >>> [DEFAULT] >>> clean_nodes = true >>> cleanup = false >>> container_cli = podman >>> container_healthcheck_disabled = true >>> container_images_file = >>> /home/stack/containers-prepare-parameter.yaml >>> deployment_user = stack >>> enable_ironic = true >>> enable_ironic_inspector = true >>> enable_neutron = true >>> enable_routed_networks = false >>> generate_service_certificate = false >>> ipv6_address_mode = dhcpv6-stateful >>> ipxe_enabled = true >>> local_interface = enp8s0 >>> local_ip = aaaa:aaaa:aaaa::1/64 >>> subnets = ctlplane-subnet >>> undercloud_admin_host = aaaa:aaaa:aaaa::1 >>> undercloud_hostname = undercloud.com >>> undercloud_ntp_servers = 30.30.30.3 >>> undercloud_public_host = aaaa:aaaa:aaaa::1 >>> undercloud_timezone = UTC >>> >>> [ctlplane-subnet] >>> cidr = aaaa:aaaa:aaaa::/64 >>> dhcp_end = aaaa:aaaa:aaaa::f >>> dhcp_start = aaaa:aaaa:aaaa::a >>> gateway = aaaa:aaaa:aaaa::1 >>> inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9 >>> >>> Can someone please help in this regard. >>> >>> Anirudh Gupta >>> >>>

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

Hey, Looks like your server is booting in Legacy BIOS mode but the PXE file Ironic is offering is for UEFI. Did you try booting in UEFI mode instead of Legacy BIOS? Brendan Shephard Software Engineer Red Hat APAC <https://www.redhat.com> 193 N Quay Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit> On Thu, Jun 16, 2022 at 5:05 PM Lokendra Rathour <lokendrarathour@gmail.com> wrote:

Hi Shephard, Thanks, after changing the details as mentioned, Undercloud got installed successfully. Now as a part to test the introspection we added a single node and initiated the introspection on which we are getting errors. IP as per the inspector range is getting allocated, but soon after the IP allocation the introspection ILO gives the below error: [image: image.png] it says, Downloading NBP file.... PXE-E99: Unexpected network error.

Underlcoud.conf:

undercloud_debug = true clean_nodes = true cleanup = false container_cli = podman container_healthcheck_disabled = true container_images_file = /home/stack/containers-prepare-parameter.yaml deployment_user = stack enable_heat = true enable_ironic = true enable_ironic_inspector = true enable_neutron = true generate_service_certificate = false enable_routed_networks = false ipv6_address_mode = dhcpv6-stateful ipxe_enabled = true ironic_default_network_interface = neutron ironic_enabled_network_interfaces = neutron,flat local_interface = enp8s0 local_ip = aaaa:aaaa:aaaa::1/64 subnets = ctlplane-subnet undercloud_admin_host = aaaa:aaaa:aaaa::1 undercloud_public_host = aaaa:aaaa:aaaa::1 undercloud_hostname = undercloud.com undercloud_ntp_servers = 30.30.30.3 undercloud_timezone = UTC [ctlplane-subnet] cidr = aaaa:aaaa:aaaa::/64 dhcp_end = aaaa:aaaa:aaaa::19 dhcp_start = aaaa:aaaa:aaaa::5 gateway = aaaa:aaaa:aaaa::1 inspection_iprange = aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40

the ironic config in the container:

[root@undercloud /]# vi /etc/ironic-inspector/dnsmasq.conf port=0 interface=br-ctlplane

dhcp-range=set:ctlplane-subnet,aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40,64,10m dhcp-option-force=tag:ctlplane-subnet,option:mtu,1500 dhcp-sequential-ip dhcp-match=ipxe,175 dhcp-match=set:efi,option:client-arch,7 dhcp-match=set:efi,option:client-arch,9 dhcp-match=set:efi,option:client-arch,11 # dhcpv6s for Client System Architecture Type (61) dhcp-match=set:efi6,option6:61,0007 dhcp-match=set:efi6,option6:61,0009 dhcp-match=set:efi6,option6:61,0011 dhcp-userclass=set:ipxe6,iPXE # Client is already running iPXE; move to next stage of chainloading dhcp-boot=tag:ipxe,http://[aaaa:aaaa:aaaa::1]:8088/inspector.ipxe dhcp-option=tag:ipxe6,option6:bootfile-url,http:// [aaaa:aaaa:aaaa::1]:8088/inspector.ipxe # Client is PXE booting over EFI without iPXE ROM; send EFI version of iPXE chainloader dhcp-boot=tag:efi,tag:!ipxe,snponly.efi

dhcp-option=tag:efi6,tag:!ipxe6,option6:bootfile-url,tftp://[aaaa:aaaa:aaaa::1]/snponly.efi # Client is running PXE over BIOS; send BIOS version of iPXE chainloader dhcp-boot=undionly.kpxe,localhost.localdomain,aaaa:aaaa:aaaa::1

dhcp-hostsdir=/var/lib/ironic-inspector/dhcp-hostsdir

Please check and help me with the possible error and resolution.

Best Regards, Lokendra

On Thu, Jun 16, 2022 at 5:15 AM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hey,

Looks like that is the problem. The [ ] around the IP address are causing the issue. If I try to run dnsmasq using exactly the output you get, it gives me the same error: [root@tripleo-director ~]# /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=[aaaa:aaaa:aaaa::1] --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot

dnsmasq: bad command line options: try --help

VS without the [ ] I can see it starts up normally.

The settings in your undercloud.conf file look to be correct I believe. So I think there might be a bug here. I don't think we should be saving that value with the square brackets, or we would need to filter them out when we gather the value in that variable.

I raised a bug for it here so that we can dig into this and find what needs fixing: https://bugs.launchpad.net/tripleo/+bug/1978892

In the meantime, if you edit that hieradata value, are you able to get that container started?

Change this: [root@tripleo-director ~]# egrep -r 'tftp_bind_host' /etc/puppet/hieradata/ /etc/puppet/hieradata/service_configs.json: "ironic::pxe::tftp_bind_host": "%{lookup('ctlplane_uri')}",

To this: "ironic::pxe::tftp_bind_host": "aaaa:aaaa:aaaa::1"

Then restart the service: sudo systemctl restart tripleo_ironic_pxe_http.service tripleo_ironic_pxe_tftp.service

Does that get the container running without the error? I did the same in my environment and can see that dnsmasq is running properly like that: [root@tripleo-director ~]# ps -ef | grep aaaa root 71180 52675 0 19:24 pts/4 00:00:00 /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=aaaa:aaaa:aaaa::1 --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Thu, Jun 16, 2022 at 12:12 AM Lokendra Rathour < lokendrarathour@gmail.com> wrote:

...

Hi Shephard, I am getting the local_ip (ipv6) of the undercloud :

[root@undercloud stack]# sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml [aaaa:aaaa:aaaa::1]

is this because of some ipv6 reasons?

On Wed, Jun 15, 2022 at 6:08 PM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hey,

Ok, that command looks fine. What about that variable there? Do you get anything back when you run: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml

Mine returns: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml 192.168.24.115

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour < lokendrarathour@gmail.com> wrote:

...

Hi Shephard,

this is the command from my wallaby: [root@undercloud ~]# sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json { "cap_add": [ "NET_ADMIN", "NET_RAW", "SETUID" ], "command": [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ], "environment": { "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "TRIPLEO_CONFIG_HASH": "9fb3e4e0e35ee35fdf74cfccb16a7543" }, "healthcheck": { "test": "/openstack/healthcheck" }, "image": "undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo", "net": "host", "privileged": false, "restart": "always", "start_order": 90, "volumes": [ "/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro", "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",

"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",

"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",

"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/etc/puppet:/etc/puppet:ro",

"/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro",

"/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro", "/var/lib/ironic:/var/lib/ironic:shared,z", "/var/log/containers/ironic:/var/log/ironic:z", "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z" ] }[root@undercloud ~]#

Comparing both, they look alike. please check once.

On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hi,

Looks like the command was in a different file in Wallaby, can you check: sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json

That one should have the dnsmasq command it's trying to run. For example, here it is from my Wallaby environment: [stack@undercloud-0 ~]$ sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json | jq .command [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ]

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour < lokendrarathour@gmail.com> wrote:

> Hi Shephard, > Here is the o/p of the file: > > [root@undercloud ~]# sudo cat > /var/lib/kolla/config_files/ironic_pxe_tftp.json > { > "config_files": [ > { > "dest": "/", > "merge": true, > "preserve_properties": true, > "source": "/var/lib/kolla/config_files/src/*" > } > ], > "permissions": [ > { > "owner": "ironic:ironic", > "path": "/var/log/ironic", > "recurse": true > }, > { > "owner": "ironic:ironic", > "path": "/var/lib/ironic", > "recurse": true > } > ] > }[root@undercloud ~]# > > > Thanks once agan. > > -Lokendra > > > On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard < > bshephar@redhat.com> wrote: > >> Looks like something wrong with the dnsmasq command the container >> is being launched with. What command is it trying to run? >> >> sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json >> >> Brendan Shephard >> >> Software Engineer >> >> Red Hat APAC <https://www.redhat.com> >> >> 193 N Quay >> >> Brisbane City QLD 4000 >> @RedHat <https://twitter.com/redhat> Red Hat >> <https://www.linkedin.com/company/red-hat> Red Hat >> <https://www.facebook.com/RedHatInc> >> <https://red.ht/sig> >> <https://redhat.com/summit> >> >> >> On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta <anyrude10@gmail.com> >> wrote: >> >>> Hi Brendan, >>> >>> Thanks for your response. >>> >>> Please find the log below. >>> >>> [stack@undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp >>> >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> >>> [stack@undercloud t2u2v2w]$ sudo podman ps --filter >>> name=ironic_pxe -a >>> CONTAINER ID IMAGE >>> COMMAND CREATED >>> STATUS PORTS NAMES >>> 02dacbc74cec >>> undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo >>> /bin/bash -c BIND... 3 hours ago Exited (1) 3 hours ago (unhealthy) >>> ironic_pxe_tftp >>> 1f8ca39fba32 >>> undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo >>> kolla_start 3 hours ago Up 3 hours ago (healthy) >>> ironic_pxe_http >>> >>> >>> Regards >>> >>> Anirudh Gupta >>> >>> On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard < >>> bshephar@redhat.com> wrote: >>> >>>> Hey Anirudh, >>>> >>>> You would need to look at the logs for the ironic_pxe_tftp container >>>> to see why it's failing. >>>> >>>> I assume the tftp container is not Up when you run this command? >>>> [stack@tripleo-director overcloud_playbooks]$ sudo podman ps >>>> --filter name=ironic_pxe -a >>>> CONTAINER ID IMAGE >>>> COMMAND CREATED STATUS >>>> PORTS NAMES >>>> 0170be36e291 >>>> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo >>>> kolla_start 12 days ago Up 30 hours ago (healthy) >>>> ironic_pxe_tftp >>>> e507f722bdf0 >>>> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo >>>> kolla_start 12 days ago Up 30 hours ago (healthy) >>>> ironic_pxe_http >>>> >>>> Then check the logs to see what the error is: >>>> [stack@tripleo-director overcloud_playbooks]$ sudo podman logs >>>> ironic_pxe_tftp >>>> >>>> >>>> >>>> Brendan Shephard >>>> >>>> Software Engineer >>>> >>>> Red Hat APAC <https://www.redhat.com> >>>> >>>> 193 N Quay >>>> >>>> Brisbane City QLD 4000 >>>> @RedHat <https://twitter.com/redhat> Red Hat >>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>> <https://www.facebook.com/RedHatInc> >>>> <https://red.ht/sig> >>>> <https://redhat.com/summit> >>>> >>>> >>>> On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta < >>>> anyrude10@gmail.com> wrote: >>>> >>>>> Hi Team, >>>>> >>>>> I am trying to deploy Openstack Wallaby Undercloud on IPv6, but >>>>> facing the below error: >>>>> >>>>> 2022-06-14 05:01:23.213708 | >>>>> 52540083-cfa2-3f20-e9dc-00000000286f | TASK | Manage container systemd >>>>> services and cleanup old systemd healthchecks for >>>>> /var/lib/tripleo-config/container-startup-config/step_4 >>>>> 2022-06-14 05:03:22.912816 | >>>>> 52540083-cfa2-3f20-e9dc-00000000286f | FATAL | Manage container systemd >>>>> services and cleanup old systemd healthchecks for >>>>> /var/lib/tripleo-config/container-startup-config/step_4 | undercloud | >>>>> error={"changed": false, "msg": "Service ironic_pxe_tftp has not started >>>>> yet"} >>>>> 2022-06-14 05:03:22.914400 | >>>>> 52540083-cfa2-3f20-e9dc-00000000286f | TIMING | tripleo_container_manage : >>>>> Manage container systemd >>>>> >>>>> Sample Undercloud.conf is as follows: >>>>> >>>>> [DEFAULT] >>>>> clean_nodes = true >>>>> cleanup = false >>>>> container_cli = podman >>>>> container_healthcheck_disabled = true >>>>> container_images_file = >>>>> /home/stack/containers-prepare-parameter.yaml >>>>> deployment_user = stack >>>>> enable_ironic = true >>>>> enable_ironic_inspector = true >>>>> enable_neutron = true >>>>> enable_routed_networks = false >>>>> generate_service_certificate = false >>>>> ipv6_address_mode = dhcpv6-stateful >>>>> ipxe_enabled = true >>>>> local_interface = enp8s0 >>>>> local_ip = aaaa:aaaa:aaaa::1/64 >>>>> subnets = ctlplane-subnet >>>>> undercloud_admin_host = aaaa:aaaa:aaaa::1 >>>>> undercloud_hostname = undercloud.com >>>>> undercloud_ntp_servers = 30.30.30.3 >>>>> undercloud_public_host = aaaa:aaaa:aaaa::1 >>>>> undercloud_timezone = UTC >>>>> >>>>> [ctlplane-subnet] >>>>> cidr = aaaa:aaaa:aaaa::/64 >>>>> dhcp_end = aaaa:aaaa:aaaa::f >>>>> dhcp_start = aaaa:aaaa:aaaa::a >>>>> gateway = aaaa:aaaa:aaaa::1 >>>>> inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9 >>>>> >>>>> Can someone please help in this regard. >>>>> >>>>> Anirudh Gupta >>>>> >>>>> > > -- > ~ Lokendra > www.inertiaspeaks.com > www.inertiagroups.com > skype: lokendrarathour > > >

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

Hey, I'm not too sure what that issue could be there then. Probably best if you raise a bug so that you can attach logs and the team that looks after Ironic will be able to help you out. https://bugs.launchpad.net/tripleo If you can attach the logs from /var/log/containers/, we should be able to find some additional clues there. Brendan Shephard Software Engineer Red Hat APAC <https://www.redhat.com> 193 N Quay Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit> On Fri, Jun 17, 2022 at 3:23 AM Lokendra Rathour <lokendrarathour@gmail.com> wrote:

Hi Team , Any support w.r.t the ipv6 based node provisioning for overcloud deployment.

We tried but is getting failed with said error.

-Lokendra

On Thu, 16 Jun 2022, 14:51 Brendan Shephard, <bshephar@redhat.com> wrote:

...

Hey,

Looks like your server is booting in Legacy BIOS mode but the PXE file Ironic is offering is for UEFI. Did you try booting in UEFI mode instead of Legacy BIOS?

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Thu, Jun 16, 2022 at 5:05 PM Lokendra Rathour < lokendrarathour@gmail.com> wrote:

...

Hi Shephard, Thanks, after changing the details as mentioned, Undercloud got installed successfully. Now as a part to test the introspection we added a single node and initiated the introspection on which we are getting errors. IP as per the inspector range is getting allocated, but soon after the IP allocation the introspection ILO gives the below error: [image: image.png] it says, Downloading NBP file.... PXE-E99: Unexpected network error.

Underlcoud.conf:

undercloud_debug = true clean_nodes = true cleanup = false container_cli = podman container_healthcheck_disabled = true container_images_file = /home/stack/containers-prepare-parameter.yaml deployment_user = stack enable_heat = true enable_ironic = true enable_ironic_inspector = true enable_neutron = true generate_service_certificate = false enable_routed_networks = false ipv6_address_mode = dhcpv6-stateful ipxe_enabled = true ironic_default_network_interface = neutron ironic_enabled_network_interfaces = neutron,flat local_interface = enp8s0 local_ip = aaaa:aaaa:aaaa::1/64 subnets = ctlplane-subnet undercloud_admin_host = aaaa:aaaa:aaaa::1 undercloud_public_host = aaaa:aaaa:aaaa::1 undercloud_hostname = undercloud.com undercloud_ntp_servers = 30.30.30.3 undercloud_timezone = UTC [ctlplane-subnet] cidr = aaaa:aaaa:aaaa::/64 dhcp_end = aaaa:aaaa:aaaa::19 dhcp_start = aaaa:aaaa:aaaa::5 gateway = aaaa:aaaa:aaaa::1 inspection_iprange = aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40

the ironic config in the container:

[root@undercloud /]# vi /etc/ironic-inspector/dnsmasq.conf port=0 interface=br-ctlplane

dhcp-range=set:ctlplane-subnet,aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40,64,10m dhcp-option-force=tag:ctlplane-subnet,option:mtu,1500 dhcp-sequential-ip dhcp-match=ipxe,175 dhcp-match=set:efi,option:client-arch,7 dhcp-match=set:efi,option:client-arch,9 dhcp-match=set:efi,option:client-arch,11 # dhcpv6s for Client System Architecture Type (61) dhcp-match=set:efi6,option6:61,0007 dhcp-match=set:efi6,option6:61,0009 dhcp-match=set:efi6,option6:61,0011 dhcp-userclass=set:ipxe6,iPXE # Client is already running iPXE; move to next stage of chainloading dhcp-boot=tag:ipxe,http://[aaaa:aaaa:aaaa::1]:8088/inspector.ipxe dhcp-option=tag:ipxe6,option6:bootfile-url,http:// [aaaa:aaaa:aaaa::1]:8088/inspector.ipxe # Client is PXE booting over EFI without iPXE ROM; send EFI version of iPXE chainloader dhcp-boot=tag:efi,tag:!ipxe,snponly.efi

dhcp-option=tag:efi6,tag:!ipxe6,option6:bootfile-url,tftp://[aaaa:aaaa:aaaa::1]/snponly.efi # Client is running PXE over BIOS; send BIOS version of iPXE chainloader dhcp-boot=undionly.kpxe,localhost.localdomain,aaaa:aaaa:aaaa::1

dhcp-hostsdir=/var/lib/ironic-inspector/dhcp-hostsdir

Please check and help me with the possible error and resolution.

Best Regards, Lokendra

On Thu, Jun 16, 2022 at 5:15 AM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hey,

Looks like that is the problem. The [ ] around the IP address are causing the issue. If I try to run dnsmasq using exactly the output you get, it gives me the same error: [root@tripleo-director ~]# /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=[aaaa:aaaa:aaaa::1] --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot

dnsmasq: bad command line options: try --help

VS without the [ ] I can see it starts up normally.

The settings in your undercloud.conf file look to be correct I believe. So I think there might be a bug here. I don't think we should be saving that value with the square brackets, or we would need to filter them out when we gather the value in that variable.

I raised a bug for it here so that we can dig into this and find what needs fixing: https://bugs.launchpad.net/tripleo/+bug/1978892

In the meantime, if you edit that hieradata value, are you able to get that container started?

Change this: [root@tripleo-director ~]# egrep -r 'tftp_bind_host' /etc/puppet/hieradata/ /etc/puppet/hieradata/service_configs.json: "ironic::pxe::tftp_bind_host": "%{lookup('ctlplane_uri')}",

To this: "ironic::pxe::tftp_bind_host": "aaaa:aaaa:aaaa::1"

Then restart the service: sudo systemctl restart tripleo_ironic_pxe_http.service tripleo_ironic_pxe_tftp.service

Does that get the container running without the error? I did the same in my environment and can see that dnsmasq is running properly like that: [root@tripleo-director ~]# ps -ef | grep aaaa root 71180 52675 0 19:24 pts/4 00:00:00 /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=aaaa:aaaa:aaaa::1 --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Thu, Jun 16, 2022 at 12:12 AM Lokendra Rathour < lokendrarathour@gmail.com> wrote:

...

Hi Shephard, I am getting the local_ip (ipv6) of the undercloud :

[root@undercloud stack]# sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml [aaaa:aaaa:aaaa::1]

is this because of some ipv6 reasons?

On Wed, Jun 15, 2022 at 6:08 PM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hey,

Ok, that command looks fine. What about that variable there? Do you get anything back when you run: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml

Mine returns: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml 192.168.24.115

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour < lokendrarathour@gmail.com> wrote:

> Hi Shephard, > > this is the command from my wallaby: > [root@undercloud ~]# sudo cat > /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json > { > "cap_add": [ > "NET_ADMIN", > "NET_RAW", > "SETUID" > ], > "command": [ > "/bin/bash", > "-c", > "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c > /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground > --log-facility=/var/log/ironic/dnsmasq.log --user=root > --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp > --tftp-root=/var/lib/ironic/tftpboot" > ], > "environment": { > "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", > "TRIPLEO_CONFIG_HASH": "9fb3e4e0e35ee35fdf74cfccb16a7543" > }, > "healthcheck": { > "test": "/openstack/healthcheck" > }, > "image": > "undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo", > "net": "host", > "privileged": false, > "restart": "always", > "start_order": 90, > "volumes": [ > "/etc/hosts:/etc/hosts:ro", > "/etc/localtime:/etc/localtime:ro", > "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro", > > "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro", > > "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro", > > "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", > "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", > "/dev/log:/dev/log", > "/etc/puppet:/etc/puppet:ro", > > "/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro", > > "/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro", > "/var/lib/ironic:/var/lib/ironic:shared,z", > "/var/log/containers/ironic:/var/log/ironic:z", > "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z" > ] > }[root@undercloud ~]# > > Comparing both, they look alike. > please check once. > > On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard < > bshephar@redhat.com> wrote: > >> Hi, >> >> Looks like the command was in a different file in Wallaby, can you >> check: >> sudo cat >> /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json >> >> That one should have the dnsmasq command it's trying to run. For >> example, here it is from my Wallaby environment: >> [stack@undercloud-0 ~]$ sudo cat >> /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json >> | jq .command >> [ >> "/bin/bash", >> "-c", >> "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c >> /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground >> --log-facility=/var/log/ironic/dnsmasq.log --user=root >> --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp >> --tftp-root=/var/lib/ironic/tftpboot" >> ] >> >> >> >> Brendan Shephard >> >> Software Engineer >> >> Red Hat APAC <https://www.redhat.com> >> >> 193 N Quay >> >> Brisbane City QLD 4000 >> @RedHat <https://twitter.com/redhat> Red Hat >> <https://www.linkedin.com/company/red-hat> Red Hat >> <https://www.facebook.com/RedHatInc> >> <https://red.ht/sig> >> <https://redhat.com/summit> >> >> >> On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour < >> lokendrarathour@gmail.com> wrote: >> >>> Hi Shephard, >>> Here is the o/p of the file: >>> >>> [root@undercloud ~]# sudo cat >>> /var/lib/kolla/config_files/ironic_pxe_tftp.json >>> { >>> "config_files": [ >>> { >>> "dest": "/", >>> "merge": true, >>> "preserve_properties": true, >>> "source": "/var/lib/kolla/config_files/src/*" >>> } >>> ], >>> "permissions": [ >>> { >>> "owner": "ironic:ironic", >>> "path": "/var/log/ironic", >>> "recurse": true >>> }, >>> { >>> "owner": "ironic:ironic", >>> "path": "/var/lib/ironic", >>> "recurse": true >>> } >>> ] >>> }[root@undercloud ~]# >>> >>> >>> Thanks once agan. >>> >>> -Lokendra >>> >>> >>> On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard < >>> bshephar@redhat.com> wrote: >>> >>>> Looks like something wrong with the dnsmasq command the container >>>> is being launched with. What command is it trying to run? >>>> >>>> sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json >>>> >>>> Brendan Shephard >>>> >>>> Software Engineer >>>> >>>> Red Hat APAC <https://www.redhat.com> >>>> >>>> 193 N Quay >>>> >>>> Brisbane City QLD 4000 >>>> @RedHat <https://twitter.com/redhat> Red Hat >>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>> <https://www.facebook.com/RedHatInc> >>>> <https://red.ht/sig> >>>> <https://redhat.com/summit> >>>> >>>> >>>> On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta < >>>> anyrude10@gmail.com> wrote: >>>> >>>>> Hi Brendan, >>>>> >>>>> Thanks for your response. >>>>> >>>>> Please find the log below. >>>>> >>>>> [stack@undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp >>>>> >>>>> dnsmasq: bad command line options: try --help >>>>> dnsmasq: bad command line options: try --help >>>>> dnsmasq: bad command line options: try --help >>>>> dnsmasq: bad command line options: try --help >>>>> dnsmasq: bad command line options: try --help >>>>> dnsmasq: bad command line options: try --help >>>>> >>>>> [stack@undercloud t2u2v2w]$ sudo podman ps --filter >>>>> name=ironic_pxe -a >>>>> CONTAINER ID IMAGE >>>>> COMMAND CREATED >>>>> STATUS PORTS NAMES >>>>> 02dacbc74cec >>>>> undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo >>>>> /bin/bash -c BIND... 3 hours ago Exited (1) 3 hours ago (unhealthy) >>>>> ironic_pxe_tftp >>>>> 1f8ca39fba32 >>>>> undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo >>>>> kolla_start 3 hours ago Up 3 hours ago (healthy) >>>>> ironic_pxe_http >>>>> >>>>> >>>>> Regards >>>>> >>>>> Anirudh Gupta >>>>> >>>>> On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard < >>>>> bshephar@redhat.com> wrote: >>>>> >>>>>> Hey Anirudh, >>>>>> >>>>>> You would need to look at the logs for the ironic_pxe_tftp container >>>>>> to see why it's failing. >>>>>> >>>>>> I assume the tftp container is not Up when you run this command? >>>>>> [stack@tripleo-director overcloud_playbooks]$ sudo podman ps >>>>>> --filter name=ironic_pxe -a >>>>>> CONTAINER ID IMAGE >>>>>> COMMAND CREATED STATUS >>>>>> PORTS NAMES >>>>>> 0170be36e291 >>>>>> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo >>>>>> kolla_start 12 days ago Up 30 hours ago (healthy) >>>>>> ironic_pxe_tftp >>>>>> e507f722bdf0 >>>>>> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo >>>>>> kolla_start 12 days ago Up 30 hours ago (healthy) >>>>>> ironic_pxe_http >>>>>> >>>>>> Then check the logs to see what the error is: >>>>>> [stack@tripleo-director overcloud_playbooks]$ sudo podman logs >>>>>> ironic_pxe_tftp >>>>>> >>>>>> >>>>>> >>>>>> Brendan Shephard >>>>>> >>>>>> Software Engineer >>>>>> >>>>>> Red Hat APAC <https://www.redhat.com> >>>>>> >>>>>> 193 N Quay >>>>>> >>>>>> Brisbane City QLD 4000 >>>>>> @RedHat <https://twitter.com/redhat> Red Hat >>>>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>>>> <https://www.facebook.com/RedHatInc> >>>>>> <https://red.ht/sig> >>>>>> <https://redhat.com/summit> >>>>>> >>>>>> >>>>>> On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta < >>>>>> anyrude10@gmail.com> wrote: >>>>>> >>>>>>> Hi Team, >>>>>>> >>>>>>> I am trying to deploy Openstack Wallaby Undercloud on IPv6, >>>>>>> but facing the below error: >>>>>>> >>>>>>> 2022-06-14 05:01:23.213708 | >>>>>>> 52540083-cfa2-3f20-e9dc-00000000286f | TASK | Manage container systemd >>>>>>> services and cleanup old systemd healthchecks for >>>>>>> /var/lib/tripleo-config/container-startup-config/step_4 >>>>>>> 2022-06-14 05:03:22.912816 | >>>>>>> 52540083-cfa2-3f20-e9dc-00000000286f | FATAL | Manage container systemd >>>>>>> services and cleanup old systemd healthchecks for >>>>>>> /var/lib/tripleo-config/container-startup-config/step_4 | undercloud | >>>>>>> error={"changed": false, "msg": "Service ironic_pxe_tftp has not started >>>>>>> yet"} >>>>>>> 2022-06-14 05:03:22.914400 | >>>>>>> 52540083-cfa2-3f20-e9dc-00000000286f | TIMING | tripleo_container_manage : >>>>>>> Manage container systemd >>>>>>> >>>>>>> Sample Undercloud.conf is as follows: >>>>>>> >>>>>>> [DEFAULT] >>>>>>> clean_nodes = true >>>>>>> cleanup = false >>>>>>> container_cli = podman >>>>>>> container_healthcheck_disabled = true >>>>>>> container_images_file = >>>>>>> /home/stack/containers-prepare-parameter.yaml >>>>>>> deployment_user = stack >>>>>>> enable_ironic = true >>>>>>> enable_ironic_inspector = true >>>>>>> enable_neutron = true >>>>>>> enable_routed_networks = false >>>>>>> generate_service_certificate = false >>>>>>> ipv6_address_mode = dhcpv6-stateful >>>>>>> ipxe_enabled = true >>>>>>> local_interface = enp8s0 >>>>>>> local_ip = aaaa:aaaa:aaaa::1/64 >>>>>>> subnets = ctlplane-subnet >>>>>>> undercloud_admin_host = aaaa:aaaa:aaaa::1 >>>>>>> undercloud_hostname = undercloud.com >>>>>>> undercloud_ntp_servers = 30.30.30.3 >>>>>>> undercloud_public_host = aaaa:aaaa:aaaa::1 >>>>>>> undercloud_timezone = UTC >>>>>>> >>>>>>> [ctlplane-subnet] >>>>>>> cidr = aaaa:aaaa:aaaa::/64 >>>>>>> dhcp_end = aaaa:aaaa:aaaa::f >>>>>>> dhcp_start = aaaa:aaaa:aaaa::a >>>>>>> gateway = aaaa:aaaa:aaaa::1 >>>>>>> inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9 >>>>>>> >>>>>>> Can someone please help in this regard. >>>>>>> >>>>>>> Anirudh Gupta >>>>>>> >>>>>>> >>> >>> -- >>> ~ Lokendra >>> www.inertiaspeaks.com >>> www.inertiagroups.com >>> skype: lokendrarathour >>> >>> >>> > > -- > ~ Lokendra > www.inertiaspeaks.com > www.inertiagroups.com > skype: lokendrarathour > > >

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

Hi Herald, please find the response in line: * Verify that the TFTP server is running? *[Loke]* *TFTP Container is running but is in unhealthy state:* *[root@undercloud ironic]# podman ps -a | grep tftpb3fb26bcf190 undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo /bin/bash -c BIND... 4 hours ago Up 4 hours ago (unhealthy) ironic_pxe_tftp[root@undercloud ironic]#* * Check the log file - /var/log/containers/ironic/dnsmasq.log *[Loke]: ERROR* *[root@undercloud ironic]# tail -f dnsmasq.logJun 18 04:41:10 dnsmasq[11]: started, version 2.79 DNS disabledJun 18 04:41:10 dnsmasq[11]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotifyJun 18 04:41:10 dnsmasq-tftp[11]: TFTP root is /var/lib/ironic/tftpbootJun 18 08:09:21 dnsmasq-tftp[11]: failed sending /var/lib/ironic/tftpboot/snponly.efi to aaaa:aaaa:aaaa::4Jun 18 08:09:28 dnsmasq-tftp[11]: failed sending /var/lib/ironic/tftpboot/snponly.efi to aaaa:aaaa:aaaa::4Jun 18 08:09:35 dnsmasq-tftp[11]: failed sending /var/lib/ironic/tftpboot/snponly.efi to aaaa:aaaa:aaaa::4* * Try to capture the TFTP traffic on br-ctlplane to a PCAP file to see *[Loke]: * *attached the same as aaaa.pcap* *Also do note that there is a similar bug reported:* *https://bugs.launchpad.net/tripleo/+bug/1964397 <https://bugs.launchpad.net/tripleo/+bug/1964397>* *the error in the bug is similar to what I have.* On Fri, Jun 17, 2022 at 5:53 PM Harald Jensas <hjensas@redhat.com> wrote:

Hi,

The error in the screenshot, the booting node is not able to dowload the network boot program from the TFTP server on the undercloud.

Can you: * Verify that the TFTP server is running? * Check the log file - /var/log/containers/ironic/dnsmasq.log * Try to capture the TFTP traffic on br-ctlplane to a PCAP file to see if we can use that to dignose the problem. * download that file manually? For example: curl -O tftp://<undercloud_ip>/snponly.efi ?

Best Regards Harald

On 6/16/22 09:05, Lokendra Rathour wrote:

...

Hi Shephard, Thanks, after changing the details as mentioned, Undercloud got installed successfully. Now as a part to test the introspection we added a single node and initiated the introspection on which we are getting errors. IP as per the inspector range is getting allocated, but soon after the IP allocation the introspection ILO gives the below error: image.png it says, Downloading NBP file.... PXE-E99: Unexpected network error.

Underlcoud.conf:

undercloud_debug = true clean_nodes = true cleanup = false container_cli = podman container_healthcheck_disabled = true container_images_file = /home/stack/containers-prepare-parameter.yaml deployment_user = stack enable_heat = true enable_ironic = true enable_ironic_inspector = true enable_neutron = true generate_service_certificate = false enable_routed_networks = false ipv6_address_mode = dhcpv6-stateful ipxe_enabled = true ironic_default_network_interface = neutron ironic_enabled_network_interfaces = neutron,flat local_interface = enp8s0 local_ip = aaaa:aaaa:aaaa::1/64 subnets = ctlplane-subnet undercloud_admin_host = aaaa:aaaa:aaaa::1 undercloud_public_host = aaaa:aaaa:aaaa::1 undercloud_hostname = undercloud.com <http://undercloud.com> undercloud_ntp_servers = 30.30.30.3 undercloud_timezone = UTC [ctlplane-subnet] cidr = aaaa:aaaa:aaaa::/64 dhcp_end = aaaa:aaaa:aaaa::19 dhcp_start = aaaa:aaaa:aaaa::5 gateway = aaaa:aaaa:aaaa::1 inspection_iprange = aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40

the ironic config in the container:

[root@undercloud /]# vi /etc/ironic-inspector/dnsmasq.conf port=0 interface=br-ctlplane

dhcp-range=set:ctlplane-subnet,aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40,64,10m

...

dhcp-option-force=tag:ctlplane-subnet,option:mtu,1500 dhcp-sequential-ip dhcp-match=ipxe,175 dhcp-match=set:efi,option:client-arch,7 dhcp-match=set:efi,option:client-arch,9 dhcp-match=set:efi,option:client-arch,11 # dhcpv6s for Client System Architecture Type (61) dhcp-match=set:efi6,option6:61,0007 dhcp-match=set:efi6,option6:61,0009 dhcp-match=set:efi6,option6:61,0011 dhcp-userclass=set:ipxe6,iPXE # Client is already running iPXE; move to next stage of chainloading dhcp-boot=tag:ipxe,http://[aaaa:aaaa:aaaa::1]:8088/inspector.ipxe dhcp-option=tag:ipxe6,option6:bootfile-url,http:// [aaaa:aaaa:aaaa::1]:8088/inspector.ipxe # Client is PXE booting over EFI without iPXE ROM; send EFI version of iPXE chainloader dhcp-boot=tag:efi,tag:!ipxe,snponly.efi

dhcp-option=tag:efi6,tag:!ipxe6,option6:bootfile-url,tftp://[aaaa:aaaa:aaaa::1]/snponly.efi

...

# Client is running PXE over BIOS; send BIOS version of iPXE chainloader dhcp-boot=undionly.kpxe,localhost.localdomain,aaaa:aaaa:aaaa::1

dhcp-hostsdir=/var/lib/ironic-inspector/dhcp-hostsdir

Please check and help me with the possible error and resolution.

Best Regards, Lokendra

On Thu, Jun 16, 2022 at 5:15 AM Brendan Shephard <bshephar@redhat.com <mailto:bshephar@redhat.com>> wrote:

Hey,

Looks like that is the problem. The [ ] around the IP address are causing the issue. If I try to run dnsmasq using exactly the output you get, it gives me the same error: [root@tripleo-director ~]# /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=[aaaa:aaaa:aaaa::1] --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot

dnsmasq: bad command line options: try --help

VS without the [ ] I can see it starts up normally.

The settings in your undercloud.conf file look to be correct I believe. So I think there might be a bug here. I don't think we should be saving that value with the square brackets, or we would need to filter them out when we gather the value in that variable.

I raised a bug for it here so that we can dig into this and find what needs fixing: https://bugs.launchpad.net/tripleo/+bug/1978892 <https://bugs.launchpad.net/tripleo/+bug/1978892>

In the meantime, if you edit that hieradata value, are you able to get that container started?

Change this: [root@tripleo-director ~]# egrep -r 'tftp_bind_host' /etc/puppet/hieradata/ /etc/puppet/hieradata/service_configs.json: "ironic::pxe::tftp_bind_host": "%{lookup('ctlplane_uri')}",

To this: "ironic::pxe::tftp_bind_host": "aaaa:aaaa:aaaa::1"

Then restart the service: sudo systemctl restart tripleo_ironic_pxe_http.service tripleo_ironic_pxe_tftp.service

Does that get the container running without the error? I did the same in my environment and can see that dnsmasq is running properly like that: [root@tripleo-director ~]# ps -ef | grep aaaa root 71180 52675 0 19:24 pts/4 00:00:00 /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=aaaa:aaaa:aaaa::1 --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000

@RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Thu, Jun 16, 2022 at 12:12 AM Lokendra Rathour <lokendrarathour@gmail.com <mailto:lokendrarathour@gmail.com>> wrote:

Hi Shephard, I am getting the local_ip (ipv6) of the undercloud :

[root@undercloud stack]# sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml [aaaa:aaaa:aaaa::1]

is this because of some ipv6 reasons?

On Wed, Jun 15, 2022 at 6:08 PM Brendan Shephard <bshephar@redhat.com <mailto:bshephar@redhat.com>> wrote:

Hey,

Ok, that command looks fine. What about that variable there? Do you get anything back when you run: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml

Mine returns: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml 192.168.24.115

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000

@RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour <lokendrarathour@gmail.com <mailto:lokendrarathour@gmail.com>> wrote:

Hi Shephard,

this is the command from my wallaby: [root@undercloud ~]# sudo cat

/var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json

...

{ "cap_add": [ "NET_ADMIN", "NET_RAW", "SETUID" ], "command": [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp

--tftp-root=/var/lib/ironic/tftpboot"

...

], "environment": { "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "TRIPLEO_CONFIG_HASH": "9fb3e4e0e35ee35fdf74cfccb16a7543" }, "healthcheck": { "test": "/openstack/healthcheck" }, "image":

"undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo",

...

"net": "host", "privileged": false, "restart": "always", "start_order": 90, "volumes": [ "/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro",

"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",

...

"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",

...

"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",

...

"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro",

...

"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/etc/puppet:/etc/puppet:ro",

"/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro",

...

"/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro",

...

"/var/lib/ironic:/var/lib/ironic:shared,z", "/var/log/containers/ironic:/var/log/ironic:z",

"/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z"

...

] }[root@undercloud ~]#

Comparing both, they look alike. please check once.

On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard <bshephar@redhat.com <mailto:bshephar@redhat.com>>

wrote:

...

Hi,

Looks like the command was in a different file in Wallaby, can you check: sudo cat

/var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json

...

That one should have the dnsmasq command it's trying to run. For example, here it is from my Wallaby environment: [stack@undercloud-0 ~]$ sudo cat

/var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json

...

| jq .command [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ]

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000

@RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour <lokendrarathour@gmail.com <mailto:lokendrarathour@gmail.com>> wrote:

Hi Shephard, Here is the o/p of the file:

[root@undercloud ~]# sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json { "config_files": [ { "dest": "/", "merge": true, "preserve_properties": true, "source":

"/var/lib/kolla/config_files/src/*"

...

} ], "permissions": [ { "owner": "ironic:ironic", "path": "/var/log/ironic", "recurse": true }, { "owner": "ironic:ironic", "path": "/var/lib/ironic", "recurse": true } ] }[root@undercloud ~]#

Thanks once agan.

-Lokendra

On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard <bshephar@redhat.com <mailto:bshephar@redhat.com>> wrote:

Looks like something wrong with the dnsmasq command the container is being launched with. What command is it trying to run?

sudo cat

/var/lib/kolla/config_files/ironic_pxe_tftp.json

...

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000

@RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta <anyrude10@gmail.com <mailto:anyrude10@gmail.com>> wrote:

Hi Brendan,

Thanks for your response.

Please find the log below.

[stack@undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp

dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help

[stack@undercloud t2u2v2w]$ sudo podman ps --filter name=ironic_pxe -a CONTAINER ID IMAGE

COMMAND CREATED STATUS PORTS NAMES 02dacbc74cec

undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo /bin/bash -c BIND... 3 hours ago Exited (1) 3 hours ago (unhealthy) ironic_pxe_tftp

...

1f8ca39fba32

undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo kolla_start 3 hours ago Up 3 hours ago (healthy) ironic_pxe_http

...

Regards

Anirudh Gupta

On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard <bshephar@redhat.com <mailto:bshephar@redhat.com>> wrote:

Hey Anirudh,

You would need to look at the logs for the ironic_pxe_tftp container to see why it's failing.

I assume the tftp container is not Up when you run this command? [stack@tripleo-director overcloud_playbooks]$ sudo podman ps --filter name=ironic_pxe -a CONTAINER ID IMAGE

COMMAND CREATED STATUS PORTS NAMES 0170be36e291

registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo

...

<

http://registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironi...

...

kolla_start 12 days ago Up 30 hours ago (healthy) ironic_pxe_tftp e507f722bdf0

registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo

...

<

http://registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironi...

...

kolla_start 12 days ago Up 30 hours ago (healthy) ironic_pxe_http

Then check the logs to see what the error is: [stack@tripleo-director overcloud_playbooks]$ sudo podman logs ironic_pxe_tftp

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com

193 N Quay

Brisbane City QLD 4000

@RedHat <https://twitter.com/redhat> Red Hat <

https://www.linkedin.com/company/red-hat>

...

Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta <anyrude10@gmail.com <mailto:anyrude10@gmail.com>> wrote:

Hi Team,

I am trying to deploy Openstack Wallaby Undercloud on IPv6, but facing the below error:

2022-06-14 05:01:23.213708 |

52540083-cfa2-3f20-e9dc-00000000286f

...

| TASK | Manage container systemd services and cleanup old systemd healthchecks for

/var/lib/tripleo-config/container-startup-config/step_4

...

2022-06-14 05:03:22.912816 |

52540083-cfa2-3f20-e9dc-00000000286f

...

| FATAL | Manage container systemd services and cleanup old systemd healthchecks for

/var/lib/tripleo-config/container-startup-config/step_4

...

| undercloud | error={"changed": false, "msg": "Service ironic_pxe_tftp has not started yet"} 2022-06-14 05:03:22.914400 |

52540083-cfa2-3f20-e9dc-00000000286f

...

| TIMING | tripleo_container_manage : Manage container systemd

Sample Undercloud.conf is as follows:

[DEFAULT] clean_nodes = true cleanup = false container_cli = podman container_healthcheck_disabled = true container_images_file =

/home/stack/containers-prepare-parameter.yaml

...

deployment_user = stack enable_ironic = true enable_ironic_inspector = true enable_neutron = true enable_routed_networks = false generate_service_certificate =

false

...

ipv6_address_mode =

dhcpv6-stateful

...

ipxe_enabled = true local_interface = enp8s0 local_ip = aaaa:aaaa:aaaa::1/64 subnets = ctlplane-subnet undercloud_admin_host = aaaa:aaaa:aaaa::1 undercloud_hostname = undercloud.com <http://undercloud.com> undercloud_ntp_servers =

30.30.30.3

...

undercloud_public_host = aaaa:aaaa:aaaa::1 undercloud_timezone = UTC

[ctlplane-subnet] cidr = aaaa:aaaa:aaaa::/64 dhcp_end = aaaa:aaaa:aaaa::f dhcp_start = aaaa:aaaa:aaaa::a gateway = aaaa:aaaa:aaaa::1 inspection_iprange =

aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9

...

Can someone please help in this regard.

Anirudh Gupta

-- ~ Lokendra www.inertiaspeaks.com <

http://www.inertiaspeaks.com>

...

www.inertiagroups.com <

http://www.inertiagroups.com>

...

skype: lokendrarathour

-- ~ Lokendra www.inertiaspeaks.com <http://www.inertiaspeaks.com> www.inertiagroups.com <http://www.inertiagroups.com> skype: lokendrarathour

-- ~ Lokendra www.inertiaspeaks.com <http://www.inertiaspeaks.com> www.inertiagroups.com <http://www.inertiagroups.com> skype: lokendrarathour

-- ~ Lokendra www.inertiaspeaks.com <http://www.inertiaspeaks.com> www.inertiagroups.com <http://www.inertiagroups.com> skype: lokendrarathour

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

Hi Team, So, I am still not able to provision overcloud baremetal nodes using IPv6. Based on the previous discussion i tried changing some settings in terms of IPV6 Notation, but no luck. Has anyone of us tried ipv6 based overcloud deployment on Wallaby? Please suggest some way forward. Thanks once again for your support. -Lokendra On Sat, Jun 18, 2022 at 10:54 PM Dan Sneddon <dsneddon@redhat.com> wrote:

It is arguable that dnsmasq should accept the IP address in brackets, since that is a standard way of representing IPv6 addresses. It gets confusing, see these ServerFault answers for more detail:

https://serverfault.com/questions/444554/what-does-mean-as-an-ip-address-bra...

http://serverfault.com/questions/1026466/ddg#1026469

We originally had a problem where IPv6 addresses needed to be in brackets to support URL representation. In TripleO at one time we even had two versions of hiera values, with and without the brackets. I don’t know if that still exists or not.

I wouldn’t hold my breath about getting such a change into dnsmasq. It is notoriously difficult to get patches accepted into dnsmasq, especially where UI is concerned.

-Dan Sneddon

On Thu, Jun 16, 2022 at 5:32 AM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hey,

Looks like that is the problem. The [ ] around the IP address are causing the issue. If I try to run dnsmasq using exactly the output you get, it gives me the same error: [root@tripleo-director ~]# /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=[aaaa:aaaa:aaaa::1] --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot

dnsmasq: bad command line options: try --help

VS without the [ ] I can see it starts up normally.

The settings in your undercloud.conf file look to be correct I believe. So I think there might be a bug here. I don't think we should be saving that value with the square brackets, or we would need to filter them out when we gather the value in that variable.

I raised a bug for it here so that we can dig into this and find what needs fixing: https://bugs.launchpad.net/tripleo/+bug/1978892

In the meantime, if you edit that hieradata value, are you able to get that container started?

Change this: [root@tripleo-director ~]# egrep -r 'tftp_bind_host' /etc/puppet/hieradata/ /etc/puppet/hieradata/service_configs.json: "ironic::pxe::tftp_bind_host": "%{lookup('ctlplane_uri')}",

To this: "ironic::pxe::tftp_bind_host": "aaaa:aaaa:aaaa::1"

Then restart the service: sudo systemctl restart tripleo_ironic_pxe_http.service tripleo_ironic_pxe_tftp.service

Does that get the container running without the error? I did the same in my environment and can see that dnsmasq is running properly like that: [root@tripleo-director ~]# ps -ef | grep aaaa root 71180 52675 0 19:24 pts/4 00:00:00 /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=aaaa:aaaa:aaaa::1 --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>

<https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>

Brisbane City QLD <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Thu, Jun 16, 2022 at 12:12 AM Lokendra Rathour < lokendrarathour@gmail.com> wrote:

...

Hi Shephard, I am getting the local_ip (ipv6) of the undercloud :

[root@undercloud stack]# sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml [aaaa:aaaa:aaaa::1]

is this because of some ipv6 reasons?

On Wed, Jun 15, 2022 at 6:08 PM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hey,

Ok, that command looks fine. What about that variable there? Do you get anything back when you run: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml

Mine returns: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml 192.168.24.115

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>

Brisbane City QLD <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> 4000 @RedHat <https://twitter.com/redhat> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> Red Hat <https://www.linkedin.com/company/red-hat> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour < lokendrarathour@gmail.com> wrote:

...

Hi Shephard,

this is the command from my wallaby: [root@undercloud ~]# sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json { "cap_add": [ "NET_ADMIN", "NET_RAW", "SETUID" ], "command": [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ], "environment": { "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "TRIPLEO_CONFIG_HASH": "9fb3e4e0e35ee35fdf74cfccb16a7543" }, "healthcheck": { "test": "/openstack/healthcheck" }, "image": "undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo", "net": "host", "privileged": false, "restart": "always", "start_order": 90, "volumes": [ "/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro", "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",

"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",

"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",

"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/etc/puppet:/etc/puppet:ro",

"/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro",

"/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro", "/var/lib/ironic:/var/lib/ironic:shared,z", "/var/log/containers/ironic:/var/log/ironic:z", "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z" ] }[root@undercloud ~]#

Comparing both, they look alike. please check once.

On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hi,

Looks like the command was in a different file in Wallaby, can you check: sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json

That one should have the dnsmasq command it's trying to run. For example, here it is from my Wallaby environment: [stack@undercloud-0 ~]$ sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json | jq .command [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ]

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>

Brisbane City QLD <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc>

<https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> <https://red.ht/sig> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour < lokendrarathour@gmail.com> wrote:

> Hi Shephard, > Here is the o/p of the file: > > [root@undercloud ~]# sudo cat > /var/lib/kolla/config_files/ironic_pxe_tftp.json > { > "config_files": [ > { > "dest": "/", > "merge": true, > "preserve_properties": true, > "source": "/var/lib/kolla/config_files/src/*" > } > ], > "permissions": [ > { > "owner": "ironic:ironic", > "path": "/var/log/ironic", > "recurse": true > }, > { > "owner": "ironic:ironic", > "path": "/var/lib/ironic", > "recurse": true > } > ] > }[root@undercloud ~]# > > > Thanks once agan. > > -Lokendra > > > On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard < > bshephar@redhat.com> wrote: > >> Looks like something wrong with the dnsmasq command the container >> is being launched with. What command is it trying to run? >> >> sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json >> >> Brendan Shephard >> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> >> >> Software Engineer >> >> Red Hat APAC <https://www.redhat.com> >> >> 193 N Quay >> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> >> >> Brisbane City QLD >> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> >> 4000 >> @RedHat <https://twitter.com/redhat> Red Hat >> <https://www.linkedin.com/company/red-hat> Red Hat >> <https://www.facebook.com/RedHatInc> >> <https://red.ht/sig> >> <https://redhat.com/summit> >> >> >> On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta <anyrude10@gmail.com> >> wrote: >> >>> Hi Brendan, >>> >>> Thanks for your response. >>> >>> Please find the log below. >>> >>> [stack@undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp >>> >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> >>> [stack@undercloud t2u2v2w]$ sudo podman ps --filter >>> name=ironic_pxe -a >>> CONTAINER ID IMAGE >>> COMMAND CREATED >>> STATUS PORTS NAMES >>> 02dacbc74cec >>> undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo >>> /bin/bash -c BIND... 3 hours ago Exited (1) 3 hours ago (unhealthy) >>> ironic_pxe_tftp >>> 1f8ca39fba32 >>> undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo >>> kolla_start 3 hours ago Up 3 hours ago (healthy) >>> ironic_pxe_http >>> >>> >>> Regards >>> >>> Anirudh Gupta >>> >>> On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard < >>> bshephar@redhat.com> wrote: >>> >>>> Hey Anirudh, >>>> >>>> You would need to look at the logs for the ironic_pxe_tftp container >>>> to see why it's failing. >>>> >>>> I assume the tftp container is not Up when you run this command? >>>> [stack@tripleo-director overcloud_playbooks]$ sudo podman ps >>>> --filter name=ironic_pxe -a >>>> CONTAINER ID IMAGE >>>> COMMAND CREATED STATUS >>>> PORTS NAMES >>>> 0170be36e291 >>>> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo >>>> kolla_start 12 days ago Up 30 hours ago (healthy) >>>> ironic_pxe_tftp >>>> e507f722bdf0 >>>> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo >>>> kolla_start 12 days ago Up 30 hours ago (healthy) >>>> ironic_pxe_http >>>> >>>> Then check the logs to see what the error is: >>>> [stack@tripleo-director overcloud_playbooks]$ sudo podman logs >>>> ironic_pxe_tftp >>>> >>>> >>>> >>>> >>>> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> >>>> >>>> >>>> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> >>>> Brendan Shephard >>>> >>>> Software Engineer >>>> >>>> >>>> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> Red >>>> Hat APAC <https://www.redhat.com> >>>> >>>> 193 N Quay >>>> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> >>>> >>>> Brisbane City QLD >>>> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> >>>> 4000 >>>> @RedHat <https://twitter.com/redhat> Red Hat >>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>> <https://www.facebook.com/RedHatInc> >>>> <https://red.ht/sig> >>>> <https://redhat.com/summit> >>>> >>>> >>>> On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta < >>>> anyrude10@gmail.com> wrote: >>>> >>>>> Hi Team, >>>>> >>>>> I am trying to deploy Openstack Wallaby Undercloud on IPv6, but >>>>> facing the below error: >>>>> >>>>> 2022-06-14 05:01:23.213708 | >>>>> 52540083-cfa2-3f20-e9dc-00000000286f | TASK | Manage container systemd >>>>> services and cleanup old systemd healthchecks for >>>>> /var/lib/tripleo-config/container-startup-config/step_4 >>>>> 2022-06-14 05:03:22.912816 | >>>>> 52540083-cfa2-3f20-e9dc-00000000286f | FATAL | Manage container systemd >>>>> services and cleanup old systemd healthchecks for >>>>> /var/lib/tripleo-config/container-startup-config/step_4 | undercloud | >>>>> error={"changed": false, "msg": "Service ironic_pxe_tftp has not started >>>>> yet"} >>>>> 2022-06-14 05:03:22.914400 | >>>>> 52540083-cfa2-3f20-e9dc-00000000286f | TIMING | tripleo_container_manage : >>>>> Manage container systemd >>>>> >>>>> Sample Undercloud.conf is as follows: >>>>> >>>>> [DEFAULT] >>>>> clean_nodes = true >>>>> cleanup = false >>>>> container_cli = podman >>>>> container_healthcheck_disabled = true >>>>> container_images_file = >>>>> /home/stack/containers-prepare-parameter.yaml >>>>> deployment_user = stack >>>>> enable_ironic = true >>>>> enable_ironic_inspector = true >>>>> enable_neutron = true >>>>> enable_routed_networks = false >>>>> generate_service_certificate = false >>>>> ipv6_address_mode = dhcpv6-stateful >>>>> ipxe_enabled = true >>>>> local_interface = enp8s0 >>>>> local_ip = aaaa:aaaa:aaaa::1/64 >>>>> subnets = ctlplane-subnet >>>>> undercloud_admin_host = aaaa:aaaa:aaaa::1 >>>>> undercloud_hostname = undercloud.com >>>>> undercloud_ntp_servers = 30.30.30.3 >>>>> undercloud_public_host = aaaa:aaaa:aaaa::1 >>>>> undercloud_timezone = UTC >>>>> >>>>> [ctlplane-subnet] >>>>> cidr = aaaa:aaaa:aaaa::/64 >>>>> dhcp_end = aaaa:aaaa:aaaa::f >>>>> dhcp_start = aaaa:aaaa:aaaa::a >>>>> gateway = aaaa:aaaa:aaaa::1 >>>>> inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9 >>>>> >>>>> Can someone please help in this regard. >>>>> >>>>> Anirudh Gupta >>>>> >>>>> > > -- > ~ Lokendra > www.inertiaspeaks.com > www.inertiagroups.com > skype: lokendrarathour > > >

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

-- Dan Sneddon | Senior Principal Software Engineer dsneddon@redhat.com | redhat.com/cloud dsneddon:irc | @dxs:twitter

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

Hey, It looks like my reply email was too big and has been blocked. The problem is the square brackets in the IPv6 address being returned from the hiera command. Can you try this to workaround the problem: Change this: [root@tripleo-director ~]# egrep -r 'tftp_bind_host' /etc/puppet/hieradata/ /etc/puppet/hieradata/service_configs.json: "ironic::pxe::tftp_bind_host": "%{lookup('ctlplane_uri')}", To this: "ironic::pxe::tftp_bind_host": "aaaa:aaaa:aaaa::1" Then restart the service: sudo systemctl restart tripleo_ironic_pxe_http.service tripleo_ironic_pxe_tftp.service I raised a bug for this for you: https://bugs.launchpad.net/tripleo/+bug/1978892 Brendan Shephard Software Engineer Red Hat APAC <https://www.redhat.com> 193 N Quay Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit> On Thu, Jun 16, 2022 at 12:12 AM Lokendra Rathour <lokendrarathour@gmail.com> wrote:

Hi Shephard, I am getting the local_ip (ipv6) of the undercloud :

[root@undercloud stack]# sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml [aaaa:aaaa:aaaa::1]

is this because of some ipv6 reasons?

On Wed, Jun 15, 2022 at 6:08 PM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hey,

Ok, that command looks fine. What about that variable there? Do you get anything back when you run: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml

Mine returns: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml 192.168.24.115

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour < lokendrarathour@gmail.com> wrote:

...

Hi Shephard,

this is the command from my wallaby: [root@undercloud ~]# sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json { "cap_add": [ "NET_ADMIN", "NET_RAW", "SETUID" ], "command": [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ], "environment": { "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "TRIPLEO_CONFIG_HASH": "9fb3e4e0e35ee35fdf74cfccb16a7543" }, "healthcheck": { "test": "/openstack/healthcheck" }, "image": "undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo", "net": "host", "privileged": false, "restart": "always", "start_order": 90, "volumes": [ "/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro", "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",

"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",

"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",

"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/etc/puppet:/etc/puppet:ro",

"/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro",

"/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro", "/var/lib/ironic:/var/lib/ironic:shared,z", "/var/log/containers/ironic:/var/log/ironic:z", "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z" ] }[root@undercloud ~]#

Comparing both, they look alike. please check once.

On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard <bshephar@redhat.com> wrote:

...

Hi,

Looks like the command was in a different file in Wallaby, can you check: sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json

That one should have the dnsmasq command it's trying to run. For example, here it is from my Wallaby environment: [stack@undercloud-0 ~]$ sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json | jq .command [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ]

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour < lokendrarathour@gmail.com> wrote:

...

Hi Shephard, Here is the o/p of the file:

[root@undercloud ~]# sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json { "config_files": [ { "dest": "/", "merge": true, "preserve_properties": true, "source": "/var/lib/kolla/config_files/src/*" } ], "permissions": [ { "owner": "ironic:ironic", "path": "/var/log/ironic", "recurse": true }, { "owner": "ironic:ironic", "path": "/var/lib/ironic", "recurse": true } ] }[root@undercloud ~]#

Thanks once agan.

-Lokendra

On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard <bshephar@redhat.com> wrote:

...

Looks like something wrong with the dnsmasq command the container is being launched with. What command is it trying to run?

sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json

Brendan Shephard

Software Engineer

Red Hat APAC <https://www.redhat.com>

193 N Quay

Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta <anyrude10@gmail.com> wrote:

> Hi Brendan, > > Thanks for your response. > > Please find the log below. > > [stack@undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp > > dnsmasq: bad command line options: try --help > dnsmasq: bad command line options: try --help > dnsmasq: bad command line options: try --help > dnsmasq: bad command line options: try --help > dnsmasq: bad command line options: try --help > dnsmasq: bad command line options: try --help > > [stack@undercloud t2u2v2w]$ sudo podman ps --filter > name=ironic_pxe -a > CONTAINER ID IMAGE > COMMAND CREATED > STATUS PORTS NAMES > 02dacbc74cec > undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo > /bin/bash -c BIND... 3 hours ago Exited (1) 3 hours ago (unhealthy) > ironic_pxe_tftp > 1f8ca39fba32 > undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo > kolla_start 3 hours ago Up 3 hours ago (healthy) > ironic_pxe_http > > > Regards > > Anirudh Gupta > > On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard < > bshephar@redhat.com> wrote: > >> Hey Anirudh, >> >> You would need to look at the logs for the ironic_pxe_tftp container >> to see why it's failing. >> >> I assume the tftp container is not Up when you run this command? >> [stack@tripleo-director overcloud_playbooks]$ sudo podman ps >> --filter name=ironic_pxe -a >> CONTAINER ID IMAGE >> COMMAND CREATED STATUS >> PORTS NAMES >> 0170be36e291 >> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo >> kolla_start 12 days ago Up 30 hours ago (healthy) >> ironic_pxe_tftp >> e507f722bdf0 >> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo >> kolla_start 12 days ago Up 30 hours ago (healthy) >> ironic_pxe_http >> >> Then check the logs to see what the error is: >> [stack@tripleo-director overcloud_playbooks]$ sudo podman logs >> ironic_pxe_tftp >> >> >> >> Brendan Shephard >> >> Software Engineer >> >> Red Hat APAC <https://www.redhat.com> >> >> 193 N Quay >> >> Brisbane City QLD 4000 >> @RedHat <https://twitter.com/redhat> Red Hat >> <https://www.linkedin.com/company/red-hat> Red Hat >> <https://www.facebook.com/RedHatInc> >> <https://red.ht/sig> >> <https://redhat.com/summit> >> >> >> On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta <anyrude10@gmail.com> >> wrote: >> >>> Hi Team, >>> >>> I am trying to deploy Openstack Wallaby Undercloud on IPv6, but >>> facing the below error: >>> >>> 2022-06-14 05:01:23.213708 | 52540083-cfa2-3f20-e9dc-00000000286f >>> | TASK | Manage container systemd services and cleanup old systemd >>> healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 >>> 2022-06-14 05:03:22.912816 | 52540083-cfa2-3f20-e9dc-00000000286f >>> | FATAL | Manage container systemd services and cleanup old systemd >>> healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 | >>> undercloud | error={"changed": false, "msg": "Service ironic_pxe_tftp has >>> not started yet"} >>> 2022-06-14 05:03:22.914400 | 52540083-cfa2-3f20-e9dc-00000000286f >>> | TIMING | tripleo_container_manage : Manage container systemd >>> >>> Sample Undercloud.conf is as follows: >>> >>> [DEFAULT] >>> clean_nodes = true >>> cleanup = false >>> container_cli = podman >>> container_healthcheck_disabled = true >>> container_images_file = >>> /home/stack/containers-prepare-parameter.yaml >>> deployment_user = stack >>> enable_ironic = true >>> enable_ironic_inspector = true >>> enable_neutron = true >>> enable_routed_networks = false >>> generate_service_certificate = false >>> ipv6_address_mode = dhcpv6-stateful >>> ipxe_enabled = true >>> local_interface = enp8s0 >>> local_ip = aaaa:aaaa:aaaa::1/64 >>> subnets = ctlplane-subnet >>> undercloud_admin_host = aaaa:aaaa:aaaa::1 >>> undercloud_hostname = undercloud.com >>> undercloud_ntp_servers = 30.30.30.3 >>> undercloud_public_host = aaaa:aaaa:aaaa::1 >>> undercloud_timezone = UTC >>> >>> [ctlplane-subnet] >>> cidr = aaaa:aaaa:aaaa::/64 >>> dhcp_end = aaaa:aaaa:aaaa::f >>> dhcp_start = aaaa:aaaa:aaaa::a >>> gateway = aaaa:aaaa:aaaa::1 >>> inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9 >>> >>> Can someone please help in this regard. >>> >>> Anirudh Gupta >>> >>>

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

Isn’t there a standard library for parsing/encoding IP addresses per language used - this is a well known type problem, and if there isn’t a standard type, then the code is likely to have assumptions that are based on IPv4 and broken by IPv6 (e.g. that NAT is a good idea ;-) )

On 16 Jun 2022, at 07:49, Brendan Shephard <bshephar@redhat.com> wrote:

Hey,

It looks like my reply email was too big and has been blocked. The problem is the square brackets in the IPv6 address being returned from the hiera command. Can you try this to workaround the problem: Change this: [root@tripleo-director ~]# egrep -r 'tftp_bind_host' /etc/puppet/hieradata/ /etc/puppet/hieradata/service_configs.json: "ironic::pxe::tftp_bind_host": "%{lookup('ctlplane_uri')}",

To this: "ironic::pxe::tftp_bind_host": "aaaa:aaaa:aaaa::1"

Then restart the service: sudo systemctl restart tripleo_ironic_pxe_http.service tripleo_ironic_pxe_tftp.service

I raised a bug for this for you: https://bugs.launchpad.net/tripleo/+bug/1978892 <https://bugs.launchpad.net/tripleo/+bug/1978892>

Brendan Shephard Software Engineer Red Hat APAC <https://www.redhat.com/> 193 N Quay Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Thu, Jun 16, 2022 at 12:12 AM Lokendra Rathour <lokendrarathour@gmail.com <mailto:lokendrarathour@gmail.com>> wrote: Hi Shephard, I am getting the local_ip (ipv6) of the undercloud :

[root@undercloud stack]# sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml [aaaa:aaaa:aaaa::1]

is this because of some ipv6 reasons?

On Wed, Jun 15, 2022 at 6:08 PM Brendan Shephard <bshephar@redhat.com <mailto:bshephar@redhat.com>> wrote: Hey,

Ok, that command looks fine. What about that variable there? Do you get anything back when you run: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml

Mine returns: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml 192.168.24.115

Brendan Shephard Software Engineer Red Hat APAC <https://www.redhat.com/> 193 N Quay Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour <lokendrarathour@gmail.com <mailto:lokendrarathour@gmail.com>> wrote: Hi Shephard,

this is the command from my wallaby: [root@undercloud ~]# sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json { "cap_add": [ "NET_ADMIN", "NET_RAW", "SETUID" ], "command": [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ], "environment": { "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "TRIPLEO_CONFIG_HASH": "9fb3e4e0e35ee35fdf74cfccb16a7543" }, "healthcheck": { "test": "/openstack/healthcheck" }, "image": "undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo", "net": "host", "privileged": false, "restart": "always", "start_order": 90, "volumes": [ "/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro", "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro", "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro", "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro", "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/etc/puppet:/etc/puppet:ro", "/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro", "/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro", "/var/lib/ironic:/var/lib/ironic:shared,z", "/var/log/containers/ironic:/var/log/ironic:z", "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z" ] }[root@undercloud ~]#

Comparing both, they look alike. please check once.

On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard <bshephar@redhat.com <mailto:bshephar@redhat.com>> wrote: Hi,

Looks like the command was in a different file in Wallaby, can you check: sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json

That one should have the dnsmasq command it's trying to run. For example, here it is from my Wallaby environment: [stack@undercloud-0 ~]$ sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json | jq .command [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ]

Brendan Shephard Software Engineer Red Hat APAC <https://www.redhat.com/> 193 N Quay Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour <lokendrarathour@gmail.com <mailto:lokendrarathour@gmail.com>> wrote: Hi Shephard, Here is the o/p of the file:

[root@undercloud ~]# sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json { "config_files": [ { "dest": "/", "merge": true, "preserve_properties": true, "source": "/var/lib/kolla/config_files/src/*" } ], "permissions": [ { "owner": "ironic:ironic", "path": "/var/log/ironic", "recurse": true }, { "owner": "ironic:ironic", "path": "/var/lib/ironic", "recurse": true } ] }[root@undercloud ~]#

Thanks once agan.

-Lokendra

On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard <bshephar@redhat.com <mailto:bshephar@redhat.com>> wrote: Looks like something wrong with the dnsmasq command the container is being launched with. What command is it trying to run?

sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json

Brendan Shephard Software Engineer Red Hat APAC <https://www.redhat.com/> 193 N Quay Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta <anyrude10@gmail.com <mailto:anyrude10@gmail.com>> wrote: Hi Brendan,

Thanks for your response.

Please find the log below.

[stack@undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp

dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help

[stack@undercloud t2u2v2w]$ sudo podman ps --filter name=ironic_pxe -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 02dacbc74cec undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo /bin/bash -c BIND... 3 hours ago Exited (1) 3 hours ago (unhealthy) ironic_pxe_tftp 1f8ca39fba32 undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo kolla_start 3 hours ago Up 3 hours ago (healthy) ironic_pxe_http

Regards Anirudh Gupta

On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard <bshephar@redhat.com <mailto:bshephar@redhat.com>> wrote: Hey Anirudh,

You would need to look at the logs for the ironic_pxe_tftp container to see why it's failing.

I assume the tftp container is not Up when you run this command? [stack@tripleo-director overcloud_playbooks]$ sudo podman ps --filter name=ironic_pxe -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0170be36e291 registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo <http://registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo> kolla_start 12 days ago Up 30 hours ago (healthy) ironic_pxe_tftp e507f722bdf0 registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo <http://registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo> kolla_start 12 days ago Up 30 hours ago (healthy) ironic_pxe_http

Then check the logs to see what the error is: [stack@tripleo-director overcloud_playbooks]$ sudo podman logs ironic_pxe_tftp

Brendan Shephard Software Engineer Red Hat APAC <https://www.redhat.com/> 193 N Quay Brisbane City QLD 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>

On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta <anyrude10@gmail.com <mailto:anyrude10@gmail.com>> wrote: Hi Team,

I am trying to deploy Openstack Wallaby Undercloud on IPv6, but facing the below error:

2022-06-14 05:01:23.213708 | 52540083-cfa2-3f20-e9dc-00000000286f | TASK | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 2022-06-14 05:03:22.912816 | 52540083-cfa2-3f20-e9dc-00000000286f | FATAL | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 | undercloud | error={"changed": false, "msg": "Service ironic_pxe_tftp has not started yet"} 2022-06-14 05:03:22.914400 | 52540083-cfa2-3f20-e9dc-00000000286f | TIMING | tripleo_container_manage : Manage container systemd

Sample Undercloud.conf is as follows:

[DEFAULT] clean_nodes = true cleanup = false container_cli = podman container_healthcheck_disabled = true container_images_file = /home/stack/containers-prepare-parameter.yaml deployment_user = stack enable_ironic = true enable_ironic_inspector = true enable_neutron = true enable_routed_networks = false generate_service_certificate = false ipv6_address_mode = dhcpv6-stateful ipxe_enabled = true local_interface = enp8s0 local_ip = aaaa:aaaa:aaaa::1/64 subnets = ctlplane-subnet undercloud_admin_host = aaaa:aaaa:aaaa::1 undercloud_hostname = undercloud.com <http://undercloud.com/> undercloud_ntp_servers = 30.30.30.3 undercloud_public_host = aaaa:aaaa:aaaa::1 undercloud_timezone = UTC

[ctlplane-subnet] cidr = aaaa:aaaa:aaaa::/64 dhcp_end = aaaa:aaaa:aaaa::f dhcp_start = aaaa:aaaa:aaaa::a gateway = aaaa:aaaa:aaaa::1 inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9

Can someone please help in this regard.

Anirudh Gupta

-- ~ Lokendra www.inertiaspeaks.com <http://www.inertiaspeaks.com/> www.inertiagroups.com <http://www.inertiagroups.com/> skype: lokendrarathour

-- ~ Lokendra www.inertiaspeaks.com <http://www.inertiaspeaks.com/> www.inertiagroups.com <http://www.inertiagroups.com/> skype: lokendrarathour

-- ~ Lokendra www.inertiaspeaks.com <http://www.inertiaspeaks.com/> www.inertiagroups.com <http://www.inertiagroups.com/> skype: lokendrarathour