Hey,

Looks like your server is booting in Legacy BIOS mode but the PXE file Ironic is offering is for UEFI. Did you try booting in UEFI mode instead of Legacy BIOS?

Brendan Shephard

Software Engineer

Red Hat APAC

193 N Quay

Brisbane City QLD 4000


On Thu, Jun 16, 2022 at 5:05 PM Lokendra Rathour <lokendrarathour@gmail.com> wrote:
Hi Shephard,
Thanks, after changing the details as mentioned, Undercloud got installed successfully.
Now as a part to test the introspection we added a single node and initiated the introspection on which we are getting errors.
IP as per the inspector range is getting allocated, but soon after the IP allocation the introspection ILO gives the below error:
image.png
it says, Downloading NBP file....
PXE-E99: Unexpected network error.

Underlcoud.conf:
undercloud_debug = true
clean_nodes = true
cleanup = false
container_cli = podman
container_healthcheck_disabled = true
container_images_file = /home/stack/containers-prepare-parameter.yaml
deployment_user = stack
enable_heat = true
enable_ironic = true
enable_ironic_inspector = true
enable_neutron = true
generate_service_certificate = false
enable_routed_networks = false
ipv6_address_mode = dhcpv6-stateful
ipxe_enabled = true
ironic_default_network_interface = neutron
ironic_enabled_network_interfaces = neutron,flat
local_interface = enp8s0
local_ip = aaaa:aaaa:aaaa::1/64
subnets = ctlplane-subnet
undercloud_admin_host = aaaa:aaaa:aaaa::1
undercloud_public_host = aaaa:aaaa:aaaa::1
undercloud_hostname = undercloud.com
undercloud_ntp_servers = 30.30.30.3
undercloud_timezone = UTC
[ctlplane-subnet]
cidr = aaaa:aaaa:aaaa::/64
dhcp_end = aaaa:aaaa:aaaa::19
dhcp_start = aaaa:aaaa:aaaa::5
gateway = aaaa:aaaa:aaaa::1
inspection_iprange = aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40

the ironic config in the container:

[root@undercloud /]# vi /etc/ironic-inspector/dnsmasq.conf
port=0
interface=br-ctlplane

dhcp-range=set:ctlplane-subnet,aaaa:aaaa:aaaa::20,aaaa:aaaa:aaaa::40,64,10m
dhcp-option-force=tag:ctlplane-subnet,option:mtu,1500
dhcp-sequential-ip
dhcp-match=ipxe,175
dhcp-match=set:efi,option:client-arch,7
dhcp-match=set:efi,option:client-arch,9
dhcp-match=set:efi,option:client-arch,11
# dhcpv6s for Client System Architecture Type (61)
dhcp-match=set:efi6,option6:61,0007
dhcp-match=set:efi6,option6:61,0009
dhcp-match=set:efi6,option6:61,0011
dhcp-userclass=set:ipxe6,iPXE
# Client is already running iPXE; move to next stage of chainloading
dhcp-boot=tag:ipxe,http://[aaaa:aaaa:aaaa::1]:8088/inspector.ipxe
dhcp-option=tag:ipxe6,option6:bootfile-url,http://[aaaa:aaaa:aaaa::1]:8088/inspector.ipxe
# Client is PXE booting over EFI without iPXE ROM; send EFI version of iPXE chainloader
dhcp-boot=tag:efi,tag:!ipxe,snponly.efi
dhcp-option=tag:efi6,tag:!ipxe6,option6:bootfile-url,tftp://[aaaa:aaaa:aaaa::1]/snponly.efi
# Client is running PXE over BIOS; send BIOS version of iPXE chainloader
dhcp-boot=undionly.kpxe,localhost.localdomain,aaaa:aaaa:aaaa::1

dhcp-hostsdir=/var/lib/ironic-inspector/dhcp-hostsdir


Please check and help me with the possible error and resolution. 

Best Regards,
Lokendra




On Thu, Jun 16, 2022 at 5:15 AM Brendan Shephard <bshephar@redhat.com> wrote:
Hey,

Looks like that is the problem. The [ ] around the IP address are causing the issue. If I try to run dnsmasq using exactly the output you get, it gives me the same error:
[root@tripleo-director ~]# /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=[aaaa:aaaa:aaaa::1] --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot

dnsmasq: bad command line options: try --help

VS without the [ ] I can see it starts up normally.

The settings in your undercloud.conf file look to be correct I believe. So I think there might be a bug here. I don't think we should be saving that value with the square brackets, or we would need to filter them out when we gather the value in that variable.

I raised a bug for it here so that we can dig into this and find what needs fixing:
https://bugs.launchpad.net/tripleo/+bug/1978892

In the meantime, if you edit that hieradata value, are you able to get that container started?

Change this:
[root@tripleo-director ~]# egrep -r 'tftp_bind_host' /etc/puppet/hieradata/
/etc/puppet/hieradata/service_configs.json:    "ironic::pxe::tftp_bind_host": "%{lookup('ctlplane_uri')}",

To this:
 "ironic::pxe::tftp_bind_host": "aaaa:aaaa:aaaa::1"

Then restart the service:
sudo systemctl restart tripleo_ironic_pxe_http.service tripleo_ironic_pxe_tftp.service

Does that get the container running without the error? I did the same in my environment and can see that dnsmasq is running properly like that:
[root@tripleo-director ~]# ps -ef | grep aaaa
root       71180   52675  0 19:24 pts/4    00:00:00 /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=aaaa:aaaa:aaaa::1 --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot

Brendan Shephard

Software Engineer

Red Hat APAC

193 N Quay

Brisbane City QLD 4000


On Thu, Jun 16, 2022 at 12:12 AM Lokendra Rathour <lokendrarathour@gmail.com> wrote:
Hi Shephard,
I am getting the local_ip (ipv6) of the undercloud :

[root@undercloud stack]# sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml
[aaaa:aaaa:aaaa::1]

is this because of some ipv6 reasons?


On Wed, Jun 15, 2022 at 6:08 PM Brendan Shephard <bshephar@redhat.com> wrote:
Hey,

Ok, that command looks fine. What about that variable there? Do you get anything back when you run:
sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml

Mine returns:
sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml
192.168.24.115

Brendan Shephard

Software Engineer

Red Hat APAC

193 N Quay

Brisbane City QLD 4000


On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour <lokendrarathour@gmail.com> wrote:
Hi Shephard,

this is the command from my wallaby:
[root@undercloud ~]# sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json
{
  "cap_add": [
    "NET_ADMIN",
    "NET_RAW",
    "SETUID"
  ],
  "command": [
    "/bin/bash",
    "-c",
    "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot"
  ],
  "environment": {
    "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS",
    "TRIPLEO_CONFIG_HASH": "9fb3e4e0e35ee35fdf74cfccb16a7543"
  },
  "healthcheck": {
    "test": "/openstack/healthcheck"
  },
  "image": "undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo",
  "net": "host",
  "privileged": false,
  "restart": "always",
  "start_order": 90,
  "volumes": [
    "/etc/hosts:/etc/hosts:ro",
    "/etc/localtime:/etc/localtime:ro",
    "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
    "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",
    "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
    "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro",
    "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro",
    "/dev/log:/dev/log",
    "/etc/puppet:/etc/puppet:ro",
    "/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro",
    "/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro",
    "/var/lib/ironic:/var/lib/ironic:shared,z",
    "/var/log/containers/ironic:/var/log/ironic:z",
    "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z"
  ]
}[root@undercloud ~]#

Comparing both, they look alike.
please check once.

On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard <bshephar@redhat.com> wrote:
Hi,

Looks like the command was in a different file in Wallaby, can you check:
sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json

That one should have the dnsmasq command it's trying to run. For example, here it is from my Wallaby environment:
[stack@undercloud-0 ~]$ sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json | jq .command
[
  "/bin/bash",
  "-c",
  "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot"
]



Brendan Shephard

Software Engineer

Red Hat APAC

193 N Quay

Brisbane City QLD 4000


On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour <lokendrarathour@gmail.com> wrote:
Hi Shephard,
Here is the o/p of the file:

[root@undercloud ~]# sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json
{
  "config_files": [
    {
      "dest": "/",
      "merge": true,
      "preserve_properties": true,
      "source": "/var/lib/kolla/config_files/src/*"
    }
  ],
  "permissions": [
    {
      "owner": "ironic:ironic",
      "path": "/var/log/ironic",
      "recurse": true
    },
    {
      "owner": "ironic:ironic",
      "path": "/var/lib/ironic",
      "recurse": true
    }
  ]
}[root@undercloud ~]#


Thanks once agan. 

-Lokendra


On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard <bshephar@redhat.com> wrote:
Looks like something wrong with the dnsmasq command the container is being launched with. What command is it trying to run?

sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json

Brendan Shephard

Software Engineer

Red Hat APAC

193 N Quay

Brisbane City QLD 4000


On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta <anyrude10@gmail.com> wrote:
Hi Brendan,

Thanks for your response.

Please find the log below.

[stack@undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp

dnsmasq: bad command line options: try --help
dnsmasq: bad command line options: try --help
dnsmasq: bad command line options: try --help
dnsmasq: bad command line options: try --help
dnsmasq: bad command line options: try --help
dnsmasq: bad command line options: try --help

[stack@undercloud t2u2v2w]$  sudo podman ps --filter name=ironic_pxe -a
CONTAINER ID  IMAGE                                                                                     COMMAND               CREATED      STATUS                              PORTS       NAMES
02dacbc74cec  undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo  /bin/bash -c BIND...  3 hours ago  Exited (1) 3 hours ago (unhealthy)              ironic_pxe_tftp
1f8ca39fba32  undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo  kolla_start           3 hours ago  Up 3 hours ago (healthy)                        ironic_pxe_http


Regards

Anirudh Gupta


On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard <bshephar@redhat.com> wrote:
Hey Anirudh,

You would need to look at the logs for the ironic_pxe_tftp container to see why it's failing.

I assume the tftp container is not Up when you run this command?
[stack@tripleo-director overcloud_playbooks]$ sudo podman ps --filter name=ironic_pxe -a
CONTAINER ID  IMAGE                                                                                       COMMAND      CREATED      STATUS                     PORTS       NAMES
0170be36e291  registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo  kolla_start  12 days ago  Up 30 hours ago (healthy)              ironic_pxe_tftp
e507f722bdf0  registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo  kolla_start  12 days ago  Up 30 hours ago (healthy)              ironic_pxe_http

Then check the logs to see what the error is:
[stack@tripleo-director overcloud_playbooks]$ sudo podman logs ironic_pxe_tftp



Brendan Shephard

Software Engineer

Red Hat APAC

193 N Quay

Brisbane City QLD 4000


On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta <anyrude10@gmail.com> wrote:
Hi Team,

I am trying to deploy Openstack Wallaby Undercloud on IPv6, but facing the below error:

2022-06-14 05:01:23.213708 | 52540083-cfa2-3f20-e9dc-00000000286f | TASK | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4
2022-06-14 05:03:22.912816 | 52540083-cfa2-3f20-e9dc-00000000286f | FATAL | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 | undercloud | error={"changed": false, "msg": "Service ironic_pxe_tftp has not started yet"}
2022-06-14 05:03:22.914400 | 52540083-cfa2-3f20-e9dc-00000000286f | TIMING | tripleo_container_manage : Manage container systemd

Sample Undercloud.conf is as follows:

[DEFAULT]
clean_nodes = true
cleanup = false
container_cli = podman
container_healthcheck_disabled = true
container_images_file = /home/stack/containers-prepare-parameter.yaml
deployment_user = stack
enable_ironic = true
enable_ironic_inspector = true
enable_neutron = true
enable_routed_networks = false
generate_service_certificate = false
ipv6_address_mode = dhcpv6-stateful
ipxe_enabled = true
local_interface = enp8s0
local_ip = aaaa:aaaa:aaaa::1/64
subnets = ctlplane-subnet
undercloud_admin_host = aaaa:aaaa:aaaa::1
undercloud_hostname = undercloud.com
undercloud_ntp_servers = 30.30.30.3
undercloud_public_host = aaaa:aaaa:aaaa::1
undercloud_timezone = UTC

[ctlplane-subnet]
cidr = aaaa:aaaa:aaaa::/64
dhcp_end = aaaa:aaaa:aaaa::f
dhcp_start = aaaa:aaaa:aaaa::a
gateway = aaaa:aaaa:aaaa::1
inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9

Can someone please help in this regard.

Anirudh Gupta



--
~ Lokendra
skype: lokendrarathour




--
~ Lokendra
skype: lokendrarathour




--
~ Lokendra
skype: lokendrarathour




--
~ Lokendra
skype: lokendrarathour