Hi Team, So, I am still not able to provision overcloud baremetal nodes using IPv6. Based on the previous discussion i tried changing some settings in terms of IPV6 Notation, but no luck. Has anyone of us tried ipv6 based overcloud deployment on Wallaby? Please suggest some way forward. Thanks once again for your support. -Lokendra On Sat, Jun 18, 2022 at 10:54 PM Dan Sneddon <dsneddon@redhat.com> wrote:
It is arguable that dnsmasq should accept the IP address in brackets, since that is a standard way of representing IPv6 addresses. It gets confusing, see these ServerFault answers for more detail:
https://serverfault.com/questions/444554/what-does-mean-as-an-ip-address-bra...
http://serverfault.com/questions/1026466/ddg#1026469
We originally had a problem where IPv6 addresses needed to be in brackets to support URL representation. In TripleO at one time we even had two versions of hiera values, with and without the brackets. I don’t know if that still exists or not.
I wouldn’t hold my breath about getting such a change into dnsmasq. It is notoriously difficult to get patches accepted into dnsmasq, especially where UI is concerned.
-Dan Sneddon
On Thu, Jun 16, 2022 at 5:32 AM Brendan Shephard <bshephar@redhat.com> wrote:
Hey,
Looks like that is the problem. The [ ] around the IP address are causing the issue. If I try to run dnsmasq using exactly the output you get, it gives me the same error: [root@tripleo-director ~]# /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=[aaaa:aaaa:aaaa::1] --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot
dnsmasq: bad command line options: try --help
VS without the [ ] I can see it starts up normally.
The settings in your undercloud.conf file look to be correct I believe. So I think there might be a bug here. I don't think we should be saving that value with the square brackets, or we would need to filter them out when we gather the value in that variable.
I raised a bug for it here so that we can dig into this and find what needs fixing: https://bugs.launchpad.net/tripleo/+bug/1978892
In the meantime, if you edit that hieradata value, are you able to get that container started?
Change this: [root@tripleo-director ~]# egrep -r 'tftp_bind_host' /etc/puppet/hieradata/ /etc/puppet/hieradata/service_configs.json: "ironic::pxe::tftp_bind_host": "%{lookup('ctlplane_uri')}",
To this: "ironic::pxe::tftp_bind_host": "aaaa:aaaa:aaaa::1"
Then restart the service: sudo systemctl restart tripleo_ironic_pxe_http.service tripleo_ironic_pxe_tftp.service
Does that get the container running without the error? I did the same in my environment and can see that dnsmasq is running properly like that: [root@tripleo-director ~]# ps -ef | grep aaaa root 71180 52675 0 19:24 pts/4 00:00:00 /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=aaaa:aaaa:aaaa::1 --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot
Brendan Shephard
Software Engineer
Red Hat APAC <https://www.redhat.com>
<https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g>
Brisbane City QLD <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://redhat.com/summit>
On Thu, Jun 16, 2022 at 12:12 AM Lokendra Rathour < lokendrarathour@gmail.com> wrote:
Hi Shephard, I am getting the local_ip (ipv6) of the undercloud :
[root@undercloud stack]# sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml [aaaa:aaaa:aaaa::1]
is this because of some ipv6 reasons?
On Wed, Jun 15, 2022 at 6:08 PM Brendan Shephard <bshephar@redhat.com> wrote:
Hey,
Ok, that command looks fine. What about that variable there? Do you get anything back when you run: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml
Mine returns: sudo hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml 192.168.24.115
Brendan Shephard
Software Engineer
Red Hat APAC <https://www.redhat.com>
Brisbane City QLD <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> 4000 @RedHat <https://twitter.com/redhat> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> Red Hat <https://www.linkedin.com/company/red-hat> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> <https://redhat.com/summit>
On Wed, Jun 15, 2022 at 8:20 PM Lokendra Rathour < lokendrarathour@gmail.com> wrote:
Hi Shephard,
this is the command from my wallaby: [root@undercloud ~]# sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json { "cap_add": [ "NET_ADMIN", "NET_RAW", "SETUID" ], "command": [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ], "environment": { "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "TRIPLEO_CONFIG_HASH": "9fb3e4e0e35ee35fdf74cfccb16a7543" }, "healthcheck": { "test": "/openstack/healthcheck" }, "image": "undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo", "net": "host", "privileged": false, "restart": "always", "start_order": 90, "volumes": [ "/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro", "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",
"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/etc/puppet:/etc/puppet:ro",
"/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro",
"/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro", "/var/lib/ironic:/var/lib/ironic:shared,z", "/var/log/containers/ironic:/var/log/ironic:z", "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z" ] }[root@undercloud ~]#
Comparing both, they look alike. please check once.
On Wed, Jun 15, 2022 at 3:30 PM Brendan Shephard <bshephar@redhat.com> wrote:
Hi,
Looks like the command was in a different file in Wallaby, can you check: sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json
That one should have the dnsmasq command it's trying to run. For example, here it is from my Wallaby environment: [stack@undercloud-0 ~]$ sudo cat /var/lib/tripleo-config/container-startup-config/step_4/ironic_pxe_tftp.json | jq .command [ "/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/dnsmasq --keep-in-foreground --log-facility=/var/log/ironic/dnsmasq.log --user=root --conf-file=/dev/null --listen-address=$BIND_HOST --port=0 --enable-tftp --tftp-root=/var/lib/ironic/tftpboot" ]
Brendan Shephard
Software Engineer
Red Hat APAC <https://www.redhat.com>
Brisbane City QLD <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> 4000 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc>
<https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> <https://red.ht/sig> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> <https://redhat.com/summit>
On Wed, Jun 15, 2022 at 7:19 PM Lokendra Rathour < lokendrarathour@gmail.com> wrote:
> Hi Shephard, > Here is the o/p of the file: > > [root@undercloud ~]# sudo cat > /var/lib/kolla/config_files/ironic_pxe_tftp.json > { > "config_files": [ > { > "dest": "/", > "merge": true, > "preserve_properties": true, > "source": "/var/lib/kolla/config_files/src/*" > } > ], > "permissions": [ > { > "owner": "ironic:ironic", > "path": "/var/log/ironic", > "recurse": true > }, > { > "owner": "ironic:ironic", > "path": "/var/lib/ironic", > "recurse": true > } > ] > }[root@undercloud ~]# > > > Thanks once agan. > > -Lokendra > > > On Wed, Jun 15, 2022 at 2:38 PM Brendan Shephard < > bshephar@redhat.com> wrote: > >> Looks like something wrong with the dnsmasq command the container >> is being launched with. What command is it trying to run? >> >> sudo cat /var/lib/kolla/config_files/ironic_pxe_tftp.json >> >> Brendan Shephard >> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> >> >> Software Engineer >> >> Red Hat APAC <https://www.redhat.com> >> >> 193 N Quay >> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> >> >> Brisbane City QLD >> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> >> 4000 >> @RedHat <https://twitter.com/redhat> Red Hat >> <https://www.linkedin.com/company/red-hat> Red Hat >> <https://www.facebook.com/RedHatInc> >> <https://red.ht/sig> >> <https://redhat.com/summit> >> >> >> On Wed, Jun 15, 2022 at 6:22 PM Anirudh Gupta <anyrude10@gmail.com> >> wrote: >> >>> Hi Brendan, >>> >>> Thanks for your response. >>> >>> Please find the log below. >>> >>> [stack@undercloud t2u2v2w]$ sudo podman logs ironic_pxe_tftp >>> >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> dnsmasq: bad command line options: try --help >>> >>> [stack@undercloud t2u2v2w]$ sudo podman ps --filter >>> name=ironic_pxe -a >>> CONTAINER ID IMAGE >>> COMMAND CREATED >>> STATUS PORTS NAMES >>> 02dacbc74cec >>> undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo >>> /bin/bash -c BIND... 3 hours ago Exited (1) 3 hours ago (unhealthy) >>> ironic_pxe_tftp >>> 1f8ca39fba32 >>> undercloud.ctlplane.localdomain:8787/tripleowallaby/openstack-ironic-pxe:current-tripleo >>> kolla_start 3 hours ago Up 3 hours ago (healthy) >>> ironic_pxe_http >>> >>> >>> Regards >>> >>> Anirudh Gupta >>> >>> On Wed, Jun 15, 2022 at 11:30 AM Brendan Shephard < >>> bshephar@redhat.com> wrote: >>> >>>> Hey Anirudh, >>>> >>>> You would need to look at the logs for the ironic_pxe_tftp container >>>> to see why it's failing. >>>> >>>> I assume the tftp container is not Up when you run this command? >>>> [stack@tripleo-director overcloud_playbooks]$ sudo podman ps >>>> --filter name=ironic_pxe -a >>>> CONTAINER ID IMAGE >>>> COMMAND CREATED STATUS >>>> PORTS NAMES >>>> 0170be36e291 >>>> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo >>>> kolla_start 12 days ago Up 30 hours ago (healthy) >>>> ironic_pxe_tftp >>>> e507f722bdf0 >>>> registry.okd4.bne-shift.net:8443/tripleomastercentos9/openstack-ironic-pxe:current-tripleo >>>> kolla_start 12 days ago Up 30 hours ago (healthy) >>>> ironic_pxe_http >>>> >>>> Then check the logs to see what the error is: >>>> [stack@tripleo-director overcloud_playbooks]$ sudo podman logs >>>> ironic_pxe_tftp >>>> >>>> >>>> >>>> >>>> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> >>>> >>>> >>>> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> >>>> Brendan Shephard >>>> >>>> Software Engineer >>>> >>>> >>>> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> Red >>>> Hat APAC <https://www.redhat.com> >>>> >>>> 193 N Quay >>>> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> >>>> >>>> Brisbane City QLD >>>> <https://www.google.com/maps/search/193+N+Quay+%0D%0A+++++++%0D%0A+++++++%0D%0A+++++++++Brisbane+City+QLD?entry=gmail&source=g> >>>> 4000 >>>> @RedHat <https://twitter.com/redhat> Red Hat >>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>> <https://www.facebook.com/RedHatInc> >>>> <https://red.ht/sig> >>>> <https://redhat.com/summit> >>>> >>>> >>>> On Wed, Jun 15, 2022 at 7:53 AM Anirudh Gupta < >>>> anyrude10@gmail.com> wrote: >>>> >>>>> Hi Team, >>>>> >>>>> I am trying to deploy Openstack Wallaby Undercloud on IPv6, but >>>>> facing the below error: >>>>> >>>>> 2022-06-14 05:01:23.213708 | >>>>> 52540083-cfa2-3f20-e9dc-00000000286f | TASK | Manage container systemd >>>>> services and cleanup old systemd healthchecks for >>>>> /var/lib/tripleo-config/container-startup-config/step_4 >>>>> 2022-06-14 05:03:22.912816 | >>>>> 52540083-cfa2-3f20-e9dc-00000000286f | FATAL | Manage container systemd >>>>> services and cleanup old systemd healthchecks for >>>>> /var/lib/tripleo-config/container-startup-config/step_4 | undercloud | >>>>> error={"changed": false, "msg": "Service ironic_pxe_tftp has not started >>>>> yet"} >>>>> 2022-06-14 05:03:22.914400 | >>>>> 52540083-cfa2-3f20-e9dc-00000000286f | TIMING | tripleo_container_manage : >>>>> Manage container systemd >>>>> >>>>> Sample Undercloud.conf is as follows: >>>>> >>>>> [DEFAULT] >>>>> clean_nodes = true >>>>> cleanup = false >>>>> container_cli = podman >>>>> container_healthcheck_disabled = true >>>>> container_images_file = >>>>> /home/stack/containers-prepare-parameter.yaml >>>>> deployment_user = stack >>>>> enable_ironic = true >>>>> enable_ironic_inspector = true >>>>> enable_neutron = true >>>>> enable_routed_networks = false >>>>> generate_service_certificate = false >>>>> ipv6_address_mode = dhcpv6-stateful >>>>> ipxe_enabled = true >>>>> local_interface = enp8s0 >>>>> local_ip = aaaa:aaaa:aaaa::1/64 >>>>> subnets = ctlplane-subnet >>>>> undercloud_admin_host = aaaa:aaaa:aaaa::1 >>>>> undercloud_hostname = undercloud.com >>>>> undercloud_ntp_servers = 30.30.30.3 >>>>> undercloud_public_host = aaaa:aaaa:aaaa::1 >>>>> undercloud_timezone = UTC >>>>> >>>>> [ctlplane-subnet] >>>>> cidr = aaaa:aaaa:aaaa::/64 >>>>> dhcp_end = aaaa:aaaa:aaaa::f >>>>> dhcp_start = aaaa:aaaa:aaaa::a >>>>> gateway = aaaa:aaaa:aaaa::1 >>>>> inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9 >>>>> >>>>> Can someone please help in this regard. >>>>> >>>>> Anirudh Gupta >>>>> >>>>> > > -- > ~ Lokendra > www.inertiaspeaks.com > www.inertiagroups.com > skype: lokendrarathour > > >
-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour
-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour
-- Dan Sneddon | Senior Principal Software Engineer dsneddon@redhat.com | redhat.com/cloud dsneddon:irc | @dxs:twitter
-- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour