I got the metadata network accessible over a routed subnet! The breakthrough was making sure I pointed 169.254.169.254 to the Neutron DHCP address rather than the physical or bridge IP address, this route added to the physical router (L3 switch) that serves the subnet hosting the Ironic controller. Also, I made sure to add a host route for 169.254.169.254 to the baremetal remote subnet in Neutron so the baremetal nodes will have a proper route to the metadata subnet. There is still some network tweaking left to do. We have a separate subnet for ESXi mgmt vmkernels since they use virtual MAC addresses and neutron doesn't recognize them. However, enabling DHCP relay for that separate subnet breaks routing which also breaks cleaning and deploying, and I need to find out where. This is huge for our metal-as-a-service offering! Thanks, Tom King On Sat, Aug 22, 2020 at 12:53 AM Thomas King <thomas.king@gmail.com> wrote:

Ok, thanks.

On Fri, Aug 21, 2020, 11:39 PM Ruslanas Gžibovskis <ruslanas@lpic.lt> wrote:

...

No, I didn't. In the beginning I wanted to say "yes" but that is for overcloud.

Now I am thinking, maybe even I do no have metadata on undercloud....

Will need to check on Monday.

2020-08-22, št 00:51, Thomas King <thomas.king@gmail.com> rašė:

...

Finally got it worked out except for reaching the metadata service IP (169.254.169.254) from a remote network. Did you put specific routes in place for that IP address on your physical network?

Tom

On Wed, Jul 22, 2020 at 3:19 AM Ruslanas Gžibovskis <ruslanas@lpic.lt> wrote:

...

Ok

here is a small copy paste: http://paste.openstack.org/show/o4Uay0DYbAdkcfUJGOLV/

also have relaunched previous commands, as I have redeployed overcloud :)

and maybe you will be interested, in undercloud containers running:

(undercloud) [stack@remote-u overcloud]$ sudo podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 39c684172ce7 docker.io/tripleomaster/centos-binary-neutron-dhcp-agent:current-tripleo /usr/sbin/dnsmasq... 7 days ago Up 7 days ago neutron-dnsmasq-qdhcp-e6d6c50b-fb69-4375-bb7d-0f6e0cfed5cb 30ccb169e666 docker.io/tripleomaster/centos-binary-ironic-pxe:current-tripleo kolla_start 7 days ago Up 7 days ago ironic_pxe_http 6a62e08e8e10 docker.io/tripleomaster/centos-binary-ironic-pxe:current-tripleo /bin/bash -c BIND... 7 days ago Up 7 days ago ironic_pxe_tftp e4cbad7f7488 docker.io/tripleomaster/centos-binary-neutron-l3-agent:current-tripleo kolla_start 7 days ago Up 7 days ago neutron_l3_agent d78d3828c420 docker.io/tripleomaster/centos-binary-neutron-openvswitch-agent:current-tripleo kolla_start 7 days ago Up 7 days ago neutron_ovs_agent 3b67b6a69a87 docker.io/tripleomaster/centos-binary-neutron-dhcp-agent:current-tripleo kolla_start 7 days ago Up 7 days ago neutron_dhcp 46d403293b54 docker.io/tripleomaster/centos-binary-swift-proxy-server:current-tripleo kolla_start 7 days ago Up 7 days ago swift_proxy 15a9792d6f62 docker.io/tripleomaster/centos-binary-swift-object:current-tripleo kolla_start 7 days ago Up 7 days ago swift_rsync 0e6cd094d6dc docker.io/tripleomaster/centos-binary-swift-object:current-tripleo kolla_start 7 days ago Up 7 days ago swift_object_updater 80b7f26e742d docker.io/tripleomaster/centos-binary-swift-object:current-tripleo kolla_start 7 days ago Up 7 days ago swift_object_server 765f79633499 docker.io/tripleomaster/centos-binary-swift-proxy-server:current-tripleo kolla_start 7 days ago Up 7 days ago swift_object_expirer 0e53da61af88 docker.io/tripleomaster/centos-binary-swift-container:current-tripleo kolla_start 7 days ago Up 7 days ago swift_container_updater 8c14cef58eb3 docker.io/tripleomaster/centos-binary-swift-account:current-tripleo kolla_start 7 days ago Up 7 days ago swift_account_server 12930a63dc12 docker.io/tripleomaster/centos-binary-swift-container:current-tripleo kolla_start 7 days ago Up 7 days ago swift_container_server 8a5cd45208c0 docker.io/tripleomaster/centos-binary-swift-account:current-tripleo kolla_start 7 days ago Up 7 days ago swift_account_reaper b871400bfdd9 docker.io/tripleomaster/centos-binary-mistral-executor:current-tripleo kolla_start 7 days ago Up 7 days ago mistral_executor 3e98e5f83f09 docker.io/tripleomaster/centos-binary-mistral-event-engine:current-tripleo kolla_start 7 days ago Up 7 days ago mistral_event_engine 52d711dffca3 docker.io/tripleomaster/centos-binary-mistral-engine:current-tripleo kolla_start 7 days ago Up 7 days ago mistral_engine 23dec12e650f docker.io/tripleomaster/centos-binary-iscsid:current-tripleo kolla_start 7 days ago Up 7 days ago iscsid bafff227d9d7 docker.io/tripleomaster/centos-binary-haproxy:current-tripleo kolla_start 7 days ago Up 7 days ago haproxy 4d17cbd60698 docker.io/tripleomaster/centos-binary-ironic-inspector:current-tripleo kolla_start 7 days ago Up 7 days ago ironic_inspector_dnsmasq f77e49246c35 docker.io/tripleomaster/centos-binary-ironic-inspector:current-tripleo kolla_start 7 days ago Up 7 days ago ironic_inspector ec70f40bba04 docker.io/tripleomaster/centos-binary-rabbitmq:current-tripleo kolla_start 5 weeks ago Up 4 weeks ago rabbitmq f09c9a129d85 docker.io/tripleomaster/centos-binary-memcached:current-tripleo kolla_start 5 weeks ago Up 4 weeks ago memcached 0eb1e953dcaa docker.io/tripleomaster/centos-binary-nova-compute-ironic:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago nova_compute ac78778d2cb0 docker.io/tripleomaster/centos-binary-ironic-neutron-agent:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago ironic_neutron_agent 8660bf80fd9c docker.io/tripleomaster/centos-binary-ironic-conductor:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago ironic_conductor 9dead5068168 docker.io/tripleomaster/centos-binary-mistral-api:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago mistral_api f5c1c9d6166c docker.io/tripleomaster/centos-binary-ironic-api:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago ironic_api 48899d782dd5 docker.io/tripleomaster/centos-binary-nova-api:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago nova_api_cron 1b19c94834e8 docker.io/tripleomaster/centos-binary-nova-api:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago nova_api 5b66d128c930 docker.io/tripleomaster/centos-binary-glance-api:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago glance_api 7e63cdb8d6b4 docker.io/tripleomaster/centos-binary-placement-api:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago placement_api 2db105676f8b docker.io/tripleomaster/centos-binary-zaqar-wsgi:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago zaqar_websocket d3e5ae7368e6 docker.io/tripleomaster/centos-binary-zaqar-wsgi:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago zaqar 6332e26b40ec docker.io/tripleomaster/centos-binary-nova-scheduler:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago nova_scheduler a5b0e0904f0c docker.io/tripleomaster/centos-binary-nova-conductor:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago nova_conductor 4460302d97c8 docker.io/tripleomaster/centos-binary-neutron-server:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago neutron_api d418deb9ea13 docker.io/tripleomaster/centos-binary-cron:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago logrotate_crond 7225ff80b26d docker.io/tripleomaster/centos-binary-heat-engine:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago heat_engine 84d22b3b1663 docker.io/tripleomaster/centos-binary-heat-api:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago heat_api_cron 226e0d839772 docker.io/tripleomaster/centos-binary-heat-api:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago heat_api 6f7871a64325 docker.io/tripleomaster/centos-binary-keystone:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago keystone 8402ff823012 docker.io/tripleomaster/centos-binary-mariadb:current-tripleo kolla_start 7 weeks ago Up 4 weeks ago mysql (undercloud) [stack@remote-u overcloud]$