I got the metadata network accessible over a routed subnet! 

The breakthrough was making sure I pointed 169.254.169.254 to the Neutron DHCP address rather than the physical or bridge IP address, this route added to the physical router (L3 switch) that serves the subnet hosting the Ironic controller.

Also, I made sure to add a host route for 169.254.169.254 to the baremetal remote subnet in Neutron so the baremetal nodes will have a proper route to the metadata subnet. 

There is still some network tweaking left to do. We have a separate subnet for ESXi mgmt vmkernels since they use virtual MAC addresses and neutron doesn't recognize them. However, enabling DHCP relay for that separate subnet breaks routing which also breaks cleaning and deploying, and I need to find out where. 

This is huge for our metal-as-a-service offering! 

Thanks, 
Tom King

On Sat, Aug 22, 2020 at 12:53 AM Thomas King <thomas.king@gmail.com> wrote:
Ok, thanks. 

On Fri, Aug 21, 2020, 11:39 PM Ruslanas Gžibovskis <ruslanas@lpic.lt> wrote:
No, I didn't. In the beginning I wanted to say "yes" but that is for overcloud.

Now I am thinking, maybe even I do no have metadata on undercloud....

Will need to check on Monday.

2020-08-22, št 00:51, Thomas King <thomas.king@gmail.com> rašė:
Finally got it worked out except for reaching the metadata service IP (169.254.169.254) from a remote network. Did you put specific routes in place for that IP address on your physical network? 

Tom

On Wed, Jul 22, 2020 at 3:19 AM Ruslanas Gžibovskis <ruslanas@lpic.lt> wrote:
Ok

here is a small copy paste:
http://paste.openstack.org/show/o4Uay0DYbAdkcfUJGOLV/ 

also have relaunched previous commands, as I have redeployed overcloud :)

and maybe you will be interested, in undercloud containers running:

 (undercloud) [stack@remote-u overcloud]$ sudo podman ps
CONTAINER ID  IMAGE                                                                            COMMAND               CREATED      STATUS          PORTS  NAMES
39c684172ce7  docker.io/tripleomaster/centos-binary-neutron-dhcp-agent:current-tripleo         /usr/sbin/dnsmasq...  7 days ago   Up 7 days ago          neutron-dnsmasq-qdhcp-e6d6c50b-fb69-4375-bb7d-0f6e0cfed5cb
30ccb169e666  docker.io/tripleomaster/centos-binary-ironic-pxe:current-tripleo                 kolla_start           7 days ago   Up 7 days ago          ironic_pxe_http
6a62e08e8e10  docker.io/tripleomaster/centos-binary-ironic-pxe:current-tripleo                 /bin/bash -c BIND...  7 days ago   Up 7 days ago          ironic_pxe_tftp
e4cbad7f7488  docker.io/tripleomaster/centos-binary-neutron-l3-agent:current-tripleo           kolla_start           7 days ago   Up 7 days ago          neutron_l3_agent
d78d3828c420  docker.io/tripleomaster/centos-binary-neutron-openvswitch-agent:current-tripleo  kolla_start           7 days ago   Up 7 days ago          neutron_ovs_agent
3b67b6a69a87  docker.io/tripleomaster/centos-binary-neutron-dhcp-agent:current-tripleo         kolla_start           7 days ago   Up 7 days ago          neutron_dhcp
46d403293b54  docker.io/tripleomaster/centos-binary-swift-proxy-server:current-tripleo         kolla_start           7 days ago   Up 7 days ago          swift_proxy
15a9792d6f62  docker.io/tripleomaster/centos-binary-swift-object:current-tripleo               kolla_start           7 days ago   Up 7 days ago          swift_rsync
0e6cd094d6dc  docker.io/tripleomaster/centos-binary-swift-object:current-tripleo               kolla_start           7 days ago   Up 7 days ago          swift_object_updater
80b7f26e742d  docker.io/tripleomaster/centos-binary-swift-object:current-tripleo               kolla_start           7 days ago   Up 7 days ago          swift_object_server
765f79633499  docker.io/tripleomaster/centos-binary-swift-proxy-server:current-tripleo         kolla_start           7 days ago   Up 7 days ago          swift_object_expirer
0e53da61af88  docker.io/tripleomaster/centos-binary-swift-container:current-tripleo            kolla_start           7 days ago   Up 7 days ago          swift_container_updater
8c14cef58eb3  docker.io/tripleomaster/centos-binary-swift-account:current-tripleo              kolla_start           7 days ago   Up 7 days ago          swift_account_server
12930a63dc12  docker.io/tripleomaster/centos-binary-swift-container:current-tripleo            kolla_start           7 days ago   Up 7 days ago          swift_container_server
8a5cd45208c0  docker.io/tripleomaster/centos-binary-swift-account:current-tripleo              kolla_start           7 days ago   Up 7 days ago          swift_account_reaper
b871400bfdd9  docker.io/tripleomaster/centos-binary-mistral-executor:current-tripleo           kolla_start           7 days ago   Up 7 days ago          mistral_executor
3e98e5f83f09  docker.io/tripleomaster/centos-binary-mistral-event-engine:current-tripleo       kolla_start           7 days ago   Up 7 days ago          mistral_event_engine
52d711dffca3  docker.io/tripleomaster/centos-binary-mistral-engine:current-tripleo             kolla_start           7 days ago   Up 7 days ago          mistral_engine
23dec12e650f  docker.io/tripleomaster/centos-binary-iscsid:current-tripleo                     kolla_start           7 days ago   Up 7 days ago          iscsid
bafff227d9d7  docker.io/tripleomaster/centos-binary-haproxy:current-tripleo                    kolla_start           7 days ago   Up 7 days ago          haproxy
4d17cbd60698  docker.io/tripleomaster/centos-binary-ironic-inspector:current-tripleo           kolla_start           7 days ago   Up 7 days ago          ironic_inspector_dnsmasq
f77e49246c35  docker.io/tripleomaster/centos-binary-ironic-inspector:current-tripleo           kolla_start           7 days ago   Up 7 days ago          ironic_inspector
ec70f40bba04  docker.io/tripleomaster/centos-binary-rabbitmq:current-tripleo                   kolla_start           5 weeks ago  Up 4 weeks ago         rabbitmq
f09c9a129d85  docker.io/tripleomaster/centos-binary-memcached:current-tripleo                  kolla_start           5 weeks ago  Up 4 weeks ago         memcached
0eb1e953dcaa  docker.io/tripleomaster/centos-binary-nova-compute-ironic:current-tripleo        kolla_start           7 weeks ago  Up 4 weeks ago         nova_compute
ac78778d2cb0  docker.io/tripleomaster/centos-binary-ironic-neutron-agent:current-tripleo       kolla_start           7 weeks ago  Up 4 weeks ago         ironic_neutron_agent
8660bf80fd9c  docker.io/tripleomaster/centos-binary-ironic-conductor:current-tripleo           kolla_start           7 weeks ago  Up 4 weeks ago         ironic_conductor
9dead5068168  docker.io/tripleomaster/centos-binary-mistral-api:current-tripleo                kolla_start           7 weeks ago  Up 4 weeks ago         mistral_api
f5c1c9d6166c  docker.io/tripleomaster/centos-binary-ironic-api:current-tripleo                 kolla_start           7 weeks ago  Up 4 weeks ago         ironic_api
48899d782dd5  docker.io/tripleomaster/centos-binary-nova-api:current-tripleo                   kolla_start           7 weeks ago  Up 4 weeks ago         nova_api_cron
1b19c94834e8  docker.io/tripleomaster/centos-binary-nova-api:current-tripleo                   kolla_start           7 weeks ago  Up 4 weeks ago         nova_api
5b66d128c930  docker.io/tripleomaster/centos-binary-glance-api:current-tripleo                 kolla_start           7 weeks ago  Up 4 weeks ago         glance_api
7e63cdb8d6b4  docker.io/tripleomaster/centos-binary-placement-api:current-tripleo              kolla_start           7 weeks ago  Up 4 weeks ago         placement_api
2db105676f8b  docker.io/tripleomaster/centos-binary-zaqar-wsgi:current-tripleo                 kolla_start           7 weeks ago  Up 4 weeks ago         zaqar_websocket
d3e5ae7368e6  docker.io/tripleomaster/centos-binary-zaqar-wsgi:current-tripleo                 kolla_start           7 weeks ago  Up 4 weeks ago         zaqar
6332e26b40ec  docker.io/tripleomaster/centos-binary-nova-scheduler:current-tripleo             kolla_start           7 weeks ago  Up 4 weeks ago         nova_scheduler
a5b0e0904f0c  docker.io/tripleomaster/centos-binary-nova-conductor:current-tripleo             kolla_start           7 weeks ago  Up 4 weeks ago         nova_conductor
4460302d97c8  docker.io/tripleomaster/centos-binary-neutron-server:current-tripleo             kolla_start           7 weeks ago  Up 4 weeks ago         neutron_api
d418deb9ea13  docker.io/tripleomaster/centos-binary-cron:current-tripleo                       kolla_start           7 weeks ago  Up 4 weeks ago         logrotate_crond
7225ff80b26d  docker.io/tripleomaster/centos-binary-heat-engine:current-tripleo                kolla_start           7 weeks ago  Up 4 weeks ago         heat_engine
84d22b3b1663  docker.io/tripleomaster/centos-binary-heat-api:current-tripleo                   kolla_start           7 weeks ago  Up 4 weeks ago         heat_api_cron
226e0d839772  docker.io/tripleomaster/centos-binary-heat-api:current-tripleo                   kolla_start           7 weeks ago  Up 4 weeks ago         heat_api
6f7871a64325  docker.io/tripleomaster/centos-binary-keystone:current-tripleo                   kolla_start           7 weeks ago  Up 4 weeks ago         keystone
8402ff823012  docker.io/tripleomaster/centos-binary-mariadb:current-tripleo                    kolla_start           7 weeks ago  Up 4 weeks ago         mysql
(undercloud) [stack@remote-u overcloud]$