Hi Derek,
The symptoms you’ve described sound very much like a metadata service issue, whether the service is unavailable or not functioning properly. Sometimes this can be determined by looking at the VM console to see if access to http://169.254.169.254 is repeatedly tried (and failing). If it’s intermittent, there may be a particular instance of the metadata service having issues. For VMs on provider networks (w/ OVS or LXB, specifically), metadata access may provided via a static route that’s pushed by the DHCP server that answered the call. I recommend checking out the metadata agent and dhcp agent logs. If you’re using OVN, then it could still be a metadata issue but the solution could look different. Can you describe your topology?
Hope that helps,
James Denton Rackspace Private Cloud
From: Derek O keeffe derekokeeffe85@yahoo.ie Date: Thursday, August 18, 2022 at 3:41 AM To: Openstack-discuss openstack-discuss@lists.openstack.org Subject: Weird behaviour on a provider net CAUTION: This message originated externally, please use caution when clicking on links or opening attachments!
Hi all,
We have an Openstack Victoria cluster deployed and a provider network (IPv4) created that worked fine for months. Lately we've noticed that about 30-40% of the VM's that are spun up using this network are inaccessible over ssh, Permission denied (publickey). Also, if you add a customisation script to create a password for the Ubuntu user that fails too.
Initially we thought it was confined to one/two compute nodes but further testing this week shows that it can be any of the 6 compute nodes we have and always the original two that we identified as being the problem. If we create the VM's from the cli and send them to specific hosts everything works fine including the original two hosts that had issues.
We're finding it hard to nail down the cause of this because it's not consistent behaviour so any tips on where to investigate would be welcome, we have a second provider net (IPv4 & IPv6) that works fine and have went through all the configs to see if there's any difference and all are identical over the hosts.
Thanks in advance for any info.
Regards, Derek