[kolla][neutron] VM is getting the wrong IP, can't contact metadata service
Hello,
I am having a weird issue. I am running openstack Victoria with OVN. Generally, everything is working. I created a VM using a fixed port with an IP of 10.210.64.6. There are no issues with this VM. When I create a new VM, I get this route info:
[ 29.363664] cloud-init[564]: ci-info: ++++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++++ [ 29.368568] cloud-init[564]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+ [ 29.369945] cloud-init[564]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | [ 29.372441] cloud-init[564]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+ [ 29.376434] cloud-init[564]: ci-info: | 0 | 0.0.0.0 | 10.210.64.1 | 0.0.0.0 | ens3 | UG | [ 29.377820] cloud-init[564]: ci-info: | 1 | 10.210.64.0 | 0.0.0.0 | 255.255.255.192 | ens3 | U | [ 29.380417] cloud-init[564]: ci-info: | 2 | 169.254.169.254 | 10.210.64.6 | 255.255.255.255 | ens3 | UGH | [ 29.384448] cloud-init[564]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
The IP is set correctly, but you can see I am getting a gateway of 10.210.64.6 for the metadata ser. ice. This is preventing the VM from reaching the service so the SSH key is not being added preventing me from logging in. As far as I can tell, there are no error and every VM I build is getting this .6 IP for this route.
Thanks, Chris
10.210.64.6 must be your computer node IP, it will redirect metadata requests to your controller node. 169.254.169.254 is a special IP address for meta services.
On Thu, Feb 17, 2022 at 1:55 PM Chris DiLorenzo cdilorenzo@gmail.com wrote:
Hello,
I am having a weird issue. I am running openstack Victoria with OVN. Generally, everything is working. I created a VM using a fixed port with an IP of 10.210.64.6. There are no issues with this VM. When I create a new VM, I get this route info:
[ 29.363664] cloud-init[564]: ci-info: ++++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++++ [ 29.368568] cloud-init[564]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+ [ 29.369945] cloud-init[564]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | [ 29.372441] cloud-init[564]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+ [ 29.376434] cloud-init[564]: ci-info: | 0 | 0.0.0.0 | 10.210.64.1 | 0.0.0.0 | ens3 | UG | [ 29.377820] cloud-init[564]: ci-info: | 1 | 10.210.64.0 | 0.0.0.0 | 255.255.255.192 | ens3 | U | [ 29.380417] cloud-init[564]: ci-info: | 2 | 169.254.169.254 | 10.210.64.6 | 255.255.255.255 | ens3 | UGH | [ 29.384448] cloud-init[564]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
The IP is set correctly, but you can see I am getting a gateway of 10.210.64.6 for the metadata ser. ice. This is preventing the VM from reaching the service so the SSH key is not being added preventing me from logging in. As far as I can tell, there are no error and every VM I build is getting this .6 IP for this route.
Thanks, Chris
Disregard my above email, I didn't see that you created vm using a fixed IP address. When you create a network/subnet during that time OVN creates this special localport IP. In my case its 10.1.1.5
I believe you shouldn't use that IP address for VM.
port 0e36e2b1-12d2-4eb2-a9ea-556644a73067 type: localport addresses: ["fa:16:3e:6e:36:f9 10.1.1.5"]
On Thu, Feb 17, 2022 at 2:00 PM Satish Patel satish.txt@gmail.com wrote:
10.210.64.6 must be your computer node IP, it will redirect metadata requests to your controller node. 169.254.169.254 is a special IP address for meta services.
On Thu, Feb 17, 2022 at 1:55 PM Chris DiLorenzo cdilorenzo@gmail.com wrote:
Hello,
I am having a weird issue. I am running openstack Victoria with OVN. Generally, everything is working. I created a VM using a fixed port with an IP of 10.210.64.6. There are no issues with this VM. When I create a new VM, I get this route info:
[ 29.363664] cloud-init[564]: ci-info: ++++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++++ [ 29.368568] cloud-init[564]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+ [ 29.369945] cloud-init[564]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | [ 29.372441] cloud-init[564]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+ [ 29.376434] cloud-init[564]: ci-info: | 0 | 0.0.0.0 | 10.210.64.1 | 0.0.0.0 | ens3 | UG | [ 29.377820] cloud-init[564]: ci-info: | 1 | 10.210.64.0 | 0.0.0.0 | 255.255.255.192 | ens3 | U | [ 29.380417] cloud-init[564]: ci-info: | 2 | 169.254.169.254 | 10.210.64.6 | 255.255.255.255 | ens3 | UGH | [ 29.384448] cloud-init[564]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
The IP is set correctly, but you can see I am getting a gateway of 10.210.64.6 for the metadata ser. ice. This is preventing the VM from reaching the service so the SSH key is not being added preventing me from logging in. As far as I can tell, there are no error and every VM I build is getting this .6 IP for this route.
Thanks, Chris
Hello Chris:
10.210.65.6 is the IP address of the metadata namespace interface. The metadata service is on 169.254.169.254 address; this route is redirecting this traffic. Check the network ID and then the corresponding metadata: root@u20ovn:~# ip netns ovnmeta-5274fa45-c33a-4c66-be3a-b8ecd294c769 (id: 3) ovnmeta-ee1597ee-579d-49dc-9b90-fbb8f98baef9 (id: 2)
root@u20ovn:~# ip netns exec $meta ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: tap5274fa45-c1@if62: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:dd:2b:2c brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.0.10.3/26 brd 10.0.10.63 scope global tap5274fa45-c1 valid_lft forever preferred_lft forever inet 169.254.169.254/32 brd 169.254.169.254 scope global tap5274fa45-c1
Check if the metadata agent logs and verify you are not hitting [1]. Check you have [2].
Regards.
[1]https://bugs.launchpad.net/neutron/+bug/1952550 [2]https://review.opendev.org/c/openstack/neutron/+/822669
On Thu, Feb 17, 2022 at 8:14 PM Satish Patel satish.txt@gmail.com wrote:
Disregard my above email, I didn't see that you created vm using a fixed IP address. When you create a network/subnet during that time OVN creates this special localport IP. In my case its 10.1.1.5
I believe you shouldn't use that IP address for VM.
port 0e36e2b1-12d2-4eb2-a9ea-556644a73067 type: localport addresses: ["fa:16:3e:6e:36:f9 10.1.1.5"]
On Thu, Feb 17, 2022 at 2:00 PM Satish Patel satish.txt@gmail.com wrote:
10.210.64.6 must be your computer node IP, it will redirect metadata requests to your controller node. 169.254.169.254 is a special IP address for meta services.
On Thu, Feb 17, 2022 at 1:55 PM Chris DiLorenzo cdilorenzo@gmail.com
wrote:
Hello,
I am having a weird issue. I am running openstack Victoria with OVN.
Generally, everything is working. I created a VM using a fixed port with an IP of 10.210.64.6. There are no issues with this VM. When I create a new VM, I get this route info:
[ 29.363664] cloud-init[564]: ci-info:
++++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++++
[ 29.368568] cloud-init[564]: ci-info:
+-------+-----------------+-------------+-----------------+-----------+-------+
[ 29.369945] cloud-init[564]: ci-info: | Route | Destination |
Gateway | Genmask | Interface | Flags |
[ 29.372441] cloud-init[564]: ci-info:
+-------+-----------------+-------------+-----------------+-----------+-------+
[ 29.376434] cloud-init[564]: ci-info: | 0 | 0.0.0.0 |
10.210.64.1 | 0.0.0.0 | ens3 | UG |
[ 29.377820] cloud-init[564]: ci-info: | 1 | 10.210.64.0 |
0.0.0.0 | 255.255.255.192 | ens3 | U |
[ 29.380417] cloud-init[564]: ci-info: | 2 | 169.254.169.254 |
10.210.64.6 | 255.255.255.255 | ens3 | UGH |
[ 29.384448] cloud-init[564]: ci-info:
+-------+-----------------+-------------+-----------------+-----------+-------+
The IP is set correctly, but you can see I am getting a gateway of
10.210.64.6 for the metadata ser. ice. This is preventing the VM from reaching the service so the SSH key is not being added preventing me from logging in. As far as I can tell, there are no error and every VM I build is getting this .6 IP for this route.
Thanks, Chris
I ended up figuring it out. All the interfaces were created correctly. However, for some reason the DHCP options for this subnet were incorrect. The static route for the subnet was set to the .6 address but neutron had setup the metadata interface on .16. Once I modified the DHCP options and built new instances everything worked fine.
Thanks Chris
On Mon, Feb 21, 2022 at 6:49 AM Rodolfo Alonso Hernandez < ralonsoh@redhat.com> wrote:
Hello Chris:
10.210.65.6 is the IP address of the metadata namespace interface. The metadata service is on 169.254.169.254 address; this route is redirecting this traffic. Check the network ID and then the corresponding metadata: root@u20ovn:~# ip netns ovnmeta-5274fa45-c33a-4c66-be3a-b8ecd294c769 (id: 3) ovnmeta-ee1597ee-579d-49dc-9b90-fbb8f98baef9 (id: 2)
root@u20ovn:~# ip netns exec $meta ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: tap5274fa45-c1@if62: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:dd:2b:2c brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.0.10.3/26 brd 10.0.10.63 scope global tap5274fa45-c1 valid_lft forever preferred_lft forever inet 169.254.169.254/32 brd 169.254.169.254 scope global tap5274fa45-c1
Check if the metadata agent logs and verify you are not hitting [1]. Check you have [2].
Regards.
[1]https://bugs.launchpad.net/neutron/+bug/1952550 [2]https://review.opendev.org/c/openstack/neutron/+/822669
On Thu, Feb 17, 2022 at 8:14 PM Satish Patel satish.txt@gmail.com wrote:
Disregard my above email, I didn't see that you created vm using a fixed IP address. When you create a network/subnet during that time OVN creates this special localport IP. In my case its 10.1.1.5
I believe you shouldn't use that IP address for VM.
port 0e36e2b1-12d2-4eb2-a9ea-556644a73067 type: localport addresses: ["fa:16:3e:6e:36:f9 10.1.1.5"]
On Thu, Feb 17, 2022 at 2:00 PM Satish Patel satish.txt@gmail.com wrote:
10.210.64.6 must be your computer node IP, it will redirect metadata requests to your controller node. 169.254.169.254 is a special IP address for meta services.
On Thu, Feb 17, 2022 at 1:55 PM Chris DiLorenzo cdilorenzo@gmail.com
wrote:
Hello,
I am having a weird issue. I am running openstack Victoria with
OVN. Generally, everything is working. I created a VM using a fixed port with an IP of 10.210.64.6. There are no issues with this VM. When I create a new VM, I get this route info:
[ 29.363664] cloud-init[564]: ci-info:
++++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++++
[ 29.368568] cloud-init[564]: ci-info:
+-------+-----------------+-------------+-----------------+-----------+-------+
[ 29.369945] cloud-init[564]: ci-info: | Route | Destination |
Gateway | Genmask | Interface | Flags |
[ 29.372441] cloud-init[564]: ci-info:
+-------+-----------------+-------------+-----------------+-----------+-------+
[ 29.376434] cloud-init[564]: ci-info: | 0 | 0.0.0.0 |
10.210.64.1 | 0.0.0.0 | ens3 | UG |
[ 29.377820] cloud-init[564]: ci-info: | 1 | 10.210.64.0 |
0.0.0.0 | 255.255.255.192 | ens3 | U |
[ 29.380417] cloud-init[564]: ci-info: | 2 | 169.254.169.254 |
10.210.64.6 | 255.255.255.255 | ens3 | UGH |
[ 29.384448] cloud-init[564]: ci-info:
+-------+-----------------+-------------+-----------------+-----------+-------+
The IP is set correctly, but you can see I am getting a gateway of
10.210.64.6 for the metadata ser. ice. This is preventing the VM from reaching the service so the SSH key is not being added preventing me from logging in. As far as I can tell, there are no error and every VM I build is getting this .6 IP for this route.
Thanks, Chris
participants (3)
-
Chris DiLorenzo
-
Rodolfo Alonso Hernandez
-
Satish Patel