Eugen, Thank you again for your response. Your help is greatly appreciated. When I run iptables -nvL, I see that the REJECT in the input chain does have some packets that hit that rule, but the one in the FORWARD chain has a 0 packet count. However, when I ping the 10.61.157.1 gateway address or 8.8.8.8 from the cirros instance, that counter does not increment, so I don't think that is the issue 🙁 I believe that those REJECT rules are the "default" ones that take effect if no other rule applies. Is my understanding correct there? I am willing to try to make adjustments to my iptables rules, but am not sure how without breaking things. Do you have any suggestions? Here is the output of the iptables -nvL Every 2.0s: iptables -nvL l21652.ldschurch.org: Tue Feb 4 08:05:49 2025 Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 5843K 1115M ACCEPT 6 -- * * 10.61.157.59 0.0.0.0/0 multiport dports 5671,5672 /* 001 amqp incoming amqp_10.61.157.59 */ 0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 8042 /* 001 aodh-api incoming aodh_api */ 268K 21M ACCEPT 6 -- * * 10.61.157.59 0.0.0.0/0 multiport dports 3260 /* 001 cinder incoming cinder_10.61.157.59 */ 36713 4659K ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 8776 /* 001 cinder-api incoming cinder_api */ 1409 293K ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 9292 /* 001 glance incoming glance_api */ 150K 30M ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 8041 /* 001 gnocchi-api incoming gnocchi_api */ 7846 1851K ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 80 /* 001 horizon 80 incoming */ 246K 31M ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 5000 /* 001 keystone incoming keystone */ 23M 3860M ACCEPT 6 -- * * 10.61.157.59 0.0.0.0/0 multiport dports 3306 /* 001 mariadb incoming mariadb_10.61.157.59 */ 155K 62M ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 9696 /* 001 neutron server incoming neutron_server_10.61.157.59 */ 0 0 ACCEPT 17 -- * * 10.61.157.59 0.0.0.0/0 udp dpt:6081 /* 001 neutron tunnel port incoming neutron_tunnel_10.61.157.59_10.61.157.59 */ 163K 32M ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 8773,8774,8775,8778 /* 001 nova api incoming nova_api */ 177K 11M ACCEPT 6 -- * * 10.61.157.59 0.0.0.0/0 multiport dports 5900:5999 /* 001 nova compute incoming nova_compute */ 33021 8785K ACCEPT 6 -- * * 10.61.157.59 0.0.0.0/0 multiport dports 16509,49152:49215 /* 001 nova qemu migration incoming nova_qemu_migration_10.61.157.59_10.61.157.59 */ 171K 9554K ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 6080 /* 001 novncproxy incoming */ 928K 84M ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 6641 /* 001 ovn northd incoming ovn_northd_10.61.157.59 */ 1544K 139M ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 6642 /* 001 ovn southd incoming ovn_southd_10.61.157.59 */ 7224K 1214M ACCEPT 6 -- * * 10.61.157.59 0.0.0.0/0 tcp dpt:6379 /* 001 redis service incoming redis service from 10.61.157.59 */ 6 428 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 8080 /* 001 swift proxy incoming swift_proxy */ 17015 2244K ACCEPT 6 -- * * 10.61.157.59 0.0.0.0/0 multiport dports 6000,6001,6002,873 /* 001 swift storage and rsync incoming swift_storage_and_rsync_10.61.157.59 */ 43M 8270M ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 20 1456 ACCEPT 1 -- * * 0.0.0.0/0 0.0.0.0/0 11403 684K ACCEPT 0 -- lo * 0.0.0.0/0 0.0.0.0/0 72 3800 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22 5880 1317K REJECT 0 -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT 0 -- br-ex * 0.0.0.0/0 0.0.0.0/0 /* 000 forward in */ 0 0 ACCEPT 0 -- * br-ex 0.0.0.0/0 0.0.0.0/0 /* 000 forward out */ 0 0 REJECT 0 -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT 83M packets, 16G bytes) pkts bytes target prot opt in out source destination Thanks! Collin ________________________________ From: Eugen Block <eblock@nde.ag> Sent: Tuesday, February 4, 2025 2:42 AM To: Collin Linford <clinford@familysearch.org> Cc: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> Subject: Re: [Ext:] Re: openstack instances unable to reach the internet Do you see actual packet drops or rejections if you run 'watch iptables -nvL'? I have those REJECT rules neither in my test clusters nor in our production cluster:
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Zitat von Collin Linford <clinford@familysearch.org>:
Yes, the security group allows icmp traffic as well as ssh traffic on top of the other default security group settings.
Selinux is running in permissive mode (so shouldn't be blocking anything) and firewalld is disabled.
We don't use apparmor.
I also am not an iptables expert, but I haven't seen anything in the rules that jump out to me as being problematic:
Here are the iptables rules: # iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT tcp -- l21652.ldschurch.org anywhere multiport dports amqps,amqp /* 001 amqp incoming amqp_10.61.157.59 */ ACCEPT tcp -- anywhere anywhere multiport dports fs-agent /* 001 aodh-api incoming aodh_api */ ACCEPT tcp -- l21652.ldschurch.org anywhere multiport dports iscsi-target /* 001 cinder incoming cinder_10.61.157.59 */ ACCEPT tcp -- anywhere anywhere multiport dports 8776 /* 001 cinder-api incoming cinder_api */ ACCEPT tcp -- anywhere anywhere multiport dports armtechdaemon /* 001 glance incoming glance_api */ ACCEPT tcp -- anywhere anywhere multiport dports 8041 /* 001 gnocchi-api incoming gnocchi_api */ ACCEPT tcp -- anywhere anywhere multiport dports http /* 001 horizon 80 incoming */ ACCEPT tcp -- anywhere anywhere multiport dports commplex-main /* 001 keystone incoming keystone */ ACCEPT tcp -- l21652.ldschurch.org anywhere multiport dports mysql /* 001 mariadb incoming mariadb_10.61.157.59 */ ACCEPT tcp -- anywhere anywhere multiport dports 9696 /* 001 neutron server incoming neutron_server_10.61.157.59 */ ACCEPT udp -- l21652.ldschurch.org anywhere udp dpt:geneve /* 001 neutron tunnel port incoming neutron_tunnel_10.61.157.59_10.61.157.59 */ ACCEPT tcp -- anywhere anywhere multiport dports 8773,8774,8775,uec /* 001 nova api incoming nova_api */ ACCEPT tcp -- l21652.ldschurch.org anywhere multiport dports rfb:cvsup /* 001 nova compute incoming nova_compute */ ACCEPT tcp -- l21652.ldschurch.org anywhere multiport dports 16509,49152:49215 /* 001 nova qemu migration incoming nova_qemu_migration_10.61.157.59_10.61.157.59 */ ACCEPT tcp -- anywhere anywhere multiport dports 6080 /* 001 novncproxy incoming */ ACCEPT tcp -- anywhere anywhere multiport dports 6641 /* 001 ovn northd incoming ovn_northd_10.61.157.59 */ ACCEPT tcp -- anywhere anywhere multiport dports 6642 /* 001 ovn southd incoming ovn_southd_10.61.157.59 */ ACCEPT tcp -- l21652.ldschurch.org anywhere tcp dpt:redis /* 001 redis service incoming redis service from 10.61.157.59 */ ACCEPT tcp -- anywhere anywhere multiport dports webcache /* 001 swift proxy incoming swift_proxy */ ACCEPT tcp -- l21652.ldschurch.org anywhere multiport dports x11,6001,6002,rsync /* 001 swift storage and rsync incoming swift_storage_and_rsync_10.61.157.59 */ ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT icmp -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere /* 000 forward in */ ACCEPT all -- anywhere anywhere /* 000 forward out */ REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT) target prot opt source destination
Thanks! Collin ________________________________ From: Eugen Block <eblock@nde.ag> Sent: Monday, February 3, 2025 8:20 AM To: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> Subject: [Ext:] Re: openstack instances unable to reach the internet
[External Email]
We use flat,vlan and vxlan. Did you check if the instance's security-group allows icmp traffic? Is there anything else potentially blocking traffic like firewalld, apparmor, selinux and so on? I guess looking at iptables could help here as well, but I'm not sure if I can really help with that.
Zitat von collinl@churchofjesuschrist.org:
Eugen, Thank you very much for your response.
What network type do you typically use? flat? vxlan?
I don't believe that there is routing between the controller/compute node and the gateway. . .and from my testing that is where the pings are getting lost.
Here is an example from the cirros instance pinging the controller/compute node (successfully), and the external gateway(not successfully): $ ping 10.61.157.59 PING 10.61.157.59 (10.61.157.59): 56 data bytes 64 bytes from 10.61.157.59: seq=0 ttl=63 time=1.378 ms 64 bytes from 10.61.157.59: seq=1 ttl=63 time=1.317 ms 64 bytes from 10.61.157.59: seq=2 ttl=63 time=0.901 ms ^C --- 10.61.157.59 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.901/1.198/1.378 ms $ ping 10.61.157.1 PING 10.61.157.1 (10.61.157.1): 56 data bytes ^C --- 10.61.157.1 ping statistics --- 9 packets transmitted, 0 packets received, 100% packet loss
How would I go about determining what is causing the packets to get lost at that point?
From the host 10.61.157.59 itself, it CAN ping the 10.61.157.1 gateway just fine. . .and all sites externally (i.e. 8.8.8.8)
here is the traceroute output from the controller/compute node, showing a successful output, and with only one hop. . .so I don't think there is anything between the host and the gateway network wise: # traceroute 10.61.157.1 traceroute to 10.61.157.1 (10.61.157.1), 30 hops max, 60 byte packets 1 _gateway (10.61.157.1) 0.374 ms 0.322 ms 0.297 ms