Hi: It seems like the database container didn’t use the bridge network mode. please follow the next steps to collect more logs: 1. Check the docker configuration file : /etc/docker/daemon.json 2. Check the docker network : docker network ls 3. Check ip information: ip a 4. Docker inspect database command output. 5. You can also check the container network accessibility by the following steps - ln -s /var/run/docker/netns/ /var/run/netns - ip nets - ip netns exec <netns)> ip a. netns should be something like “ce9b0139b759” 6. If possible, please also collect the guest-agent logs. You can paste the log to https://etherpad.opendev.org/. And share the link to this email. thanks.

2023年12月28日 03:40，dieter.dorra@hsnr.de 写道：

Tony Breeds wrote:

...

On Tue, 19 Dec 2023 at 21:25, ddorra@t-online.de ddorra@t-online.de wrote:

...

Hello, I can't connect to to my trove mysql DB from an instance in a selfservice network. Strange thing is I CAN connect to the trove instance via ssh: ubuntu@dozbox2:~$ telnet 192.168.0.176 22 Trying 192.168.0.176... Connected to 192.168.0.176. Escape character is '^]'. SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.7 But I CANNOT reach the DB port from on the troave instance: ubuntu@box2:~$ telnet 192.168.0.176 3306 Trying 192.168.0.176... telnet: Unable to connect to remote host: No route to host Within the Trove instance I CAN work with the client... root@ms3:/var/log/trove# mysql -h 192.168.0.176 -u dbadmin -p -P 3306 Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 331 I don't know much about trove but I wonder if the database process is

only listening on localhost? That would explain the behaviour you're seeing.

Yes I think that this is the right direction. For some time now Trove installs the DB into a docker container.

root@ms57plain:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ac9694b81e61 mysql:5.7.29 "docker-entrypoint.s…" 36 minutes ago Up 36 minutes database root@ms57plain:~#

I wonder why no ports are shown... maybe this is managed by iptable rules?

Surprisingly I see no listening to port 3306 via tcp on host level.

root@ms57plain:~# netstat -anp Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 3867/systemd-resolv tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1074/sshd tcp 0 0 10.10.10.84:54326 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp 0 304 10.9.9.126:22 10.9.9.128:47442 ESTABLISHED 5157/sshd: root@pts tcp 0 0 10.9.9.126:22 10.9.9.128:57180 ESTABLISHED 5054/sshd: root@pts tcp 0 0 10.10.10.84:44266 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp6 0 0 :::3306 :::* LISTEN 4881/mysqld tcp6 0 0 :::22 :::* LISTEN 1074/sshd

But the docker config shows the 3306

root@ms57plain:~# docker inspect ac9694b81e61 [ { ...... "AttachStderr": false, "ExposedPorts": { "3306/tcp": {}, "33060/tcp": {} }, "Tty": false, .....

I'm wondering that the ssh is explicitly mentioned in the iptable rules due to security group settings - but 3306 is missing?? Bug or feature??

 root@ms57plain:~# iptables -S
 -P INPUT ACCEPT
 -P FORWARD ACCEPT
 -P OUTPUT ACCEPT
 -N DOCKER
 -N DOCKER-ISOLATION-STAGE-1
 -N DOCKER-ISOLATION-STAGE-2
 -N DOCKER-USER
 -N openstack-INPUT
 -A INPUT -j openstack-INPUT
 -A FORWARD -j DOCKER-USER
 -A FORWARD -j DOCKER-ISOLATION-STAGE-1
 -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -A FORWARD -o docker0 -j DOCKER
 -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
 -A FORWARD -i docker0 -o docker0 -j ACCEPT
 -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
 -A DOCKER-ISOLATION-STAGE-1 -j RETURN
 -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
 -A DOCKER-ISOLATION-STAGE-2 -j RETURN
 -A DOCKER-USER -j RETURN
 -A openstack-INPUT -i lo -j ACCEPT
 -A openstack-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
 -A openstack-INPUT -p tcp -m tcp --dport 22 -j ACCEPT
 -A openstack-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
 -A openstack-INPUT -p tcp -m state --state NEW -m tcp --dport 19885 -j ACCEPT
 -A openstack-INPUT -s 172.24.4.0/23 -p udp -m udp --dport 69 -j ACCEPT
 -A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 6385 -j ACCEPT
 -A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 80 -j ACCEPT
 -A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 8000 -j ACCEPT
 -A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 8003 -j ACCEPT
 -A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 8004 -j ACCEPT

Any Ideas?

BR Dieter

According to the logs, the database should already be set up correctly. Could you confirm whether there is the same ip address in your network? You can execute the “arp -a” command to compare the certain Mac address with the Mac address within the guest vm.

2023年12月29日 05:03，dieter.dorra@hsnr.de 写道：

Hi thanks for the help. please find all the requested files in this etherpad https://etherpad.opendev.org/p/trove-cant-connect-to-docker Best regards Dieter

On Fri, 29 Dec 2023 at 19:58, dieter.dorra@hsnr.de wrote:

...

From the guest vm I can ssh into the trove instance (so port 22 is open and accepting) From the guest vm I cannot reach port 3306 From within the trove instance I can reach port 3306 and mysql is functional

It feels like the security group does not allow 3306, but the entry is there (see etherpad)

What puzzles me is that netstat in the trove instance does only show listen to 3306 for tcp6 but not for tcp - whereas 22 listens for tcp6 AND tcp

root@ms57plain:~# netstat -anp | head -17 Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 3867/systemd-resolv tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1074/sshd tcp 0 0 10.9.9.126:22 10.9.9.128:60126 ESTABLISHED 9412/sshd: root@pts tcp 0 0 10.10.10.84:54326 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp 0 0 10.9.9.126:22 10.9.9.128:57880 ESTABLISHED 9021/sshd: root@pts tcp 0 0 10.10.10.84:44266 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp6 0 0 :::3306 :::* LISTEN 4881/mysqld

You can see here that mysql is listening on all interfaces via an IPv4-mapped IPv6 address (https://en.wikipedia.org/wiki/IPv6#IPv4-mapped_IPv6_addresses), and from your previous post we can see that there is an explicit ACCEPT for the lo interface. That's how you're able to get access to the DB from within the guest.

It looks to me that whatever added the ' -A openstack-INPUT -p tcp -m tcp --dport 22 -j ACCEPT' to the guest's firewall didn't also add an explicit rule for the database. At this point I don't know if that's trove, nova or neutron. If I had to guess I'd say that there is a mismatch in the trove configuration and the networks assigned to the guest.

For example how does --- network_driver = trove.network.neutron.NeutronDriver # provider management_networks = 98b0de37-40af-4859-81f2-d89a4df2d6df # trovemgmt # provider ## management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # trove-mgmt 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 management_security_groups = 26c4af6e-56f0-4644-8c55-e96983c74c1d ---

relate to the "dozenet" and "trovemgmt" networks.

Tony.