[trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works
Hello, I can't connect to to my trove mysql DB from an instance in a selfservice network. Strange thing is I CAN connect to the trove instance via ssh: ubuntu@dozbox2:~$ telnet 192.168.0.176 22 Trying 192.168.0.176... Connected to 192.168.0.176. Escape character is '^]'. SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.7 But I CANNOT reach the DB port from on the troave instance: ubuntu@box2:~$ telnet 192.168.0.176 3306 Trying 192.168.0.176... telnet: Unable to connect to remote host: No route to host Within the Trove instance I CAN work with the client... root@ms3:/var/log/trove# mysql -h 192.168.0.176 -u dbadmin -p -P 3306 Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 331 The security group for that trove instance allows ssh AND 3306 (see additional infos/rules below). Any help is really appreciated! ------- Additional Infos ------------------ root@voscontrol:~# openstack server show 534fe146-0d51-4848-98ed-95dc9aaf89b1 +-------------------------------------+------------------------------------------------------------------+ | Field | Value | +-------------------------------------+------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | voscompute2 | | OS-EXT-SRV-ATTR:hypervisor_hostname | voscompute2 | | OS-EXT-SRV-ATTR:instance_name | instance-000000d0 | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2023-12-18T23:05:15.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | doznet=192.168.0.176; trovemgmt=10.10.10.134 | | config_drive | True | | created | 2023-12-18T23:04:59Z | | flavor | m1.large (72a7e41e-e0e9-48fe-abc2-8c64838e9e4a) | | hostId | 618af9a8f27a256f291621d611d30ae2186de8c2e8fa03e5a5ba2081 | | id | 534fe146-0d51-4848-98ed-95dc9aaf89b1 | | image | trove-guest-ubuntu-bionic (773aeec6-9caa-48a3-a80a-f9bf440b0d9e) | | key_name | trovekey | | name | ms3 | | progress | 0 | | project_id | d2e35c5479c441dc9ba4dba8e9b914bc | | properties | | | security_groups | name='trove_sg-27547af3-9bb0-4df5-b4ff-e85daa72c6ef' | | | name='default' | | status | ACTIVE | | updated | 2023-12-18T23:05:15Z | | user_id | a72e578837da479895fe4207107ca2fa | | volumes_attached | id='bff9037e-2831-473b-b6ae-8027ee4053dc' | +-------------------------------------+------------------------------------------------------------------+ root@voscontrol:~# openstack security group show trove_sg-27547af3-9bb0-4df5-b4ff-e85daa72c6ef +-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------+----------------------------------------------------------------------------------- | created_at | 2023-12-18T23:04:51Z | description | Security group for trove instance 27547af3-9bb0-4df5-b4ff-e85daa72c6ef | id | d73ff24a-5288-486b-b10b-fe9902f80e15 | name | trove_sg-27547af3-9bb0-4df5-b4ff-e85daa72c6ef | project_id | d2e35c5479c441dc9ba4dba8e9b914bc | revision_number | 3 | rules | created_at='2023-12-18T23:04:52Z', direction='ingress', ethertype='IPv4', id='0484ddff-c945-45d0-b271-7d30e7fbf619', port_range_max='3306', port_range_min='3306', protocol='tcp', remote_ip_prefix='0.0.0.0/0', updated_at='2023-12-18T23:04:52Z' | | | created_at='2023-12-18T23:04:51Z', direction='egress', ethertype='IPv6', id='27ed1225-984a-454e-86da-656de3669038', updated_at='2023-12-18T23:04:51Z' | | | created_at='2023-12-18T23:10:33Z', direction='ingress', ethertype='IPv4', id='c2c7c390-8b13-4b00-ab7c-0a8a7687ac08', port_range_max='22', port_range_min='22', protocol='tcp', remote_ip_prefix='0.0.0.0/0', updated_at='2023-12-18T23:10:33Z' | | | created_at='2023-12-18T23:04:51Z', direction='egress', ethertype='IPv4', id='d1fdc662-fc24-46bd-80db-3aca725f96aa', updated_at='2023-12-18T23:04:51Z' | | stateful | True | tags | [] | updated_at | 2023-12-18T23:10:33Z +---------------------------------------------------------------------------------------------+
On Tue, 19 Dec 2023 at 21:25, ddorra@t-online.de <ddorra@t-online.de> wrote:
Hello,
I can't connect to to my trove mysql DB from an instance in a selfservice network.
Strange thing is I CAN connect to the trove instance via ssh:
ubuntu@dozbox2:~$ telnet 192.168.0.176 22 Trying 192.168.0.176... Connected to 192.168.0.176. Escape character is '^]'. SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.7
But I CANNOT reach the DB port from on the troave instance:
ubuntu@box2:~$ telnet 192.168.0.176 3306 Trying 192.168.0.176... telnet: Unable to connect to remote host: No route to host
Within the Trove instance I CAN work with the client...
root@ms3:/var/log/trove# mysql -h 192.168.0.176 -u dbadmin -p -P 3306 Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 331
I don't know much about trove but I wonder if the database process is only listening on localhost? That would explain the behaviour you're seeing.
Tony Breeds wrote:
On Tue, 19 Dec 2023 at 21:25, ddorra@t-online.de ddorra@t-online.de wrote:
Hello, I can't connect to to my trove mysql DB from an instance in a selfservice network. Strange thing is I CAN connect to the trove instance via ssh: ubuntu@dozbox2:~$ telnet 192.168.0.176 22 Trying 192.168.0.176... Connected to 192.168.0.176. Escape character is '^]'. SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.7 But I CANNOT reach the DB port from on the troave instance: ubuntu@box2:~$ telnet 192.168.0.176 3306 Trying 192.168.0.176... telnet: Unable to connect to remote host: No route to host Within the Trove instance I CAN work with the client... root@ms3:/var/log/trove# mysql -h 192.168.0.176 -u dbadmin -p -P 3306 Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 331 I don't know much about trove but I wonder if the database process is only listening on localhost? That would explain the behaviour you're seeing.
Yes I think that this is the right direction. For some time now Trove installs the DB into a docker container. root@ms57plain:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ac9694b81e61 mysql:5.7.29 "docker-entrypoint.s…" 36 minutes ago Up 36 minutes database root@ms57plain:~# I wonder why no ports are shown... maybe this is managed by iptable rules? Surprisingly I see no listening to port 3306 via tcp on host level. root@ms57plain:~# netstat -anp Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 3867/systemd-resolv tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1074/sshd tcp 0 0 10.10.10.84:54326 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp 0 304 10.9.9.126:22 10.9.9.128:47442 ESTABLISHED 5157/sshd: root@pts tcp 0 0 10.9.9.126:22 10.9.9.128:57180 ESTABLISHED 5054/sshd: root@pts tcp 0 0 10.10.10.84:44266 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp6 0 0 :::3306 :::* LISTEN 4881/mysqld tcp6 0 0 :::22 :::* LISTEN 1074/sshd But the docker config shows the 3306 root@ms57plain:~# docker inspect ac9694b81e61 [ { ...... "AttachStderr": false, "ExposedPorts": { "3306/tcp": {}, "33060/tcp": {} }, "Tty": false, ..... I'm wondering that the ssh is explicitly mentioned in the iptable rules due to security group settings - but 3306 is missing?? Bug or feature?? root@ms57plain:~# iptables -S -P INPUT ACCEPT -P FORWARD ACCEPT -P OUTPUT ACCEPT -N DOCKER -N DOCKER-ISOLATION-STAGE-1 -N DOCKER-ISOLATION-STAGE-2 -N DOCKER-USER -N openstack-INPUT -A INPUT -j openstack-INPUT -A FORWARD -j DOCKER-USER -A FORWARD -j DOCKER-ISOLATION-STAGE-1 -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A FORWARD -o docker0 -j DOCKER -A FORWARD -i docker0 ! -o docker0 -j ACCEPT -A FORWARD -i docker0 -o docker0 -j ACCEPT -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2 -A DOCKER-ISOLATION-STAGE-1 -j RETURN -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP -A DOCKER-ISOLATION-STAGE-2 -j RETURN -A DOCKER-USER -j RETURN -A openstack-INPUT -i lo -j ACCEPT -A openstack-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT -A openstack-INPUT -p tcp -m tcp --dport 22 -j ACCEPT -A openstack-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A openstack-INPUT -p tcp -m state --state NEW -m tcp --dport 19885 -j ACCEPT -A openstack-INPUT -s 172.24.4.0/23 -p udp -m udp --dport 69 -j ACCEPT -A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 6385 -j ACCEPT -A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 80 -j ACCEPT -A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 8000 -j ACCEPT -A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 8003 -j ACCEPT -A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 8004 -j ACCEPT Any Ideas? BR Dieter
Hi: It seems like the database container didn’t use the bridge network mode. please follow the next steps to collect more logs: 1. Check the docker configuration file : /etc/docker/daemon.json 2. Check the docker network : docker network ls 3. Check ip information: ip a 4. Docker inspect database command output. 5. You can also check the container network accessibility by the following steps - ln -s /var/run/docker/netns/ /var/run/netns - ip nets - ip netns exec <netns)> ip a. netns should be something like “ce9b0139b759” 6. If possible, please also collect the guest-agent logs. You can paste the log to https://etherpad.opendev.org/. And share the link to this email. thanks.
2023年12月28日 03:40，dieter.dorra@hsnr.de 写道：
Tony Breeds wrote:
On Tue, 19 Dec 2023 at 21:25, ddorra@t-online.de ddorra@t-online.de wrote:
Hello, I can't connect to to my trove mysql DB from an instance in a selfservice network. Strange thing is I CAN connect to the trove instance via ssh: ubuntu@dozbox2:~$ telnet 192.168.0.176 22 Trying 192.168.0.176... Connected to 192.168.0.176. Escape character is '^]'. SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.7 But I CANNOT reach the DB port from on the troave instance: ubuntu@box2:~$ telnet 192.168.0.176 3306 Trying 192.168.0.176... telnet: Unable to connect to remote host: No route to host Within the Trove instance I CAN work with the client... root@ms3:/var/log/trove# mysql -h 192.168.0.176 -u dbadmin -p -P 3306 Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 331 I don't know much about trove but I wonder if the database process is only listening on localhost? That would explain the behaviour you're seeing.
Yes I think that this is the right direction. For some time now Trove installs the DB into a docker container.
root@ms57plain:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ac9694b81e61 mysql:5.7.29 "docker-entrypoint.s…" 36 minutes ago Up 36 minutes database root@ms57plain:~#
I wonder why no ports are shown... maybe this is managed by iptable rules?
Surprisingly I see no listening to port 3306 via tcp on host level.
root@ms57plain:~# netstat -anp Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 3867/systemd-resolv tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1074/sshd tcp 0 0 10.10.10.84:54326 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp 0 304 10.9.9.126:22 10.9.9.128:47442 ESTABLISHED 5157/sshd: root@pts tcp 0 0 10.9.9.126:22 10.9.9.128:57180 ESTABLISHED 5054/sshd: root@pts tcp 0 0 10.10.10.84:44266 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp6 0 0 :::3306 :::* LISTEN 4881/mysqld tcp6 0 0 :::22 :::* LISTEN 1074/sshd
But the docker config shows the 3306
root@ms57plain:~# docker inspect ac9694b81e61 [ { ...... "AttachStderr": false, "ExposedPorts": { "3306/tcp": {}, "33060/tcp": {} }, "Tty": false, .....
I'm wondering that the ssh is explicitly mentioned in the iptable rules due to security group settings - but 3306 is missing?? Bug or feature??
root@ms57plain:~# iptables -S -P INPUT ACCEPT -P FORWARD ACCEPT -P OUTPUT ACCEPT -N DOCKER -N DOCKER-ISOLATION-STAGE-1 -N DOCKER-ISOLATION-STAGE-2 -N DOCKER-USER -N openstack-INPUT -A INPUT -j openstack-INPUT -A FORWARD -j DOCKER-USER -A FORWARD -j DOCKER-ISOLATION-STAGE-1 -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A FORWARD -o docker0 -j DOCKER -A FORWARD -i docker0 ! -o docker0 -j ACCEPT -A FORWARD -i docker0 -o docker0 -j ACCEPT -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2 -A DOCKER-ISOLATION-STAGE-1 -j RETURN -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP -A DOCKER-ISOLATION-STAGE-2 -j RETURN -A DOCKER-USER -j RETURN -A openstack-INPUT -i lo -j ACCEPT -A openstack-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT -A openstack-INPUT -p tcp -m tcp --dport 22 -j ACCEPT -A openstack-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A openstack-INPUT -p tcp -m state --state NEW -m tcp --dport 19885 -j ACCEPT -A openstack-INPUT -s 172.24.4.0/23 -p udp -m udp --dport 69 -j ACCEPT -A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 6385 -j ACCEPT -A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 80 -j ACCEPT -A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 8000 -j ACCEPT -A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 8003 -j ACCEPT -A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 8004 -j ACCEPT
Any Ideas?
BR Dieter
Hi thanks for the help. please find all the requested files in this etherpad https://etherpad.opendev.org/p/trove-cant-connect-to-docker Best regards Dieter
According to the logs, the database should already be set up correctly. Could you confirm whether there is the same ip address in your network? You can execute the “arp -a” command to compare the certain Mac address with the Mac address within the guest vm.
2023年12月29日 05:03，dieter.dorra@hsnr.de 写道：
Hi thanks for the help. please find all the requested files in this etherpad https://etherpad.opendev.org/p/trove-cant-connect-to-docker Best regards Dieter
From the guest vm I can ssh into the trove instance (so port 22 is open and accepting) From the guest vm I cannot reach port 3306 From within the trove instance I can reach port 3306 and mysql is functional It feels like the security group does not allow 3306, but the entry is there (see etherpad) What puzzles me is that netstat in the trove instance does only show listen to 3306 for tcp6 but not for tcp - whereas 22 listens for tcp6 AND tcp root@ms57plain:~# netstat -anp | head -17 Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 3867/systemd-resolv tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1074/sshd tcp 0 0 10.9.9.126:22 10.9.9.128:60126 ESTABLISHED 9412/sshd: root@pts tcp 0 0 10.10.10.84:54326 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp 0 0 10.9.9.126:22 10.9.9.128:57880 ESTABLISHED 9021/sshd: root@pts tcp 0 0 10.10.10.84:44266 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp6 0 0 :::3306 :::* LISTEN 4881/mysqld tcp6 0 0 :::22 :::* LISTEN 1074/sshd udp 0 0 127.0.0.53:53 0.0.0.0:* 3867/systemd-resolv udp 0 0 10.10.10.84:68 0.0.0.0:* 3850/systemd-networ udp 0 0 10.9.9.126:68 0.0.0.0:* 3850/systemd-networ raw6 0 0 :::58 :::* 7 3850/systemd-networ raw6 0 0 :::58 :::* 7 3850/systemd-networ So how can I reach from within the 3306 at all? Does docker interfere here?
On Fri, 29 Dec 2023 at 19:58, <dieter.dorra@hsnr.de> wrote:
From the guest vm I can ssh into the trove instance (so port 22 is open and accepting) From the guest vm I cannot reach port 3306 From within the trove instance I can reach port 3306 and mysql is functional
It feels like the security group does not allow 3306, but the entry is there (see etherpad)
What puzzles me is that netstat in the trove instance does only show listen to 3306 for tcp6 but not for tcp - whereas 22 listens for tcp6 AND tcp
root@ms57plain:~# netstat -anp | head -17 Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 3867/systemd-resolv tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1074/sshd tcp 0 0 10.9.9.126:22 10.9.9.128:60126 ESTABLISHED 9412/sshd: root@pts tcp 0 0 10.10.10.84:54326 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp 0 0 10.9.9.126:22 10.9.9.128:57880 ESTABLISHED 9021/sshd: root@pts tcp 0 0 10.10.10.84:44266 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp6 0 0 :::3306 :::* LISTEN 4881/mysqld
You can see here that mysql is listening on all interfaces via an IPv4-mapped IPv6 address (https://en.wikipedia.org/wiki/IPv6#IPv4-mapped_IPv6_addresses), and from your previous post we can see that there is an explicit ACCEPT for the lo interface. That's how you're able to get access to the DB from within the guest. It looks to me that whatever added the ' -A openstack-INPUT -p tcp -m tcp --dport 22 -j ACCEPT' to the guest's firewall didn't also add an explicit rule for the database. At this point I don't know if that's trove, nova or neutron. If I had to guess I'd say that there is a mismatch in the trove configuration and the networks assigned to the guest. For example how does --- network_driver = trove.network.neutron.NeutronDriver # provider management_networks = 98b0de37-40af-4859-81f2-d89a4df2d6df # trovemgmt # provider ## management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # trove-mgmt 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 management_security_groups = 26c4af6e-56f0-4644-8c55-e96983c74c1d --- relate to the "dozenet" and "trovemgmt" networks. Tony.
The problem is solved! I tested a newer trove image from https://tarballs.opendev.org/openstack/trove/images/ The Victoria image caused the problem, it never showed the exposed ports in "docker ps". The Zed image is working: ubuntu@dd:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1741ae3e7815 mysql:5.7.29 "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp database ubuntu@dd:~$ Thank you all so much for your assistance and help!!! Happy new year!!
participants (4)
-
ddorra＠t-online.de
-
dieter.dorra＠hsnr.de
-
Tony Breeds
-
W Ch