RE: openstack-discuss Digest, Vol 62, Issue 108- Neutron Issue | Urgent Assistance
Hi Neutron_ Team I have deployed Multinode cluster of Openstack through kolla-ansible zed release. The problem I am facing is instances created cannot pick any Ip addresses (Rather than loop back ip or ipv6 addresses) from the created tenant networks with the default network type is vxlan. However, if I can assign ip manually to the eth0 interface in cirros instance, the interface is up with the ip address, but still, it can’t able to ping even its router gateway. Router can’t be able to ping instance ip address but only ping itself as verified below: The following troubleshooting steps, I have followed and concluded: ~ The Tenant network is created and its port: network: router_interface, network:dhcp, compute:nova are in admin up and active state. ~ The router interfaces are in up state and in complete nova zone. ~ The instance is running and active state and automatically pick ip address from the created tenant network as displayed in gui but inside the vm it can’t pick any ip address. ~ The security groups have been defined to allow icmp, tcp, udp for inbound/outbound traffic. Hence, I may not be able to ping and ssh instance from the controller node further these instances won’t be able to connect to the internet until the issue get resolved. Can you guide me how to resolve these issues related to network connectivity so I may able to send you the required files to troubleshoot easily? -----Original Message----- From: openstack-discuss-request@lists.openstack.org <openstack-discuss-request@lists.openstack.org> Sent: Tuesday, January 2, 2024 7:39 PM To: openstack-discuss@lists.openstack.org Subject: openstack-discuss Digest, Vol 62, Issue 108 Send openstack-discuss mailing list submissions to openstack-discuss@lists.openstack.org To subscribe or unsubscribe via email, send a message with subject or body 'help' to openstack-discuss-request@lists.openstack.org You can reach the person managing the list at openstack-discuss-owner@lists.openstack.org When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works (Tony Breeds) 2. Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works (dieter.dorra@hsnr.de) 3. Re: Trove installation filed (dieter.dorra@hsnr.de) 4. Re: Trove installation filed (KEREM CELIKER) 5. Re: [kolla] ceilometer and nova compute agent (KEREM CELIKER) 6. Re: ceilometer cli client install help (KEREM CELIKER) 7. [neutron] Bug deputy report (last week of 2023) (Bernard Cafarelli) 8. New OpenStack machine on new node not responding (kjme001@gmail.com) 9. Re: New OpenStack machine on new node not responding ( 韩光宇 ) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 Dec 2023 23:28:42 +1100 From: Tony Breeds <tony@bakeyournoodle.com> Subject: Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works To: dieter.dorra@hsnr.de Cc: openstack-discuss@lists.openstack.org Message-ID: <CAFT5D2RzaXi6==waA732R0PVe6H0OeKBwzF_qUSOTHPVfCu7Ww@mail.gmail.com> Content-Type: text/plain; charset="UTF-8" On Fri, 29 Dec 2023 at 19:58, <dieter.dorra@hsnr.de> wrote:
From the guest vm I can ssh into the trove instance (so port 22 is open and accepting) From the guest vm I cannot reach port 3306 From within the trove instance I can reach port 3306 and mysql is functional
It feels like the security group does not allow 3306, but the entry is there (see etherpad)
What puzzles me is that netstat in the trove instance does only show listen to 3306 for tcp6 but not for tcp - whereas 22 listens for tcp6 AND tcp
root@ms57plain:~# netstat -anp | head -17 Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 3867/systemd-resolv tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1074/sshd tcp 0 0 10.9.9.126:22 10.9.9.128:60126 ESTABLISHED 9412/sshd: root@pts tcp 0 0 10.10.10.84:54326 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp 0 0 10.9.9.126:22 10.9.9.128:57880 ESTABLISHED 9021/sshd: root@pts tcp 0 0 10.10.10.84:44266 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp6 0 0 :::3306 :::* LISTEN 4881/mysqld
You can see here that mysql is listening on all interfaces via an IPv4-mapped IPv6 address (https://en.wikipedia.org/wiki/IPv6#IPv4-mapped_IPv6_addresses), and from your previous post we can see that there is an explicit ACCEPT for the lo interface. That's how you're able to get access to the DB from within the guest. It looks to me that whatever added the ' -A openstack-INPUT -p tcp -m tcp --dport 22 -j ACCEPT' to the guest's firewall didn't also add an explicit rule for the database. At this point I don't know if that's trove, nova or neutron. If I had to guess I'd say that there is a mismatch in the trove configuration and the networks assigned to the guest. For example how does --- network_driver = trove.network.neutron.NeutronDriver # provider management_networks = 98b0de37-40af-4859-81f2-d89a4df2d6df # trovemgmt # provider ## management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # trove-mgmt 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 management_security_groups = 26c4af6e-56f0-4644-8c55-e96983c74c1d --- relate to the "dozenet" and "trovemgmt" networks. Tony. ------------------------------ Message: 2 Date: Sun, 31 Dec 2023 11:27:37 -0000 From: dieter.dorra@hsnr.de Subject: Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works To: openstack-discuss@lists.openstack.org Message-ID: <170402205718.409.7027996557330367167@lists01.opendev.org> Content-Type: text/plain; charset="utf-8" The problem is solved! I tested a newer trove image from https://tarballs.opendev.org/openstack/trove/images/ The Victoria image caused the problem, it never showed the exposed ports in "docker ps". The Zed image is working: ubuntu@dd:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1741ae3e7815 mysql:5.7.29 "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp database ubuntu@dd:~$ Thank you all so much for your assistance and help!!! Happy new year!! ------------------------------ Message: 3 Date: Sun, 31 Dec 2023 16:49:39 -0000 From: dieter.dorra@hsnr.de Subject: Re: Trove installation filed To: openstack-discuss@lists.openstack.org Message-ID: <170404137950.409.18389774625483655904@lists01.opendev.org> Content-Type: text/plain; charset="utf-8" Hi, when you log in as user trove in the service project - can you launch an instance? "no valid host was found" sounds that nova has a problem... BR Dieter ------------------------------ Message: 4 Date: Mon, 01 Jan 2024 05:40:46 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com> Subject: Re: Trove installation filed To: openstack-discuss@lists.openstack.org Message-ID: <170408764655.409.16151116383780837301@lists01.opendev.org> Content-Type: text/plain; charset="utf-8" Please provide and check the Trove guest agent log file (by default, /var/log/trove/trove-guestagent.log) for any errors or warnings.. Also share with us the logs output that we can take a look. By the way, dont forget to check this following link to troubleshoot; https://docs.openstack.org/trove/latest/admin/troubleshooting.html Kerem Çeliker ------------------------------ Message: 5 Date: Mon, 01 Jan 2024 05:46:32 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com> Subject: Re: [kolla] ceilometer and nova compute agent To: openstack-discuss@lists.openstack.org Message-ID: <170408799208.407.10760598762569499228@lists01.opendev.org> Content-Type: text/plain; charset="utf-8" You dont need to create a custom nova container to measure instances metrics. Only can use the “openstack host show” and “openstack server list” to get basic stats on resource usage for hosts and instances. Also use tools, such as Ganglia or Graphite, to gather more detailed data. It’s up to you! Kerem Çeliker ------------------------------ Message: 6 Date: Mon, 01 Jan 2024 06:07:02 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com> Subject: Re: ceilometer cli client install help To: openstack-discuss@lists.openstack.org Message-ID: <170408922210.407.17546497674894904858@lists01.opendev.org> Content-Type: text/plain; charset="utf-8" It seems that you have not gnocchi plugin on OSP All-In-One. Also your OSP has not correct openstack client version or you needed the install the gnocchi plugin correctly. Please check gnocchi plugin if installed.. Here is the cli’s ; • openstack plugin list and check for gnocchi in the output any issues.. or try it; • pip install python-gnocchiclient. Kerem Çeliker ------------------------------ Message: 7 Date: Tue, 2 Jan 2024 09:52:31 +0100 From: Bernard Cafarelli <bcafarel@redhat.com> Subject: [neutron] Bug deputy report (last week of 2023) To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Message-ID: <CABHdKwov77XQ31h26vq6UvzyOjaANcU1F6zjY1opS_H2XZp=PQ@mail.gmail.com> Content-Type: multipart/alternative; boundary="0000000000001b8f3a060df29f3c" Hey neutrinos, and happy new year! Here is the bug reported between 2023-12-25 and 2023-12-31 (as per tradition, it was a quiet week) The firewall group is ACTIVE without associate with a router or any firewall policies - https://bugs.launchpad.net/neutron/+bug/2047325 neutron-fwaas bug, patch proposed: https://review.opendev.org/c/openstack/neutron-fwaas/+/904317 Best wishes -- Bernard Cafarelli
Hello Asma: Please report the backend you are using (OVS, OVN, Linux Bridge, etc). With this information, we can start debugging the issue. Regards. On Wed, Jan 3, 2024 at 3:14 PM Asma Naz Shariq <asma.naz@techavenue.biz> wrote:
Hi Neutron_ Team
I have deployed Multinode cluster of Openstack through kolla-ansible zed release. The problem I am facing is instances created cannot pick any Ip addresses (Rather than loop back ip or ipv6 addresses) from the created tenant networks with the default network type is vxlan. However, if I can assign ip manually to the eth0 interface in cirros instance, the interface is up with the ip address, but still, it can’t able to ping even its router gateway. Router can’t be able to ping instance ip address but only ping itself as verified below:
The following troubleshooting steps, I have followed and concluded: ~ The Tenant network is created and its port: network: router_interface, network:dhcp, compute:nova are in admin up and active state. ~ The router interfaces are in up state and in complete nova zone. ~ The instance is running and active state and automatically pick ip address from the created tenant network as displayed in gui but inside the vm it can’t pick any ip address. ~ The security groups have been defined to allow icmp, tcp, udp for inbound/outbound traffic. Hence, I may not be able to ping and ssh instance from the controller node further these instances won’t be able to connect to the internet until the issue get resolved.
Can you guide me how to resolve these issues related to network connectivity so I may able to send you the required files to troubleshoot easily?
-----Original Message----- From: openstack-discuss-request@lists.openstack.org < openstack-discuss-request@lists.openstack.org> Sent: Tuesday, January 2, 2024 7:39 PM To: openstack-discuss@lists.openstack.org Subject: openstack-discuss Digest, Vol 62, Issue 108
Send openstack-discuss mailing list submissions to openstack-discuss@lists.openstack.org
To subscribe or unsubscribe via email, send a message with subject or body 'help' to openstack-discuss-request@lists.openstack.org
You can reach the person managing the list at openstack-discuss-owner@lists.openstack.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..."
Today's Topics:
1. Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works (Tony Breeds) 2. Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works (dieter.dorra@hsnr.de) 3. Re: Trove installation filed (dieter.dorra@hsnr.de) 4. Re: Trove installation filed (KEREM CELIKER) 5. Re: [kolla] ceilometer and nova compute agent (KEREM CELIKER) 6. Re: ceilometer cli client install help (KEREM CELIKER) 7. [neutron] Bug deputy report (last week of 2023) (Bernard Cafarelli) 8. New OpenStack machine on new node not responding (kjme001@gmail.com) 9. Re: New OpenStack machine on new node not responding ( 韩光宇 )
----------------------------------------------------------------------
Message: 1 Date: Fri, 29 Dec 2023 23:28:42 +1100 From: Tony Breeds <tony@bakeyournoodle.com> Subject: Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works To: dieter.dorra@hsnr.de Cc: openstack-discuss@lists.openstack.org Message-ID: <CAFT5D2RzaXi6== waA732R0PVe6H0OeKBwzF_qUSOTHPVfCu7Ww@mail.gmail.com> Content-Type: text/plain; charset="UTF-8"
On Fri, 29 Dec 2023 at 19:58, <dieter.dorra@hsnr.de> wrote:
From the guest vm I can ssh into the trove instance (so port 22 is open and accepting) From the guest vm I cannot reach port 3306 From within the trove instance I can reach port 3306 and mysql is functional
It feels like the security group does not allow 3306, but the entry is there (see etherpad)
What puzzles me is that netstat in the trove instance does only show listen to 3306 for tcp6 but not for tcp - whereas 22 listens for tcp6 AND tcp
root@ms57plain:~# netstat -anp | head -17 Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 3867/systemd-resolv tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1074/sshd tcp 0 0 10.9.9.126:22 10.9.9.128:60126 ESTABLISHED 9412/sshd: root@pts tcp 0 0 10.10.10.84:54326 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp 0 0 10.9.9.126:22 10.9.9.128:57880 ESTABLISHED 9021/sshd: root@pts tcp 0 0 10.10.10.84:44266 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp6 0 0 :::3306 :::* LISTEN 4881/mysqld
You can see here that mysql is listening on all interfaces via an IPv4-mapped IPv6 address ( https://en.wikipedia.org/wiki/IPv6#IPv4-mapped_IPv6_addresses), and from your previous post we can see that there is an explicit ACCEPT for the lo interface. That's how you're able to get access to the DB from within the guest.
It looks to me that whatever added the ' -A openstack-INPUT -p tcp -m tcp --dport 22 -j ACCEPT' to the guest's firewall didn't also add an explicit rule for the database. At this point I don't know if that's trove, nova or neutron. If I had to guess I'd say that there is a mismatch in the trove configuration and the networks assigned to the guest.
For example how does --- network_driver = trove.network.neutron.NeutronDriver # provider management_networks = 98b0de37-40af-4859-81f2-d89a4df2d6df # trovemgmt # provider ## management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # trove-mgmt 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 management_security_groups = 26c4af6e-56f0-4644-8c55-e96983c74c1d ---
relate to the "dozenet" and "trovemgmt" networks.
Tony.
------------------------------
Message: 2 Date: Sun, 31 Dec 2023 11:27:37 -0000 From: dieter.dorra@hsnr.de Subject: Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works To: openstack-discuss@lists.openstack.org Message-ID: <170402205718.409.7027996557330367167@lists01.opendev.org> Content-Type: text/plain; charset="utf-8"
The problem is solved! I tested a newer trove image from https://tarballs.opendev.org/openstack/trove/images/ The Victoria image caused the problem, it never showed the exposed ports in "docker ps". The Zed image is working:
ubuntu@dd:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1741ae3e7815 mysql:5.7.29 "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp database ubuntu@dd:~$
Thank you all so much for your assistance and help!!! Happy new year!!
------------------------------
Message: 3 Date: Sun, 31 Dec 2023 16:49:39 -0000 From: dieter.dorra@hsnr.de Subject: Re: Trove installation filed To: openstack-discuss@lists.openstack.org Message-ID: <170404137950.409.18389774625483655904@lists01.opendev.org> Content-Type: text/plain; charset="utf-8"
Hi, when you log in as user trove in the service project - can you launch an instance? "no valid host was found" sounds that nova has a problem... BR Dieter
------------------------------
Message: 4 Date: Mon, 01 Jan 2024 05:40:46 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com> Subject: Re: Trove installation filed To: openstack-discuss@lists.openstack.org Message-ID: <170408764655.409.16151116383780837301@lists01.opendev.org> Content-Type: text/plain; charset="utf-8"
Please provide and check the Trove guest agent log file (by default, /var/log/trove/trove-guestagent.log) for any errors or warnings..
Also share with us the logs output that we can take a look.
By the way, dont forget to check this following link to troubleshoot;
https://docs.openstack.org/trove/latest/admin/troubleshooting.html
Kerem Çeliker
------------------------------
Message: 5 Date: Mon, 01 Jan 2024 05:46:32 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com> Subject: Re: [kolla] ceilometer and nova compute agent To: openstack-discuss@lists.openstack.org Message-ID: <170408799208.407.10760598762569499228@lists01.opendev.org> Content-Type: text/plain; charset="utf-8"
You dont need to create a custom nova container to measure instances metrics. Only can use the “openstack host show” and “openstack server list” to get basic stats on resource usage for hosts and instances.
Also use tools, such as Ganglia or Graphite, to gather more detailed data. It’s up to you!
Kerem Çeliker
------------------------------
Message: 6 Date: Mon, 01 Jan 2024 06:07:02 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com> Subject: Re: ceilometer cli client install help To: openstack-discuss@lists.openstack.org Message-ID: <170408922210.407.17546497674894904858@lists01.opendev.org> Content-Type: text/plain; charset="utf-8"
It seems that you have not gnocchi plugin on OSP All-In-One. Also your OSP has not correct openstack client version or you needed the install the gnocchi plugin correctly.
Please check gnocchi plugin if installed.. Here is the cli’s ; • openstack plugin list and check for gnocchi in the output any issues.. or try it; • pip install python-gnocchiclient.
Kerem Çeliker
------------------------------
Message: 7 Date: Tue, 2 Jan 2024 09:52:31 +0100 From: Bernard Cafarelli <bcafarel@redhat.com> Subject: [neutron] Bug deputy report (last week of 2023) To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Message-ID: <CABHdKwov77XQ31h26vq6UvzyOjaANcU1F6zjY1opS_H2XZp= PQ@mail.gmail.com> Content-Type: multipart/alternative; boundary="0000000000001b8f3a060df29f3c"
Hey neutrinos, and happy new year! Here is the bug reported between 2023-12-25 and 2023-12-31 (as per tradition, it was a quiet week)
The firewall group is ACTIVE without associate with a router or any firewall policies - https://bugs.launchpad.net/neutron/+bug/2047325 neutron-fwaas <https://bugs.launchpad.net/neutron/+bug/2047325neutron-fwaas> bug, patch proposed: https://review.opendev.org/c/openstack/neutron-fwaas/+/904317
Best wishes -- Bernard Cafarelli
Hello Rodolfo, The network driver is OVS. Please guide to debug the issue. thanks From: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> Sent: Thursday, January 4, 2024 12:29 PM To: Asma Naz Shariq <asma.naz@techavenue.biz> Cc: openstack-discuss@lists.openstack.org Subject: Re: openstack-discuss Digest, Vol 62, Issue 108- Neutron Issue | Urgent Assistance Hello Asma: Please report the backend you are using (OVS, OVN, Linux Bridge, etc). With this information, we can start debugging the issue. Regards. On Wed, Jan 3, 2024 at 3:14 PM Asma Naz Shariq <asma.naz@techavenue.biz <mailto:asma.naz@techavenue.biz> > wrote: Hi Neutron_ Team I have deployed Multinode cluster of Openstack through kolla-ansible zed release. The problem I am facing is instances created cannot pick any Ip addresses (Rather than loop back ip or ipv6 addresses) from the created tenant networks with the default network type is vxlan. However, if I can assign ip manually to the eth0 interface in cirros instance, the interface is up with the ip address, but still, it can’t able to ping even its router gateway. Router can’t be able to ping instance ip address but only ping itself as verified below: The following troubleshooting steps, I have followed and concluded: ~ The Tenant network is created and its port: network: router_interface, network:dhcp, compute:nova are in admin up and active state. ~ The router interfaces are in up state and in complete nova zone. ~ The instance is running and active state and automatically pick ip address from the created tenant network as displayed in gui but inside the vm it can’t pick any ip address. ~ The security groups have been defined to allow icmp, tcp, udp for inbound/outbound traffic. Hence, I may not be able to ping and ssh instance from the controller node further these instances won’t be able to connect to the internet until the issue get resolved. Can you guide me how to resolve these issues related to network connectivity so I may able to send you the required files to troubleshoot easily? -----Original Message----- From: openstack-discuss-request@lists.openstack.org <mailto:openstack-discuss-request@lists.openstack.org> <openstack-discuss-request@lists.openstack.org <mailto:openstack-discuss-request@lists.openstack.org> > Sent: Tuesday, January 2, 2024 7:39 PM To: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> Subject: openstack-discuss Digest, Vol 62, Issue 108 Send openstack-discuss mailing list submissions to openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> To subscribe or unsubscribe via email, send a message with subject or body 'help' to openstack-discuss-request@lists.openstack.org <mailto:openstack-discuss-request@lists.openstack.org> You can reach the person managing the list at openstack-discuss-owner@lists.openstack.org <mailto:openstack-discuss-owner@lists.openstack.org> When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works (Tony Breeds) 2. Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works (dieter.dorra@hsnr.de <mailto:dieter.dorra@hsnr.de> ) 3. Re: Trove installation filed (dieter.dorra@hsnr.de <mailto:dieter.dorra@hsnr.de> ) 4. Re: Trove installation filed (KEREM CELIKER) 5. Re: [kolla] ceilometer and nova compute agent (KEREM CELIKER) 6. Re: ceilometer cli client install help (KEREM CELIKER) 7. [neutron] Bug deputy report (last week of 2023) (Bernard Cafarelli) 8. New OpenStack machine on new node not responding (kjme001@gmail.com <mailto:kjme001@gmail.com> ) 9. Re: New OpenStack machine on new node not responding ( 韩光宇 ) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 Dec 2023 23:28:42 +1100 From: Tony Breeds <tony@bakeyournoodle.com <mailto:tony@bakeyournoodle.com> > Subject: Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works To: dieter.dorra@hsnr.de <mailto:dieter.dorra@hsnr.de> Cc: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> Message-ID: <CAFT5D2RzaXi6==waA732R0PVe6H0OeKBwzF_qUSOTHPVfCu7Ww@mail.gmail.com <mailto:waA732R0PVe6H0OeKBwzF_qUSOTHPVfCu7Ww@mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" On Fri, 29 Dec 2023 at 19:58, <dieter.dorra@hsnr.de <mailto:dieter.dorra@hsnr.de> > wrote:
From the guest vm I can ssh into the trove instance (so port 22 is open and accepting) From the guest vm I cannot reach port 3306 From within the trove instance I can reach port 3306 and mysql is functional
It feels like the security group does not allow 3306, but the entry is there (see etherpad)
What puzzles me is that netstat in the trove instance does only show listen to 3306 for tcp6 but not for tcp - whereas 22 listens for tcp6 AND tcp
root@ms57plain:~# netstat -anp | head -17 Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.53:53 <http://127.0.0.53:53> 0.0.0.0:* LISTEN 3867/systemd-resolv tcp 0 0 0.0.0.0:22 <http://0.0.0.0:22> 0.0.0.0:* LISTEN 1074/sshd tcp 0 0 10.9.9.126:22 <http://10.9.9.126:22> 10.9.9.128:60126 <http://10.9.9.128:60126> ESTABLISHED 9412/sshd: root@pts tcp 0 0 10.10.10.84:54326 <http://10.10.10.84:54326> 10.0.0.120:5672 <http://10.0.0.120:5672> ESTABLISHED 1002/python3 tcp 0 0 10.9.9.126:22 <http://10.9.9.126:22> 10.9.9.128:57880 <http://10.9.9.128:57880> ESTABLISHED 9021/sshd: root@pts tcp 0 0 10.10.10.84:44266 <http://10.10.10.84:44266> 10.0.0.120:5672 <http://10.0.0.120:5672> ESTABLISHED 1002/python3 tcp6 0 0 :::3306 :::* LISTEN 4881/mysqld
You can see here that mysql is listening on all interfaces via an IPv4-mapped IPv6 address (https://en.wikipedia.org/wiki/IPv6#IPv4-mapped_IPv6_addresses), and from your previous post we can see that there is an explicit ACCEPT for the lo interface. That's how you're able to get access to the DB from within the guest. It looks to me that whatever added the ' -A openstack-INPUT -p tcp -m tcp --dport 22 -j ACCEPT' to the guest's firewall didn't also add an explicit rule for the database. At this point I don't know if that's trove, nova or neutron. If I had to guess I'd say that there is a mismatch in the trove configuration and the networks assigned to the guest. For example how does --- network_driver = trove.network.neutron.NeutronDriver # provider management_networks = 98b0de37-40af-4859-81f2-d89a4df2d6df # trovemgmt # provider ## management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # trove-mgmt 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 management_security_groups = 26c4af6e-56f0-4644-8c55-e96983c74c1d --- relate to the "dozenet" and "trovemgmt" networks. Tony. ------------------------------ Message: 2 Date: Sun, 31 Dec 2023 11:27:37 -0000 From: dieter.dorra@hsnr.de <mailto:dieter.dorra@hsnr.de> Subject: Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works To: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> Message-ID: <170402205718.409.7027996557330367167@lists01.opendev.org <mailto:170402205718.409.7027996557330367167@lists01.opendev.org> > Content-Type: text/plain; charset="utf-8" The problem is solved! I tested a newer trove image from https://tarballs.opendev.org/openstack/trove/images/ The Victoria image caused the problem, it never showed the exposed ports in "docker ps". The Zed image is working: ubuntu@dd:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1741ae3e7815 mysql:5.7.29 "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp database ubuntu@dd:~$ Thank you all so much for your assistance and help!!! Happy new year!! ------------------------------ Message: 3 Date: Sun, 31 Dec 2023 16:49:39 -0000 From: dieter.dorra@hsnr.de <mailto:dieter.dorra@hsnr.de> Subject: Re: Trove installation filed To: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> Message-ID: <170404137950.409.18389774625483655904@lists01.opendev.org <mailto:170404137950.409.18389774625483655904@lists01.opendev.org> > Content-Type: text/plain; charset="utf-8" Hi, when you log in as user trove in the service project - can you launch an instance? "no valid host was found" sounds that nova has a problem... BR Dieter ------------------------------ Message: 4 Date: Mon, 01 Jan 2024 05:40:46 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com <mailto:kmceliker@gmail.com> > Subject: Re: Trove installation filed To: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> Message-ID: <170408764655.409.16151116383780837301@lists01.opendev.org <mailto:170408764655.409.16151116383780837301@lists01.opendev.org> > Content-Type: text/plain; charset="utf-8" Please provide and check the Trove guest agent log file (by default, /var/log/trove/trove-guestagent.log) for any errors or warnings.. Also share with us the logs output that we can take a look. By the way, dont forget to check this following link to troubleshoot; https://docs.openstack.org/trove/latest/admin/troubleshooting.html Kerem Çeliker ------------------------------ Message: 5 Date: Mon, 01 Jan 2024 05:46:32 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com <mailto:kmceliker@gmail.com> > Subject: Re: [kolla] ceilometer and nova compute agent To: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> Message-ID: <170408799208.407.10760598762569499228@lists01.opendev.org <mailto:170408799208.407.10760598762569499228@lists01.opendev.org> > Content-Type: text/plain; charset="utf-8" You dont need to create a custom nova container to measure instances metrics. Only can use the “openstack host show” and “openstack server list” to get basic stats on resource usage for hosts and instances. Also use tools, such as Ganglia or Graphite, to gather more detailed data. It’s up to you! Kerem Çeliker ------------------------------ Message: 6 Date: Mon, 01 Jan 2024 06:07:02 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com <mailto:kmceliker@gmail.com> > Subject: Re: ceilometer cli client install help To: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> Message-ID: <170408922210.407.17546497674894904858@lists01.opendev.org <mailto:170408922210.407.17546497674894904858@lists01.opendev.org> > Content-Type: text/plain; charset="utf-8" It seems that you have not gnocchi plugin on OSP All-In-One. Also your OSP has not correct openstack client version or you needed the install the gnocchi plugin correctly. Please check gnocchi plugin if installed.. Here is the cli’s ; • openstack plugin list and check for gnocchi in the output any issues.. or try it; • pip install python-gnocchiclient. Kerem Çeliker ------------------------------ Message: 7 Date: Tue, 2 Jan 2024 09:52:31 +0100 From: Bernard Cafarelli <bcafarel@redhat.com <mailto:bcafarel@redhat.com> > Subject: [neutron] Bug deputy report (last week of 2023) To: OpenStack Discuss <openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> > Message-ID: <CABHdKwov77XQ31h26vq6UvzyOjaANcU1F6zjY1opS_H2XZp=PQ@mail.gmail.com <mailto:PQ@mail.gmail.com> > Content-Type: multipart/alternative; boundary="0000000000001b8f3a060df29f3c" Hey neutrinos, and happy new year! Here is the bug reported between 2023-12-25 and 2023-12-31 (as per tradition, it was a quiet week) The firewall group is ACTIVE without associate with a router or any firewall policies - https://bugs.launchpad.net/neutron/+bug/2047325 <https://bugs.launchpad.net/neutron/+bug/2047325neutron-fwaas> neutron-fwaas bug, patch proposed: https://review.opendev.org/c/openstack/neutron-fwaas/+/904317 Best wishes -- Bernard Cafarelli
Hello Asma: The first thing you need to check is the VM cloud init logs, to check if the VM is sending the DHCP discover and request messages. You should also check the DHCP agent: where is located (you can also have HA - multiple DHCP agents attending to the same network) and if the node where the DHCP agent is located have connectivity to the compute node. Please check the DHCP agent logs in order to check if there is any error. You should also check if a dnsmasq process is spawned for the related network (you can grep using the network ID). And in general, if you have a networking problem, you should trace the packets. You should be able to track the DHCP messages from the VM TAP port, then going to OVS integration bridge, then to the tunnel bridge and to the VTEP interface for VXLAN. At this point, you should do the same in the node with the DHCP agent. With these quick steps, you can start debugging the issue you have. Regards. On Thu, Jan 4, 2024 at 8:32 AM Asma Naz Shariq <asma.naz@techavenue.biz> wrote:
Hello Rodolfo,
The network driver is OVS. Please guide to debug the issue.
thanks
*From:* Rodolfo Alonso Hernandez <ralonsoh@redhat.com> *Sent:* Thursday, January 4, 2024 12:29 PM *To:* Asma Naz Shariq <asma.naz@techavenue.biz> *Cc:* openstack-discuss@lists.openstack.org *Subject:* Re: openstack-discuss Digest, Vol 62, Issue 108- Neutron Issue | Urgent Assistance
Hello Asma:
Please report the backend you are using (OVS, OVN, Linux Bridge, etc). With this information, we can start debugging the issue.
Regards.
On Wed, Jan 3, 2024 at 3:14 PM Asma Naz Shariq <asma.naz@techavenue.biz> wrote:
Hi Neutron_ Team
I have deployed Multinode cluster of Openstack through kolla-ansible zed release. The problem I am facing is instances created cannot pick any Ip addresses (Rather than loop back ip or ipv6 addresses) from the created tenant networks with the default network type is vxlan. However, if I can assign ip manually to the eth0 interface in cirros instance, the interface is up with the ip address, but still, it can’t able to ping even its router gateway. Router can’t be able to ping instance ip address but only ping itself as verified below:
The following troubleshooting steps, I have followed and concluded: ~ The Tenant network is created and its port: network: router_interface, network:dhcp, compute:nova are in admin up and active state. ~ The router interfaces are in up state and in complete nova zone. ~ The instance is running and active state and automatically pick ip address from the created tenant network as displayed in gui but inside the vm it can’t pick any ip address. ~ The security groups have been defined to allow icmp, tcp, udp for inbound/outbound traffic. Hence, I may not be able to ping and ssh instance from the controller node further these instances won’t be able to connect to the internet until the issue get resolved.
Can you guide me how to resolve these issues related to network connectivity so I may able to send you the required files to troubleshoot easily?
-----Original Message----- From: openstack-discuss-request@lists.openstack.org < openstack-discuss-request@lists.openstack.org> Sent: Tuesday, January 2, 2024 7:39 PM To: openstack-discuss@lists.openstack.org Subject: openstack-discuss Digest, Vol 62, Issue 108
Send openstack-discuss mailing list submissions to openstack-discuss@lists.openstack.org
To subscribe or unsubscribe via email, send a message with subject or body 'help' to openstack-discuss-request@lists.openstack.org
You can reach the person managing the list at openstack-discuss-owner@lists.openstack.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..."
Today's Topics:
1. Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works (Tony Breeds) 2. Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works (dieter.dorra@hsnr.de) 3. Re: Trove installation filed (dieter.dorra@hsnr.de) 4. Re: Trove installation filed (KEREM CELIKER) 5. Re: [kolla] ceilometer and nova compute agent (KEREM CELIKER) 6. Re: ceilometer cli client install help (KEREM CELIKER) 7. [neutron] Bug deputy report (last week of 2023) (Bernard Cafarelli) 8. New OpenStack machine on new node not responding (kjme001@gmail.com) 9. Re: New OpenStack machine on new node not responding ( 韩光宇 )
----------------------------------------------------------------------
Message: 1 Date: Fri, 29 Dec 2023 23:28:42 +1100 From: Tony Breeds <tony@bakeyournoodle.com> Subject: Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works To: dieter.dorra@hsnr.de Cc: openstack-discuss@lists.openstack.org Message-ID: <CAFT5D2RzaXi6== waA732R0PVe6H0OeKBwzF_qUSOTHPVfCu7Ww@mail.gmail.com> Content-Type: text/plain; charset="UTF-8"
On Fri, 29 Dec 2023 at 19:58, <dieter.dorra@hsnr.de> wrote:
From the guest vm I can ssh into the trove instance (so port 22 is open and accepting) From the guest vm I cannot reach port 3306 From within the trove instance I can reach port 3306 and mysql is functional
It feels like the security group does not allow 3306, but the entry is there (see etherpad)
What puzzles me is that netstat in the trove instance does only show listen to 3306 for tcp6 but not for tcp - whereas 22 listens for tcp6 AND tcp
root@ms57plain:~# netstat -anp | head -17 Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 3867/systemd-resolv tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1074/sshd tcp 0 0 10.9.9.126:22 10.9.9.128:60126 ESTABLISHED 9412/sshd: root@pts tcp 0 0 10.10.10.84:54326 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp 0 0 10.9.9.126:22 10.9.9.128:57880 ESTABLISHED 9021/sshd: root@pts tcp 0 0 10.10.10.84:44266 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp6 0 0 :::3306 :::* LISTEN 4881/mysqld
You can see here that mysql is listening on all interfaces via an IPv4-mapped IPv6 address ( https://en.wikipedia.org/wiki/IPv6#IPv4-mapped_IPv6_addresses), and from your previous post we can see that there is an explicit ACCEPT for the lo interface. That's how you're able to get access to the DB from within the guest.
It looks to me that whatever added the ' -A openstack-INPUT -p tcp -m tcp --dport 22 -j ACCEPT' to the guest's firewall didn't also add an explicit rule for the database. At this point I don't know if that's trove, nova or neutron. If I had to guess I'd say that there is a mismatch in the trove configuration and the networks assigned to the guest.
For example how does --- network_driver = trove.network.neutron.NeutronDriver # provider management_networks = 98b0de37-40af-4859-81f2-d89a4df2d6df # trovemgmt # provider ## management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # trove-mgmt 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 management_security_groups = 26c4af6e-56f0-4644-8c55-e96983c74c1d ---
relate to the "dozenet" and "trovemgmt" networks.
Tony.
------------------------------
Message: 2 Date: Sun, 31 Dec 2023 11:27:37 -0000 From: dieter.dorra@hsnr.de Subject: Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works To: openstack-discuss@lists.openstack.org Message-ID: <170402205718.409.7027996557330367167@lists01.opendev.org> Content-Type: text/plain; charset="utf-8"
The problem is solved! I tested a newer trove image from https://tarballs.opendev.org/openstack/trove/images/ The Victoria image caused the problem, it never showed the exposed ports in "docker ps". The Zed image is working:
ubuntu@dd:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1741ae3e7815 mysql:5.7.29 "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp database ubuntu@dd:~$
Thank you all so much for your assistance and help!!! Happy new year!!
------------------------------
Message: 3 Date: Sun, 31 Dec 2023 16:49:39 -0000 From: dieter.dorra@hsnr.de Subject: Re: Trove installation filed To: openstack-discuss@lists.openstack.org Message-ID: <170404137950.409.18389774625483655904@lists01.opendev.org> Content-Type: text/plain; charset="utf-8"
Hi, when you log in as user trove in the service project - can you launch an instance? "no valid host was found" sounds that nova has a problem... BR Dieter
------------------------------
Message: 4 Date: Mon, 01 Jan 2024 05:40:46 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com> Subject: Re: Trove installation filed To: openstack-discuss@lists.openstack.org Message-ID: <170408764655.409.16151116383780837301@lists01.opendev.org> Content-Type: text/plain; charset="utf-8"
Please provide and check the Trove guest agent log file (by default, /var/log/trove/trove-guestagent.log) for any errors or warnings..
Also share with us the logs output that we can take a look.
By the way, dont forget to check this following link to troubleshoot;
https://docs.openstack.org/trove/latest/admin/troubleshooting.html
Kerem Çeliker
------------------------------
Message: 5 Date: Mon, 01 Jan 2024 05:46:32 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com> Subject: Re: [kolla] ceilometer and nova compute agent To: openstack-discuss@lists.openstack.org Message-ID: <170408799208.407.10760598762569499228@lists01.opendev.org> Content-Type: text/plain; charset="utf-8"
You dont need to create a custom nova container to measure instances metrics. Only can use the “openstack host show” and “openstack server list” to get basic stats on resource usage for hosts and instances.
Also use tools, such as Ganglia or Graphite, to gather more detailed data. It’s up to you!
Kerem Çeliker
------------------------------
Message: 6 Date: Mon, 01 Jan 2024 06:07:02 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com> Subject: Re: ceilometer cli client install help To: openstack-discuss@lists.openstack.org Message-ID: <170408922210.407.17546497674894904858@lists01.opendev.org> Content-Type: text/plain; charset="utf-8"
It seems that you have not gnocchi plugin on OSP All-In-One. Also your OSP has not correct openstack client version or you needed the install the gnocchi plugin correctly.
Please check gnocchi plugin if installed.. Here is the cli’s ; • openstack plugin list and check for gnocchi in the output any issues.. or try it; • pip install python-gnocchiclient.
Kerem Çeliker
------------------------------
Message: 7 Date: Tue, 2 Jan 2024 09:52:31 +0100 From: Bernard Cafarelli <bcafarel@redhat.com> Subject: [neutron] Bug deputy report (last week of 2023) To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Message-ID: <CABHdKwov77XQ31h26vq6UvzyOjaANcU1F6zjY1opS_H2XZp= PQ@mail.gmail.com> Content-Type: multipart/alternative; boundary="0000000000001b8f3a060df29f3c"
Hey neutrinos, and happy new year! Here is the bug reported between 2023-12-25 and 2023-12-31 (as per tradition, it was a quiet week)
The firewall group is ACTIVE without associate with a router or any firewall policies - https://bugs.launchpad.net/neutron/+bug/2047325 neutron-fwaas bug, patch proposed: https://review.opendev.org/c/openstack/neutron-fwaas/+/904317
Best wishes -- Bernard Cafarelli
Hi Rodolfo, Thankyou. Please see the response in yellow in previous thread. From: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> Sent: Thursday, January 4, 2024 3:36 PM To: Asma Naz Shariq <asma.naz@techavenue.biz> Cc: openstack-discuss@lists.openstack.org Subject: Re: openstack-discuss Digest, Vol 62, Issue 108- Neutron Issue | Urgent Assistance Hello Asma: The first thing you need to check is the VM cloud init logs, to check if the VM is sending the DHCP discover and request messages. You should also check the DHCP agent: where is located (you can also have HA - multiple DHCP agents attending to the same network) and if the node where the DHCP agent is located have connectivity to the compute node. DHCP agent is located at controller node and all these nodes are connected with each other verified through ansible connectivity by ansible -i multinode all -m ping Please check the DHCP agent logs in order to check if there is any error. You should also check if a dnsmasq process is spawned for the related network (you can grep using the network ID). Jan 4 16:01:04 dnsmasq-dhcp[171]: DHCPDISCOVER(tapf3d981d8-bc) X.X.X.X b0:4e:26:79:1c:9e no address available. And in general, if you have a networking problem, you should trace the packets. You should be able to track the DHCP messages from the VM TAP port, then going to OVS integration bridge, then to the tunnel bridge and to the VTEP interface for VXLAN. At this point, you should do the same in the node with the DHCP agent. From the VM TAP, I can see 0 packets captures, 0 packets received by kernel, With these quick steps, you can start debugging the issue you have. Regards. On Thu, Jan 4, 2024 at 8:32 AM Asma Naz Shariq <asma.naz@techavenue.biz <mailto:asma.naz@techavenue.biz> > wrote: Hello Rodolfo, The network driver is OVS. Please guide to debug the issue. thanks From: Rodolfo Alonso Hernandez <ralonsoh@redhat.com <mailto:ralonsoh@redhat.com> > Sent: Thursday, January 4, 2024 12:29 PM To: Asma Naz Shariq <asma.naz@techavenue.biz <mailto:asma.naz@techavenue.biz> > Cc: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> Subject: Re: openstack-discuss Digest, Vol 62, Issue 108- Neutron Issue | Urgent Assistance Hello Asma: Please report the backend you are using (OVS, OVN, Linux Bridge, etc). With this information, we can start debugging the issue. Regards. On Wed, Jan 3, 2024 at 3:14 PM Asma Naz Shariq <asma.naz@techavenue.biz <mailto:asma.naz@techavenue.biz> > wrote: Hi Neutron_ Team I have deployed Multinode cluster of Openstack through kolla-ansible zed release. The problem I am facing is instances created cannot pick any Ip addresses (Rather than loop back ip or ipv6 addresses) from the created tenant networks with the default network type is vxlan. However, if I can assign ip manually to the eth0 interface in cirros instance, the interface is up with the ip address, but still, it can’t able to ping even its router gateway. Router can’t be able to ping instance ip address but only ping itself as verified below: The following troubleshooting steps, I have followed and concluded: ~ The Tenant network is created and its port: network: router_interface, network:dhcp, compute:nova are in admin up and active state. ~ The router interfaces are in up state and in complete nova zone. ~ The instance is running and active state and automatically pick ip address from the created tenant network as displayed in gui but inside the vm it can’t pick any ip address. ~ The security groups have been defined to allow icmp, tcp, udp for inbound/outbound traffic. Hence, I may not be able to ping and ssh instance from the controller node further these instances won’t be able to connect to the internet until the issue get resolved. Can you guide me how to resolve these issues related to network connectivity so I may able to send you the required files to troubleshoot easily? -----Original Message----- From: openstack-discuss-request@lists.openstack.org <mailto:openstack-discuss-request@lists.openstack.org> <openstack-discuss-request@lists.openstack.org <mailto:openstack-discuss-request@lists.openstack.org> > Sent: Tuesday, January 2, 2024 7:39 PM To: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> Subject: openstack-discuss Digest, Vol 62, Issue 108 Send openstack-discuss mailing list submissions to openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> To subscribe or unsubscribe via email, send a message with subject or body 'help' to openstack-discuss-request@lists.openstack.org <mailto:openstack-discuss-request@lists.openstack.org> You can reach the person managing the list at openstack-discuss-owner@lists.openstack.org <mailto:openstack-discuss-owner@lists.openstack.org> When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works (Tony Breeds) 2. Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works (dieter.dorra@hsnr.de <mailto:dieter.dorra@hsnr.de> ) 3. Re: Trove installation filed (dieter.dorra@hsnr.de <mailto:dieter.dorra@hsnr.de> ) 4. Re: Trove installation filed (KEREM CELIKER) 5. Re: [kolla] ceilometer and nova compute agent (KEREM CELIKER) 6. Re: ceilometer cli client install help (KEREM CELIKER) 7. [neutron] Bug deputy report (last week of 2023) (Bernard Cafarelli) 8. New OpenStack machine on new node not responding (kjme001@gmail.com <mailto:kjme001@gmail.com> ) 9. Re: New OpenStack machine on new node not responding ( 韩光宇 ) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 Dec 2023 23:28:42 +1100 From: Tony Breeds <tony@bakeyournoodle.com <mailto:tony@bakeyournoodle.com> > Subject: Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works To: dieter.dorra@hsnr.de <mailto:dieter.dorra@hsnr.de> Cc: openstack-discuss@lists.openstack.org Message-ID: <CAFT5D2RzaXi6==waA732R0PVe6H0OeKBwzF_qUSOTHPVfCu7Ww@mail.gmail.com <mailto:waA732R0PVe6H0OeKBwzF_qUSOTHPVfCu7Ww@mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" On Fri, 29 Dec 2023 at 19:58, <dieter.dorra@hsnr.de <mailto:dieter.dorra@hsnr.de> > wrote:
From the guest vm I can ssh into the trove instance (so port 22 is open and accepting) From the guest vm I cannot reach port 3306 From within the trove instance I can reach port 3306 and mysql is functional
It feels like the security group does not allow 3306, but the entry is there (see etherpad)
What puzzles me is that netstat in the trove instance does only show listen to 3306 for tcp6 but not for tcp - whereas 22 listens for tcp6 AND tcp
root@ms57plain:~# netstat -anp | head -17 Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.53:53 <http://127.0.0.53:53> 0.0.0.0:* LISTEN 3867/systemd-resolv tcp 0 0 0.0.0.0:22 <http://0.0.0.0:22> 0.0.0.0:* LISTEN 1074/sshd tcp 0 0 10.9.9.126:22 <http://10.9.9.126:22> 10.9.9.128:60126 <http://10.9.9.128:60126> ESTABLISHED 9412/sshd: root@pts tcp 0 0 10.10.10.84:54326 <http://10.10.10.84:54326> 10.0.0.120:5672 <http://10.0.0.120:5672> ESTABLISHED 1002/python3 tcp 0 0 10.9.9.126:22 <http://10.9.9.126:22> 10.9.9.128:57880 <http://10.9.9.128:57880> ESTABLISHED 9021/sshd: root@pts tcp 0 0 10.10.10.84:44266 <http://10.10.10.84:44266> 10.0.0.120:5672 <http://10.0.0.120:5672> ESTABLISHED 1002/python3 tcp6 0 0 :::3306 :::* LISTEN 4881/mysqld
You can see here that mysql is listening on all interfaces via an IPv4-mapped IPv6 address (https://en.wikipedia.org/wiki/IPv6#IPv4-mapped_IPv6_addresses), and from your previous post we can see that there is an explicit ACCEPT for the lo interface. That's how you're able to get access to the DB from within the guest. It looks to me that whatever added the ' -A openstack-INPUT -p tcp -m tcp --dport 22 -j ACCEPT' to the guest's firewall didn't also add an explicit rule for the database. At this point I don't know if that's trove, nova or neutron. If I had to guess I'd say that there is a mismatch in the trove configuration and the networks assigned to the guest. For example how does --- network_driver = trove.network.neutron.NeutronDriver # provider management_networks = 98b0de37-40af-4859-81f2-d89a4df2d6df # trovemgmt # provider ## management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # trove-mgmt 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 management_security_groups = 26c4af6e-56f0-4644-8c55-e96983c74c1d --- relate to the "dozenet" and "trovemgmt" networks. Tony. ------------------------------ Message: 2 Date: Sun, 31 Dec 2023 11:27:37 -0000 From: dieter.dorra@hsnr.de <mailto:dieter.dorra@hsnr.de> Subject: Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works To: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> Message-ID: <170402205718.409.7027996557330367167@lists01.opendev.org <mailto:170402205718.409.7027996557330367167@lists01.opendev.org> > Content-Type: text/plain; charset="utf-8" The problem is solved! I tested a newer trove image from https://tarballs.opendev.org/openstack/trove/images/ The Victoria image caused the problem, it never showed the exposed ports in "docker ps". The Zed image is working: ubuntu@dd:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1741ae3e7815 mysql:5.7.29 "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp database ubuntu@dd:~$ Thank you all so much for your assistance and help!!! Happy new year!! ------------------------------ Message: 3 Date: Sun, 31 Dec 2023 16:49:39 -0000 From: dieter.dorra@hsnr.de <mailto:dieter.dorra@hsnr.de> Subject: Re: Trove installation filed To: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> Message-ID: <170404137950.409.18389774625483655904@lists01.opendev.org <mailto:170404137950.409.18389774625483655904@lists01.opendev.org> > Content-Type: text/plain; charset="utf-8" Hi, when you log in as user trove in the service project - can you launch an instance? "no valid host was found" sounds that nova has a problem... BR Dieter ------------------------------ Message: 4 Date: Mon, 01 Jan 2024 05:40:46 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com <mailto:kmceliker@gmail.com> > Subject: Re: Trove installation filed To: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> Message-ID: <170408764655.409.16151116383780837301@lists01.opendev.org <mailto:170408764655.409.16151116383780837301@lists01.opendev.org> > Content-Type: text/plain; charset="utf-8" Please provide and check the Trove guest agent log file (by default, /var/log/trove/trove-guestagent.log) for any errors or warnings.. Also share with us the logs output that we can take a look. By the way, dont forget to check this following link to troubleshoot; https://docs.openstack.org/trove/latest/admin/troubleshooting.html Kerem Çeliker ------------------------------ Message: 5 Date: Mon, 01 Jan 2024 05:46:32 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com <mailto:kmceliker@gmail.com> > Subject: Re: [kolla] ceilometer and nova compute agent To: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> Message-ID: <170408799208.407.10760598762569499228@lists01.opendev.org <mailto:170408799208.407.10760598762569499228@lists01.opendev.org> > Content-Type: text/plain; charset="utf-8" You dont need to create a custom nova container to measure instances metrics. Only can use the “openstack host show” and “openstack server list” to get basic stats on resource usage for hosts and instances. Also use tools, such as Ganglia or Graphite, to gather more detailed data. It’s up to you! Kerem Çeliker ------------------------------ Message: 6 Date: Mon, 01 Jan 2024 06:07:02 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com <mailto:kmceliker@gmail.com> > Subject: Re: ceilometer cli client install help To: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> Message-ID: <170408922210.407.17546497674894904858@lists01.opendev.org <mailto:170408922210.407.17546497674894904858@lists01.opendev.org> > Content-Type: text/plain; charset="utf-8" It seems that you have not gnocchi plugin on OSP All-In-One. Also your OSP has not correct openstack client version or you needed the install the gnocchi plugin correctly. Please check gnocchi plugin if installed.. Here is the cli’s ; • openstack plugin list and check for gnocchi in the output any issues.. or try it; • pip install python-gnocchiclient. Kerem Çeliker ------------------------------ Message: 7 Date: Tue, 2 Jan 2024 09:52:31 +0100 From: Bernard Cafarelli <bcafarel@redhat.com <mailto:bcafarel@redhat.com> > Subject: [neutron] Bug deputy report (last week of 2023) To: OpenStack Discuss <openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> > Message-ID: <CABHdKwov77XQ31h26vq6UvzyOjaANcU1F6zjY1opS_H2XZp=PQ@mail.gmail.com <mailto:PQ@mail.gmail.com> > Content-Type: multipart/alternative; boundary="0000000000001b8f3a060df29f3c" Hey neutrinos, and happy new year! Here is the bug reported between 2023-12-25 and 2023-12-31 (as per tradition, it was a quiet week) The firewall group is ACTIVE without associate with a router or any firewall policies - https://bugs.launchpad.net/neutron/+bug/2047325 <https://bugs.launchpad.net/neutron/+bug/2047325neutron-fwaas> neutron-fwaas bug, patch proposed: https://review.opendev.org/c/openstack/neutron-fwaas/+/904317 Best wishes -- Bernard Cafarelli
Hello Asma: The connectivity of the DHCP agent should be done not from host to host, but from the DHCP namespace to the, for example, VM TAP port. This is the connectivity you need. You said you don't see any packet coming out from the VM. This is the first thing you need to investigate: why the VM is not sending a DHCP discover. I don't understand your second reply: if the VM is not sending any packet, where is this DHCPDISCOVER coming from? Whos this MAC address? Please provide more details to debug this. Regards. On Thu, Jan 4, 2024 at 1:27 PM Asma Naz Shariq <asma.naz@techavenue.biz> wrote:
Hi Rodolfo, Thankyou.
Please see the response in yellow in previous thread.
*From:* Rodolfo Alonso Hernandez <ralonsoh@redhat.com> *Sent:* Thursday, January 4, 2024 3:36 PM *To:* Asma Naz Shariq <asma.naz@techavenue.biz> *Cc:* openstack-discuss@lists.openstack.org *Subject:* Re: openstack-discuss Digest, Vol 62, Issue 108- Neutron Issue | Urgent Assistance
Hello Asma:
The first thing you need to check is the VM cloud init logs, to check if the VM is sending the DHCP discover and request messages.
You should also check the DHCP agent: where is located (you can also have HA - multiple DHCP agents attending to the same network) and if the node where the DHCP agent is located have connectivity to the compute node. *DHCP agent is located at controller node and all these nodes are connected with each other verified through ansible connectivity by ansible -i multinode all -m ping*
Please check the DHCP agent logs in order to check if there is any error. You should also check if a dnsmasq process is spawned for the related network (you can grep using the network ID). Jan 4 16:01:04 dnsmasq-dhcp[171]: DHCPDISCOVER(tapf3d981d8-bc) X.X.X.X b0:4e:26:79:1c:9e no address available.
And in general, if you have a networking problem, you should trace the packets. You should be able to track the DHCP messages from the VM TAP port, then going to OVS integration bridge, then to the tunnel bridge and to the VTEP interface for VXLAN. At this point, you should do the same in the node with the DHCP agent. From the VM TAP, I can see 0 packets captures, 0 packets received by kernel,
With these quick steps, you can start debugging the issue you have.
Regards.
On Thu, Jan 4, 2024 at 8:32 AM Asma Naz Shariq <asma.naz@techavenue.biz> wrote:
Hello Rodolfo,
The network driver is OVS. Please guide to debug the issue.
thanks
*From:* Rodolfo Alonso Hernandez <ralonsoh@redhat.com> *Sent:* Thursday, January 4, 2024 12:29 PM *To:* Asma Naz Shariq <asma.naz@techavenue.biz> *Cc:* openstack-discuss@lists.openstack.org *Subject:* Re: openstack-discuss Digest, Vol 62, Issue 108- Neutron Issue | Urgent Assistance
Hello Asma:
Please report the backend you are using (OVS, OVN, Linux Bridge, etc). With this information, we can start debugging the issue.
Regards.
On Wed, Jan 3, 2024 at 3:14 PM Asma Naz Shariq <asma.naz@techavenue.biz> wrote:
Hi Neutron_ Team
I have deployed Multinode cluster of Openstack through kolla-ansible zed release. The problem I am facing is instances created cannot pick any Ip addresses (Rather than loop back ip or ipv6 addresses) from the created tenant networks with the default network type is vxlan. However, if I can assign ip manually to the eth0 interface in cirros instance, the interface is up with the ip address, but still, it can’t able to ping even its router gateway. Router can’t be able to ping instance ip address but only ping itself as verified below:
The following troubleshooting steps, I have followed and concluded: ~ The Tenant network is created and its port: network: router_interface, network:dhcp, compute:nova are in admin up and active state. ~ The router interfaces are in up state and in complete nova zone. ~ The instance is running and active state and automatically pick ip address from the created tenant network as displayed in gui but inside the vm it can’t pick any ip address. ~ The security groups have been defined to allow icmp, tcp, udp for inbound/outbound traffic. Hence, I may not be able to ping and ssh instance from the controller node further these instances won’t be able to connect to the internet until the issue get resolved.
Can you guide me how to resolve these issues related to network connectivity so I may able to send you the required files to troubleshoot easily?
-----Original Message----- From: openstack-discuss-request@lists.openstack.org < openstack-discuss-request@lists.openstack.org> Sent: Tuesday, January 2, 2024 7:39 PM To: openstack-discuss@lists.openstack.org Subject: openstack-discuss Digest, Vol 62, Issue 108
Send openstack-discuss mailing list submissions to openstack-discuss@lists.openstack.org
To subscribe or unsubscribe via email, send a message with subject or body 'help' to openstack-discuss-request@lists.openstack.org
You can reach the person managing the list at openstack-discuss-owner@lists.openstack.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..."
Today's Topics:
1. Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works (Tony Breeds) 2. Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works (dieter.dorra@hsnr.de) 3. Re: Trove installation filed (dieter.dorra@hsnr.de) 4. Re: Trove installation filed (KEREM CELIKER) 5. Re: [kolla] ceilometer and nova compute agent (KEREM CELIKER) 6. Re: ceilometer cli client install help (KEREM CELIKER) 7. [neutron] Bug deputy report (last week of 2023) (Bernard Cafarelli) 8. New OpenStack machine on new node not responding (kjme001@gmail.com) 9. Re: New OpenStack machine on new node not responding ( 韩光宇 )
----------------------------------------------------------------------
Message: 1 Date: Fri, 29 Dec 2023 23:28:42 +1100 From: Tony Breeds <tony@bakeyournoodle.com> Subject: Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works To: dieter.dorra@hsnr.de Cc: openstack-discuss@lists.openstack.org Message-ID: <CAFT5D2RzaXi6== waA732R0PVe6H0OeKBwzF_qUSOTHPVfCu7Ww@mail.gmail.com> Content-Type: text/plain; charset="UTF-8"
On Fri, 29 Dec 2023 at 19:58, <dieter.dorra@hsnr.de> wrote:
From the guest vm I can ssh into the trove instance (so port 22 is open and accepting) From the guest vm I cannot reach port 3306 From within the trove instance I can reach port 3306 and mysql is functional
It feels like the security group does not allow 3306, but the entry is there (see etherpad)
What puzzles me is that netstat in the trove instance does only show listen to 3306 for tcp6 but not for tcp - whereas 22 listens for tcp6 AND tcp
root@ms57plain:~# netstat -anp | head -17 Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 3867/systemd-resolv tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1074/sshd tcp 0 0 10.9.9.126:22 10.9.9.128:60126 ESTABLISHED 9412/sshd: root@pts tcp 0 0 10.10.10.84:54326 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp 0 0 10.9.9.126:22 10.9.9.128:57880 ESTABLISHED 9021/sshd: root@pts tcp 0 0 10.10.10.84:44266 10.0.0.120:5672 ESTABLISHED 1002/python3 tcp6 0 0 :::3306 :::* LISTEN 4881/mysqld
You can see here that mysql is listening on all interfaces via an IPv4-mapped IPv6 address ( https://en.wikipedia.org/wiki/IPv6#IPv4-mapped_IPv6_addresses), and from your previous post we can see that there is an explicit ACCEPT for the lo interface. That's how you're able to get access to the DB from within the guest.
It looks to me that whatever added the ' -A openstack-INPUT -p tcp -m tcp --dport 22 -j ACCEPT' to the guest's firewall didn't also add an explicit rule for the database. At this point I don't know if that's trove, nova or neutron. If I had to guess I'd say that there is a mismatch in the trove configuration and the networks assigned to the guest.
For example how does --- network_driver = trove.network.neutron.NeutronDriver # provider management_networks = 98b0de37-40af-4859-81f2-d89a4df2d6df # trovemgmt # provider ## management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # trove-mgmt 3928eb68-edf9-4cca-8fa6-151ed26b7c04 # management_networks = 3928eb68-edf9-4cca-8fa6-151ed26b7c04 management_security_groups = 26c4af6e-56f0-4644-8c55-e96983c74c1d ---
relate to the "dozenet" and "trovemgmt" networks.
Tony.
------------------------------
Message: 2 Date: Sun, 31 Dec 2023 11:27:37 -0000 From: dieter.dorra@hsnr.de Subject: Re: [trove][victoria] can't connect to trove DB from outside - although ssh to the trove instance works To: openstack-discuss@lists.openstack.org Message-ID: <170402205718.409.7027996557330367167@lists01.opendev.org> Content-Type: text/plain; charset="utf-8"
The problem is solved! I tested a newer trove image from https://tarballs.opendev.org/openstack/trove/images/ The Victoria image caused the problem, it never showed the exposed ports in "docker ps". The Zed image is working:
ubuntu@dd:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1741ae3e7815 mysql:5.7.29 "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp database ubuntu@dd:~$
Thank you all so much for your assistance and help!!! Happy new year!!
------------------------------
Message: 3 Date: Sun, 31 Dec 2023 16:49:39 -0000 From: dieter.dorra@hsnr.de Subject: Re: Trove installation filed To: openstack-discuss@lists.openstack.org Message-ID: <170404137950.409.18389774625483655904@lists01.opendev.org> Content-Type: text/plain; charset="utf-8"
Hi, when you log in as user trove in the service project - can you launch an instance? "no valid host was found" sounds that nova has a problem... BR Dieter
------------------------------
Message: 4 Date: Mon, 01 Jan 2024 05:40:46 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com> Subject: Re: Trove installation filed To: openstack-discuss@lists.openstack.org Message-ID: <170408764655.409.16151116383780837301@lists01.opendev.org> Content-Type: text/plain; charset="utf-8"
Please provide and check the Trove guest agent log file (by default, /var/log/trove/trove-guestagent.log) for any errors or warnings..
Also share with us the logs output that we can take a look.
By the way, dont forget to check this following link to troubleshoot;
https://docs.openstack.org/trove/latest/admin/troubleshooting.html
Kerem Çeliker
------------------------------
Message: 5 Date: Mon, 01 Jan 2024 05:46:32 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com> Subject: Re: [kolla] ceilometer and nova compute agent To: openstack-discuss@lists.openstack.org Message-ID: <170408799208.407.10760598762569499228@lists01.opendev.org> Content-Type: text/plain; charset="utf-8"
You dont need to create a custom nova container to measure instances metrics. Only can use the “openstack host show” and “openstack server list” to get basic stats on resource usage for hosts and instances.
Also use tools, such as Ganglia or Graphite, to gather more detailed data. It’s up to you!
Kerem Çeliker
------------------------------
Message: 6 Date: Mon, 01 Jan 2024 06:07:02 -0000 From: "KEREM CELIKER" <kmceliker@gmail.com> Subject: Re: ceilometer cli client install help To: openstack-discuss@lists.openstack.org Message-ID: <170408922210.407.17546497674894904858@lists01.opendev.org> Content-Type: text/plain; charset="utf-8"
It seems that you have not gnocchi plugin on OSP All-In-One. Also your OSP has not correct openstack client version or you needed the install the gnocchi plugin correctly.
Please check gnocchi plugin if installed.. Here is the cli’s ; • openstack plugin list and check for gnocchi in the output any issues.. or try it; • pip install python-gnocchiclient.
Kerem Çeliker
------------------------------
Message: 7 Date: Tue, 2 Jan 2024 09:52:31 +0100 From: Bernard Cafarelli <bcafarel@redhat.com> Subject: [neutron] Bug deputy report (last week of 2023) To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Message-ID: <CABHdKwov77XQ31h26vq6UvzyOjaANcU1F6zjY1opS_H2XZp= PQ@mail.gmail.com> Content-Type: multipart/alternative; boundary="0000000000001b8f3a060df29f3c"
Hey neutrinos, and happy new year! Here is the bug reported between 2023-12-25 and 2023-12-31 (as per tradition, it was a quiet week)
The firewall group is ACTIVE without associate with a router or any firewall policies - https://bugs.launchpad.net/neutron/+bug/2047325 neutron-fwaas bug, patch proposed: https://review.opendev.org/c/openstack/neutron-fwaas/+/904317
Best wishes -- Bernard Cafarelli
participants (2)
-
Asma Naz Shariq
-
Rodolfo Alonso Hernandez