[Openstack] Can not ping the tenant router gateway from any nodes after all nodes shut down/turned on

Mitchell Chen mitchell.chen at mediatek.com
Tue Mar 3 01:45:25 UTC 2015


Hi Remo,

Sorry, I pinged the wrong gateway, 172.29.105.1 last time. Actually, I can ping the gateway, 172.29.105.101, using the router:

root at neutron:~# ip netns exec qrouter-b69f04d9-910d-472e-8f2c-ac1344c6127f ping -I qg-8aac20dc-60 172.29.105.101
PING 172.29.105.101 (172.29.105.101) from 172.29.105.101 qg-8aac20dc-60: 56(84) bytes of data.
64 bytes from 172.29.105.101: icmp_seq=1 ttl=64 time=0.055 ms
64 bytes from 172.29.105.101: icmp_seq=2 ttl=64 time=0.050 ms
64 bytes from 172.29.105.101: icmp_seq=3 ttl=64 time=0.054 ms
64 bytes from 172.29.105.101: icmp_seq=4 ttl=64 time=0.060 ms
64 bytes from 172.29.105.101: icmp_seq=5 ttl=64 time=0.056 ms
64 bytes from 172.29.105.101: icmp_seq=6 ttl=64 time=0.059 ms
64 bytes from 172.29.105.101: icmp_seq=7 ttl=64 time=0.051 ms
64 bytes from 172.29.105.101: icmp_seq=8 ttl=64 time=0.048 ms

root at neutron:~# ip netns
qrouter-b69f04d9-910d-472e-8f2c-ac1344c6127f

root at neutron:~# ovs-vsctl show
f22d3ba4-d785-4fe7-a283-16ffbc75c434
    Bridge br-int
        fail_mode: secure
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qr-63d87686-6a"
            tag: 1
            Interface "qr-63d87686-6a"
                type: internal
        Port br-int
            Interface br-int
                type: internal
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-8aac20dc-60"
            Interface "qg-8aac20dc-60"
                type: internal
        Port "eth2"
            Interface "eth2"
    Bridge br-tun
        Port "gre-c0a801cc"
            Interface "gre-c0a801cc"
                type: gre
                options: {in_key=flow, local_ip="192.168.1.220", out_key=flow, remote_ip="192.168.1.204"}
        Port br-tun
            Interface br-tun
                type: internal
        Port "gre-c0a801d5"
            Interface "gre-c0a801d5"
                type: gre
                options: {in_key=flow, local_ip="192.168.1.220", out_key=flow, remote_ip="192.168.1.213"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-c0a801d2"
            Interface "gre-c0a801d2"
                type: gre
                options: {in_key=flow, local_ip="192.168.1.220", out_key=flow, remote_ip="192.168.1.210"}
    ovs_version: "2.0.2"

But the router gateway can not be pinned from the host:
root@ neutron:~# ping -c 4 172.29.105.101
PING 172.29.105.101 (172.29.105.101) 56(84) bytes of data.

--- 172.29.105.101 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3022ms

root@ neutron:/home/stack/Openstack-mtk# neutron net-list
+--------------------------------------+------------+------------------------------------------------------+
| id                                   | name       | subnets                                              |
+--------------------------------------+------------+------------------------------------------------------+
| 2b3a3277-c8d9-4767-8aaa-ecd9f171b65c | extern-net | 9e4527ab-a63b-4b76-9757-0f802e76484b 172.29.105.0/24 |
| c5a1668d-4ddc-41fd-94eb-d5c6d8b9e974 | demo-net   | d33b74e4-7668-46f4-8f13-88034fc79624 192.168.2.0/24  |
+--------------------------------------+------------+------------------------------------------------------+
root@ neutron:/home/stack/Openstack-mtk# neutron subnet-list
+--------------------------------------+---------------+-----------------+------------------------------------------------------+
| id                                   | name          | cidr            | allocation_pools                                     |
+--------------------------------------+---------------+-----------------+------------------------------------------------------+
| 9e4527ab-a63b-4b76-9757-0f802e76484b | extern-subnet | 172.29.105.0/24 | {"start": "172.29.105.101", "end": "172.29.105.127"} |
| d33b74e4-7668-46f4-8f13-88034fc79624 | demo-subnet   | 192.168.2.0/24  | {"start": "192.168.2.2", "end": "192.168.2.254"}     |
+--------------------------------------+---------------+-----------------+------------------------------------------------------+

root@ neutron:/home/stack/Openstack-mtk# neutron subnet-show extern-subnet
+------------------+------------------------------------------------------+
| Field            | Value                                                |
+------------------+------------------------------------------------------+
| allocation_pools | {"start": "172.29.105.101", "end": "172.29.105.127"} |
| cidr             | 172.29.105.0/24                                      |
| dns_nameservers  |                                                      |
| enable_dhcp      | False                                                |
| gateway_ip       | 172.29.105.254                                       |
| host_routes      |                                                      |
| id               | 9e4527ab-a63b-4b76-9757-0f802e76484b                 |
| ip_version       | 4                                                    |
| name             | extern-subnet                                        |
| network_id       | 2b3a3277-c8d9-4767-8aaa-ecd9f171b65c                 |
| tenant_id        | 7479d4eabeb14b45a7f38269155ec0f5                     |
+------------------+------------------------------------------------------+
root@ neutron:/home/stack/Openstack-mtk# neutron subnet-show demo-subnet
+------------------+--------------------------------------------------+
| Field            | Value                                            |
+------------------+--------------------------------------------------+
| allocation_pools | {"start": "192.168.2.2", "end": "192.168.2.254"} |
| cidr             | 192.168.2.0/24                                   |
| dns_nameservers  |                                                  |
| enable_dhcp      | True                                             |
| gateway_ip       | 192.168.2.1                                      |
| host_routes      |                                                  |
| id               | d33b74e4-7668-46f4-8f13-88034fc79624             |
| ip_version       | 4                                                |
| name             | demo-subnet                                      |
| network_id       | c5a1668d-4ddc-41fd-94eb-d5c6d8b9e974             |
| tenant_id        | 1bc3de8b19384858a1a7a6395e1845e3                 |
+------------------+--------------------------------------------------+


From: Remo Mattei [mailto:remo at italy1.com]
Sent: Friday, February 27, 2015 11:39 PM
To: Mitchell Chen
Cc: openstack at lists.openstack.org
Subject: Re: [Openstack] Can not ping the tenant router gateway from any nodes after all nodes shut down/turned on

Then you know there is something wrong with that router how did you set it up? Is this rdo?

Ciao

Inviato da iPhone ()

Il giorno 27/feb/2015, alle ore 23:35, Mitchell Chen <mitchell.chen at mediatek.com<mailto:mitchell.chen at mediatek.com>> ha scritto:
It can’t be pinged:

root@ neutron:~# ip netns exec qrouter-48cc6f75-b0a2-4468-a0bd-e6567237b7ee ping -I  qg-34625a11-da 172.29.105.1
PING 172.29.105.1 (172.29.105.1) from 172.29.105.101 qg-34625a11-da: 56(84) bytes of data.
From 172.29.105.101 icmp_seq=1 Destination Host Unreachable
From 172.29.105.101 icmp_seq=2 Destination Host Unreachable
From 172.29.105.101 icmp_seq=3 Destination Host Unreachable
From 172.29.105.101 icmp_seq=4 Destination Host Unreachable
From 172.29.105.101 icmp_seq=5 Destination Host Unreachable
From 172.29.105.101 icmp_seq=6 Destination Host Unreachable
From 172.29.105.101 icmp_seq=7 Destination Host Unreachable
From 172.29.105.101 icmp_seq=8 Destination Host Unreachable
From 172.29.105.101 icmp_seq=9 Destination Host Unreachable
From 172.29.105.101 icmp_seq=10 Destination Host Unreachable
From 172.29.105.101 icmp_seq=11 Destination Host Unreachable
From 172.29.105.101 icmp_seq=12 Destination Host Unreachable
From 172.29.105.101 icmp_seq=13 Destination Host Unreachable
From 172.29.105.101 icmp_seq=14 Destination Host Unreachable

From: Remo Mattei [mailto:Remo at Italy1.com]
Sent: Friday, February 27, 2015 11:08 PM
To: Mitchell Chen
Cc: openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: Re: [Openstack] Can not ping the tenant router gateway from any nodes after all nodes shut down/turned on

Sorry No No. that is the ip address of the virtual router. You need to ping the gateway using the router. So it should look like this

ip netns exec qrouter-48cc6f75-b0a2-4468-a0bd-e6567237b7ee ping -I  qg-34625a11-da 172.29.105.1



On Feb 27, 2015, at 22:57, Mitchell Chen <mitchell.chen at mediatek.com<mailto:mitchell.chen at mediatek.com>> wrote:

Hi Remo,

Thank you for looking at this issue. Yes, I can ping the gateway from the router. Please look the red in the following my previous mail.  I followed the OpenStack installation guide, http://docs.openstack.org/icehouse/install-guide/install/apt/content/neutron_initial-tenant-network.html, to set up the network. More detail networks and router are in the following: Please let me know if you need any more information.

Thanks,
Mitchell

root at control:/# neutron net-list
+--------------------------------------+----------+------------------------------------------------------+
| id                                   | name     | subnets                                              |
+--------------------------------------+----------+------------------------------------------------------+
| 3ef88278-b944-4ba2-a39b-0c23a2f22385 | ext-net  | 35501767-4b22-4ffc-9ed6-cdbe59759b9e 172.29.105.0/24 |
| 6f4e6906-ec84-410f-bfc1-89e076791ddb | demo-net | 318125f7-3015-44cb-80fe-f8682a997392 192.168.2.0/24  |
+--------------------------------------+----------+------------------------------------------------------+
root@ control:/ # neutron subnet-list
+--------------------------------------+-------------+-----------------+------------------------------------------------------+
| id                                   | name        | cidr            | allocation_pools                                     |
+--------------------------------------+-------------+-----------------+------------------------------------------------------+
| 318125f7-3015-44cb-80fe-f8682a997392 | demo-subnet | 192.168.2.0/24  | {"start": "192.168.2.2", "end": "192.168.2.254"}     |
| 35501767-4b22-4ffc-9ed6-cdbe59759b9e | ext-subnet  | 172.29.105.0/24 | {"start": "172.29.105.101", "end": "172.29.105.127"} |
+--------------------------------------+-------------+-----------------+------------------------------------------------------+
root@ control:/ # neutron router-list
+--------------------------------------+-------------+-----------------------------------------------------------------------------+
| id                                   | name        | external_gateway_info                                                       |
+--------------------------------------+-------------+-----------------------------------------------------------------------------+
| 48cc6f75-b0a2-4468-a0bd-e6567237b7ee | demo-router | {"network_id": "3ef88278-b944-4ba2-a39b-0c23a2f22385", "enable_snat": true} |
+--------------------------------------+-------------+-----------------------------------------------------------------------------+
root at mussdhux23:/home/stack/Openstack-mtk# neutron router-show 48cc6f75-b0a2-4468-a0bd-e6567237b7ee
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "3ef88278-b944-4ba2-a39b-0c23a2f22385", "enable_snat": true} |
| id                    | 48cc6f75-b0a2-4468-a0bd-e6567237b7ee                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | 1bc3de8b19384858a1a7a6395e1845e3                                            |
+-----------------------+-----------------------------------------------------------------------------+
root at mussdhux23:/home/stack/Openstack-mtk# neutron router-port-list 48cc6f75-b0a2-4468-a0bd-e6567237b7ee
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                             |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+
| 34625a11-da44-4507-8aa8-6073b724816b |      | fa:16:3e:d7:2f:27 | {"subnet_id": "35501767-4b22-4ffc-9ed6-cdbe59759b9e", "ip_address": "172.29.105.101"} |
| 70c2c01f-8306-45df-958c-b445eb3523ee |      | fa:16:3e:27:83:6c | {"subnet_id": "318125f7-3015-44cb-80fe-f8682a997392", "ip_address": "192.168.2.1"}    |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+

From: Remo Mattei [mailto:Remo at Italy1.com]
Sent: Friday, February 27, 2015 9:08 PM
To: Mitchell Chen
Cc: openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: Re: [Openstack] Can not ping the tenant router gateway from any nodes after all nodes shut down/turned on

that only shows you ping your interface can you ping your gateway from the router? How was the router created? Did you enable provider network? You show some data but not all.


Remo
On Feb 27, 2015, at 18:52, Mitchell Chen <mitchell.chen at mediatek.com<mailto:mitchell.chen at mediatek.com>> wrote:

These is an issue that the tenant router gateway can not be pinned. Is this an OpenStack issue? It doesn’t allow all nodes to be powered off and on? Before the nodes were shut down, networking works great either from inside instances to access outside internet or from outside to access the instances. Please help:

root at neutron:/# neutron subnet-list
+--------------------------------------+-------------+-----------------+------------------------------------------------------+
| id                                   | name        | cidr            | allocation_pools                                     |
+--------------------------------------+-------------+-----------------+------------------------------------------------------+
| 318125f7-3015-44cb-80fe-f8682a997392 | demo-subnet | 192.168.2.0/24  | {"start": "192.168.2.2", "end": "192.168.2.254"}     |
| 35501767-4b22-4ffc-9ed6-cdbe59759b9e | ext-subnet  | 172.29.105.0/24 | {"start": "172.29.105.101", "end": "172.29.105.127"} |
+--------------------------------------+-------------+-----------------+------------------------------------------------------+
root@ neutron:/# neutron subnet-show 35501767-4b22-4ffc-9ed6-cdbe59759b9e
+------------------+------------------------------------------------------+
| Field            | Value                                                |
+------------------+------------------------------------------------------+
| allocation_pools | {"start": "172.29.105.101", "end": "172.29.105.127"} |
| cidr             | 172.29.105.0/24                                      |
| dns_nameservers  |                                                      |
| enable_dhcp      | False                                                |
| gateway_ip       | 172.29.105.254                                       |
| host_routes      |                                                      |
| id               | 35501767-4b22-4ffc-9ed6-cdbe59759b9e                 |
| ip_version       | 4                                                    |
| name             | ext-subnet                                           |
| network_id       | 3ef88278-b944-4ba2-a39b-0c23a2f22385                 |
| tenant_id        | 7479d4eabeb14b45a7f38269155ec0f5                     |






root@ neutron:/# ip netns
qdhcp-6f4e6906-ec84-410f-bfc1-89e076791ddb
qrouter-48cc6f75-b0a2-4468-a0bd-e6567237b7ee

root@ neutron:/# ip netns exec qrouter-48cc6f75-b0a2-4468-a0bd-e6567237b7ee ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
35: qr-70c2c01f-83: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether fa:16:3e:27:83:6c brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.1/24 brd 192.168.2.255 scope global qr-70c2c01f-83
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe27:836c/64 scope link
       valid_lft forever preferred_lft forever
36:qg-34625a11-da: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether fa:16:3e:d7:2f:27 brd ff:ff:ff:ff:ff:ff
    inet 172.29.105.101/24 brd 172.29.105.255 scope global qg-34625a11-da
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fed7:2f27/64 scope link
       valid_lft forever preferred_lft forever

// From the qrouter, the gateway can be pinned
root@ neutron:/# ip netns exec qrouter-48cc6f75-b0a2-4468-a0bd-e6567237b7ee ping -c 4 172.29.105.101
PING 172.29.105.101 (172.29.105.101) 56(84) bytes of data.
64 bytes from 172.29.105.101: icmp_seq=1 ttl=64 time=0.063 ms
64 bytes from 172.29.105.101: icmp_seq=2 ttl=64 time=0.060 ms
64 bytes from 172.29.105.101: icmp_seq=3 ttl=64 time=0.059 ms
64 bytes from 172.29.105.101: icmp_seq=4 ttl=64 time=0.058 ms

//// Can not be pinned here:
root@ neutron:/# ping -c 4  172.29.105.101
PING 172.29.105.101 (172.29.105.101) 56(84) bytes of data.

--- 172.29.105.101 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3023ms

root@ neutron:/home/stack/Openstack# neutron agent-list
+--------------------------------------+--------------------+------------+-------+----------------+
| id                                   | agent_type         | host       | alive | admin_state_up |
+--------------------------------------+--------------------+------------+-------+----------------+
| 68eba267-b885-4aad-96a4-e8d19ac9db4a | L3 agent           | mussdhux20 | :-)   | True           |
| c0726b66-a1f4-4b00-9120-7bf0ee91a3d0 | DHCP agent         | mussdhux20 | :-)   | True           |
| d8808f97-90f8-424d-a786-45b4541af755 | Open vSwitch agent | mussdhux04 | :-)   | True           |
| e8d85451-3f6f-4b2a-94a1-6dd3e9a93cff | Metadata agent     | mussdhux20 | :-)   | True           |
| f4e51873-7e18-49a4-ab6b-94c24500e25f | Open vSwitch agent | mussdhux13 | :-)   | True           |
| f8cab47d-a0db-4de2-b457-23c4e87ccb32 | Open vSwitch agent | mussdhux10 | :-)   | True           |
| fd9ab1c9-ce62-4504-8d68-215584a70a17 | Open vSwitch agent | mussdhux20 | :-)   | True           |
+--------------------------------------+--------------------+------------+-------+----------------+

I also restarted the neutron services after my neutron node booted up.
service neutron-plugin-openvswitch-agent restart
service neutron-l3-agent restart
service neutron-dhcp-agent restart
service neutron-metadata-agent restart

Anyone has solutions for this kind of issue?


Thank you so much for the help,
Mitchell

From: Mark Loza [mailto:mloza at morphlabs.com]
Sent: Friday, February 27, 2015 3:59 AM
To: Mitchell Chen; openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: Re: [Openstack] Instance can not ping each other

I assume you are using neutron instead of nova-network

On 2/27/15 7:57 PM, Mark Loza wrote:
Hi,

Did you check if the neutron agents are all alive?

# neutron agent-list


On 2/27/15 3:49 PM, Mitchell Chen wrote:
Hi All,

I followed the OpenStack icehouse installation guide to successfully install a five node system, one control node, one neutron and three compute nodes.  Inside of the instance, I can ping other instances’ virtual IP address and outside internet fine. Recently, we decided to move all the nodes to a lab. So, I powered down the five nodes before the move and powered up them afterward. All the physical network connections were exactly the same as they were before the move. But, I can not even ping other instance after the five nodes up again.

I typed “route –n” inside of the instance, no any IP is shown. What’s wrong? Please help how I can get the instance networking back.

Thanks,
Mitchell

************* Email Confidentiality Notice ********************

The information contained in this e-mail message (including any

attachments) may be confidential, proprietary, privileged, or otherwise

exempt from disclosure under applicable laws. It is intended to be

conveyed only to the designated recipient(s). Any use, dissemination,

distribution, printing, retaining or copying of this e-mail (including its

attachments) by unintended recipient(s) is strictly prohibited and may

be unlawful. If you are not an intended recipient of this e-mail, or believe

that you have received this e-mail in error, please notify the sender

immediately (by replying to this e-mail), delete any and all copies of

this e-mail (including any attachments) from your system, and do not

disclose the content of this e-mail to any other person. Thank you!



_______________________________________________

Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Post to     : openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>

Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



************* Email Confidentiality Notice ********************

The information contained in this e-mail message (including any

attachments) may be confidential, proprietary, privileged, or otherwise

exempt from disclosure under applicable laws. It is intended to be

conveyed only to the designated recipient(s). Any use, dissemination,

distribution, printing, retaining or copying of this e-mail (including its

attachments) by unintended recipient(s) is strictly prohibited and may

be unlawful. If you are not an intended recipient of this e-mail, or believe

that you have received this e-mail in error, please notify the sender

immediately (by replying to this e-mail), delete any and all copies of

this e-mail (including any attachments) from your system, and do not

disclose the content of this e-mail to any other person. Thank you!



************* Email Confidentiality Notice ********************

The information contained in this e-mail message (including any

attachments) may be confidential, proprietary, privileged, or otherwise

exempt from disclosure under applicable laws. It is intended to be

conveyed only to the designated recipient(s). Any use, dissemination,

distribution, printing, retaining or copying of this e-mail (including its

attachments) by unintended recipient(s) is strictly prohibited and may

be unlawful. If you are not an intended recipient of this e-mail, or believe

that you have received this e-mail in error, please notify the sender

immediately (by replying to this e-mail), delete any and all copies of

this e-mail (including any attachments) from your system, and do not

disclose the content of this e-mail to any other person. Thank you!

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack




!DSPAM:1,54f16872211305095711045!

************* Email Confidentiality Notice ********************

The information contained in this e-mail message (including any

attachments) may be confidential, proprietary, privileged, or otherwise

exempt from disclosure under applicable laws. It is intended to be

conveyed only to the designated recipient(s). Any use, dissemination,

distribution, printing, retaining or copying of this e-mail (including its

attachments) by unintended recipient(s) is strictly prohibited and may

be unlawful. If you are not an intended recipient of this e-mail, or believe

that you have received this e-mail in error, please notify the sender

immediately (by replying to this e-mail), delete any and all copies of

this e-mail (including any attachments) from your system, and do not

disclose the content of this e-mail to any other person. Thank you!


!DSPAM:1,54f16fd1252013253211460!

************* Email Confidentiality Notice ********************

The information contained in this e-mail message (including any

attachments) may be confidential, proprietary, privileged, or otherwise

exempt from disclosure under applicable laws. It is intended to be

conveyed only to the designated recipient(s). Any use, dissemination,

distribution, printing, retaining or copying of this e-mail (including its

attachments) by unintended recipient(s) is strictly prohibited and may

be unlawful. If you are not an intended recipient of this e-mail, or believe

that you have received this e-mail in error, please notify the sender

immediately (by replying to this e-mail), delete any and all copies of

this e-mail (including any attachments) from your system, and do not

disclose the content of this e-mail to any other person. Thank you!


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150303/0338ccd7/attachment.html>


More information about the Openstack mailing list