[Openstack] Can not ping the tenant router gateway from any nodes after all nodes shut down/turned on

Mitchell Chen mitchell.chen at mediatek.com
Sat Feb 28 09:41:58 UTC 2015


Hi Remo,

The router was set as the following OpenStack neutron commands in control node. What can be wrong?

Thanks,
Mitchell

root at control:/Openstack# source admin-openrc.sh
root@ control:/Openstack # neutron net-create ext-net --shared --router:external=True
Created a new network:
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 3ef88278-b944-4ba2-a39b-0c23a2f22385 |
| name                      | ext-net                              |
| provider:network_type     | gre                                  |
| provider:physical_network |                                      |
| provider:segmentation_id  | 1                                    |
| router:external           | True                                 |
| shared                    | True                                 |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tenant_id                 | 7479d4eabeb14b45a7f38269155ec0f5     |
+---------------------------+--------------------------------------+
root@ control:/home/stack/Openstack-mtk# neutron subnet-create ext-net --name ext-subnet --allocation-pool start=172.29.105.101,end=172.29.105.127 --disable-dhcp --gateway 172.29.105.254 172.29.105.0/24
Created a new subnet:
+------------------+------------------------------------------------------+
| Field            | Value                                                |
+------------------+------------------------------------------------------+
| allocation_pools | {"start": "172.29.105.101", "end": "172.29.105.127"} |
| cidr             | 172.29.105.0/24                                      |
| dns_nameservers  |                                                      |
| enable_dhcp      | False                                                |
| gateway_ip       | 172.29.105.254                                       |
| host_routes      |                                                      |
| id               | 35501767-4b22-4ffc-9ed6-cdbe59759b9e                 |
| ip_version       | 4                                                    |
| name             | ext-subnet                                           |
| network_id       | 3ef88278-b944-4ba2-a39b-0c23a2f22385                 |
| tenant_id        | 7479d4eabeb14b45a7f38269155ec0f5                     |
+------------------+------------------------------------------------------+
root@ control:/Openstack #  source demo-openrc.sh
root@ control:/Openstack # neutron net-create demo-net
Created a new network:
+----------------+--------------------------------------+
| Field          | Value                                |
+----------------+--------------------------------------+
| admin_state_up | True                                 |
| id             | 6f4e6906-ec84-410f-bfc1-89e076791ddb |
| name           | demo-net                             |
| shared         | False                                |
| status         | ACTIVE                               |
| subnets        |                                      |
| tenant_id      | 1bc3de8b19384858a1a7a6395e1845e3     |
+----------------+--------------------------------------+
root@ control:/Openstack # neutron subnet-create demo-net --name demo-subnet --gateway 192.168.2.1 192.168.2.0/24 --dns_nameservers list=true 8.8.8.8
Created a new subnet:
+------------------+--------------------------------------------------+
| Field            | Value                                            |
+------------------+--------------------------------------------------+
| allocation_pools | {"start": "192.168.2.2", "end": "192.168.2.254"} |
| cidr             | 192.168.2.0/24                                   |
| dns_nameservers  | 8.8.8.8                                          |
| enable_dhcp      | True                                             |
| gateway_ip       | 192.168.2.1                                      |
| host_routes      |                                                  |
| id               | 318125f7-3015-44cb-80fe-f8682a997392             |
| ip_version       | 4                                                |
| name             | demo-subnet                                      |
| network_id       | 6f4e6906-ec84-410f-bfc1-89e076791ddb             |
| tenant_id        | 1bc3de8b19384858a1a7a6395e1845e3                 |
+------------------+--------------------------------------------------+
root@ control:/Openstack # neutron router-create demo-router
Created a new router:
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| admin_state_up        | True                                 |
| external_gateway_info |                                      |
| id                    | 48cc6f75-b0a2-4468-a0bd-e6567237b7ee |
| name                  | demo-router                          |
| status                | ACTIVE                               |
| tenant_id             | 1bc3de8b19384858a1a7a6395e1845e3     |
+-----------------------+--------------------------------------+
root at control:/Openstack# neutron router-interface-add demo-router demo-subnet
Added interface 70c2c01f-8306-45df-958c-b445eb3523ee to router demo-router.
root at control:/Openstack # neutron router-gateway-set demo-router ext-net
Set gateway for router demo-router

root at control:/Openstack # ping 172.29.105.101
PING 172.29.105.101 (172.29.105.101) 56(84) bytes of data.
^C
--- 172.29.105.101 ping statistics ---
29 packets transmitted, 0 received, 100% packet loss, time 28222ms

From: Mitchell Chen [mailto:mitchell.chen at mediatek.com]
Sent: Friday, February 27, 2015 11:44 PM
To: Remo Mattei
Cc: openstack at lists.openstack.org
Subject: Re: [Openstack] Can not ping the tenant router gateway from any nodes after all nodes shut down/turned on

It’s OpenStack icehouse with  Ubuntu 14.04. I followed the steps described in the OpenStack installation guide:

 http://docs.openstack.org/icehouse/install-guide/install/apt/content/neutron_initial-tenant-network.html

From: Remo Mattei [mailto:remo at italy1.com]
Sent: Friday, February 27, 2015 11:39 PM
To: Mitchell Chen
Cc: openstack at lists.openstack.org
Subject: Re: [Openstack] Can not ping the tenant router gateway from any nodes after all nodes shut down/turned on

Then you know there is something wrong with that router how did you set it up? Is this rdo?

Ciao

Inviato da iPhone ()

Il giorno 27/feb/2015, alle ore 23:35, Mitchell Chen <mitchell.chen at mediatek.com<mailto:mitchell.chen at mediatek.com>> ha scritto:
It can’t be pinged:

root@ neutron:~# ip netns exec qrouter-48cc6f75-b0a2-4468-a0bd-e6567237b7ee ping -I  qg-34625a11-da 172.29.105.1
PING 172.29.105.1 (172.29.105.1) from 172.29.105.101 qg-34625a11-da: 56(84) bytes of data.
From 172.29.105.101 icmp_seq=1 Destination Host Unreachable
From 172.29.105.101 icmp_seq=2 Destination Host Unreachable
From 172.29.105.101 icmp_seq=3 Destination Host Unreachable
From 172.29.105.101 icmp_seq=4 Destination Host Unreachable
From 172.29.105.101 icmp_seq=5 Destination Host Unreachable
From 172.29.105.101 icmp_seq=6 Destination Host Unreachable
From 172.29.105.101 icmp_seq=7 Destination Host Unreachable
From 172.29.105.101 icmp_seq=8 Destination Host Unreachable
From 172.29.105.101 icmp_seq=9 Destination Host Unreachable
From 172.29.105.101 icmp_seq=10 Destination Host Unreachable
From 172.29.105.101 icmp_seq=11 Destination Host Unreachable
From 172.29.105.101 icmp_seq=12 Destination Host Unreachable
From 172.29.105.101 icmp_seq=13 Destination Host Unreachable
From 172.29.105.101 icmp_seq=14 Destination Host Unreachable

From: Remo Mattei [mailto:Remo at Italy1.com]
Sent: Friday, February 27, 2015 11:08 PM
To: Mitchell Chen
Cc: openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: Re: [Openstack] Can not ping the tenant router gateway from any nodes after all nodes shut down/turned on

Sorry No No. that is the ip address of the virtual router. You need to ping the gateway using the router. So it should look like this

ip netns exec qrouter-48cc6f75-b0a2-4468-a0bd-e6567237b7ee ping -I  qg-34625a11-da 172.29.105.1



On Feb 27, 2015, at 22:57, Mitchell Chen <mitchell.chen at mediatek.com<mailto:mitchell.chen at mediatek.com>> wrote:

Hi Remo,

Thank you for looking at this issue. Yes, I can ping the gateway from the router. Please look the red in the following my previous mail.  I followed the OpenStack installation guide, http://docs.openstack.org/icehouse/install-guide/install/apt/content/neutron_initial-tenant-network.html, to set up the network. More detail networks and router are in the following: Please let me know if you need any more information.

Thanks,
Mitchell

root at control:/# neutron net-list
+--------------------------------------+----------+------------------------------------------------------+
| id                                   | name     | subnets                                              |
+--------------------------------------+----------+------------------------------------------------------+
| 3ef88278-b944-4ba2-a39b-0c23a2f22385 | ext-net  | 35501767-4b22-4ffc-9ed6-cdbe59759b9e 172.29.105.0/24 |
| 6f4e6906-ec84-410f-bfc1-89e076791ddb | demo-net | 318125f7-3015-44cb-80fe-f8682a997392 192.168.2.0/24  |
+--------------------------------------+----------+------------------------------------------------------+
root@ control:/ # neutron subnet-list
+--------------------------------------+-------------+-----------------+------------------------------------------------------+
| id                                   | name        | cidr            | allocation_pools                                     |
+--------------------------------------+-------------+-----------------+------------------------------------------------------+
| 318125f7-3015-44cb-80fe-f8682a997392 | demo-subnet | 192.168.2.0/24  | {"start": "192.168.2.2", "end": "192.168.2.254"}     |
| 35501767-4b22-4ffc-9ed6-cdbe59759b9e | ext-subnet  | 172.29.105.0/24 | {"start": "172.29.105.101", "end": "172.29.105.127"} |
+--------------------------------------+-------------+-----------------+------------------------------------------------------+
root@ control:/ # neutron router-list
+--------------------------------------+-------------+-----------------------------------------------------------------------------+
| id                                   | name        | external_gateway_info                                                       |
+--------------------------------------+-------------+-----------------------------------------------------------------------------+
| 48cc6f75-b0a2-4468-a0bd-e6567237b7ee | demo-router | {"network_id": "3ef88278-b944-4ba2-a39b-0c23a2f22385", "enable_snat": true} |
+--------------------------------------+-------------+-----------------------------------------------------------------------------+
root at mussdhux23:/home/stack/Openstack-mtk# neutron router-show 48cc6f75-b0a2-4468-a0bd-e6567237b7ee
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "3ef88278-b944-4ba2-a39b-0c23a2f22385", "enable_snat": true} |
| id                    | 48cc6f75-b0a2-4468-a0bd-e6567237b7ee                                        |
| name                  | demo-router                                                                 |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | 1bc3de8b19384858a1a7a6395e1845e3                                            |
+-----------------------+-----------------------------------------------------------------------------+
root at mussdhux23:/home/stack/Openstack-mtk# neutron router-port-list 48cc6f75-b0a2-4468-a0bd-e6567237b7ee
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                             |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+
| 34625a11-da44-4507-8aa8-6073b724816b |      | fa:16:3e:d7:2f:27 | {"subnet_id": "35501767-4b22-4ffc-9ed6-cdbe59759b9e", "ip_address": "172.29.105.101"} |
| 70c2c01f-8306-45df-958c-b445eb3523ee |      | fa:16:3e:27:83:6c | {"subnet_id": "318125f7-3015-44cb-80fe-f8682a997392", "ip_address": "192.168.2.1"}    |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+

From: Remo Mattei [mailto:Remo at Italy1.com]
Sent: Friday, February 27, 2015 9:08 PM
To: Mitchell Chen
Cc: openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: Re: [Openstack] Can not ping the tenant router gateway from any nodes after all nodes shut down/turned on

that only shows you ping your interface can you ping your gateway from the router? How was the router created? Did you enable provider network? You show some data but not all.


Remo
On Feb 27, 2015, at 18:52, Mitchell Chen <mitchell.chen at mediatek.com<mailto:mitchell.chen at mediatek.com>> wrote:

These is an issue that the tenant router gateway can not be pinned. Is this an OpenStack issue? It doesn’t allow all nodes to be powered off and on? Before the nodes were shut down, networking works great either from inside instances to access outside internet or from outside to access the instances. Please help:

root at neutron:/# neutron subnet-list
+--------------------------------------+-------------+-----------------+------------------------------------------------------+
| id                                   | name        | cidr            | allocation_pools                                     |
+--------------------------------------+-------------+-----------------+------------------------------------------------------+
| 318125f7-3015-44cb-80fe-f8682a997392 | demo-subnet | 192.168.2.0/24  | {"start": "192.168.2.2", "end": "192.168.2.254"}     |
| 35501767-4b22-4ffc-9ed6-cdbe59759b9e | ext-subnet  | 172.29.105.0/24 | {"start": "172.29.105.101", "end": "172.29.105.127"} |
+--------------------------------------+-------------+-----------------+------------------------------------------------------+
root@ neutron:/# neutron subnet-show 35501767-4b22-4ffc-9ed6-cdbe59759b9e
+------------------+------------------------------------------------------+
| Field            | Value                                                |
+------------------+------------------------------------------------------+
| allocation_pools | {"start": "172.29.105.101", "end": "172.29.105.127"} |
| cidr             | 172.29.105.0/24                                      |
| dns_nameservers  |                                                      |
| enable_dhcp      | False                                                |
| gateway_ip       | 172.29.105.254                                       |
| host_routes      |                                                      |
| id               | 35501767-4b22-4ffc-9ed6-cdbe59759b9e                 |
| ip_version       | 4                                                    |
| name             | ext-subnet                                           |
| network_id       | 3ef88278-b944-4ba2-a39b-0c23a2f22385                 |
| tenant_id        | 7479d4eabeb14b45a7f38269155ec0f5                     |






root@ neutron:/# ip netns
qdhcp-6f4e6906-ec84-410f-bfc1-89e076791ddb
qrouter-48cc6f75-b0a2-4468-a0bd-e6567237b7ee

root@ neutron:/# ip netns exec qrouter-48cc6f75-b0a2-4468-a0bd-e6567237b7ee ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
35: qr-70c2c01f-83: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether fa:16:3e:27:83:6c brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.1/24 brd 192.168.2.255 scope global qr-70c2c01f-83
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe27:836c/64 scope link
       valid_lft forever preferred_lft forever
36:qg-34625a11-da: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether fa:16:3e:d7:2f:27 brd ff:ff:ff:ff:ff:ff
    inet 172.29.105.101/24 brd 172.29.105.255 scope global qg-34625a11-da
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fed7:2f27/64 scope link
       valid_lft forever preferred_lft forever

// From the qrouter, the gateway can be pinned
root@ neutron:/# ip netns exec qrouter-48cc6f75-b0a2-4468-a0bd-e6567237b7ee ping -c 4 172.29.105.101
PING 172.29.105.101 (172.29.105.101) 56(84) bytes of data.
64 bytes from 172.29.105.101: icmp_seq=1 ttl=64 time=0.063 ms
64 bytes from 172.29.105.101: icmp_seq=2 ttl=64 time=0.060 ms
64 bytes from 172.29.105.101: icmp_seq=3 ttl=64 time=0.059 ms
64 bytes from 172.29.105.101: icmp_seq=4 ttl=64 time=0.058 ms

//// Can not be pinned here:
root@ neutron:/# ping -c 4  172.29.105.101
PING 172.29.105.101 (172.29.105.101) 56(84) bytes of data.

--- 172.29.105.101 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3023ms

root@ neutron:/home/stack/Openstack# neutron agent-list
+--------------------------------------+--------------------+------------+-------+----------------+
| id                                   | agent_type         | host       | alive | admin_state_up |
+--------------------------------------+--------------------+------------+-------+----------------+
| 68eba267-b885-4aad-96a4-e8d19ac9db4a | L3 agent           | mussdhux20 | :-)   | True           |
| c0726b66-a1f4-4b00-9120-7bf0ee91a3d0 | DHCP agent         | mussdhux20 | :-)   | True           |
| d8808f97-90f8-424d-a786-45b4541af755 | Open vSwitch agent | mussdhux04 | :-)   | True           |
| e8d85451-3f6f-4b2a-94a1-6dd3e9a93cff | Metadata agent     | mussdhux20 | :-)   | True           |
| f4e51873-7e18-49a4-ab6b-94c24500e25f | Open vSwitch agent | mussdhux13 | :-)   | True           |
| f8cab47d-a0db-4de2-b457-23c4e87ccb32 | Open vSwitch agent | mussdhux10 | :-)   | True           |
| fd9ab1c9-ce62-4504-8d68-215584a70a17 | Open vSwitch agent | mussdhux20 | :-)   | True           |
+--------------------------------------+--------------------+------------+-------+----------------+

I also restarted the neutron services after my neutron node booted up.
service neutron-plugin-openvswitch-agent restart
service neutron-l3-agent restart
service neutron-dhcp-agent restart
service neutron-metadata-agent restart

Anyone has solutions for this kind of issue?


Thank you so much for the help,
Mitchell

From: Mark Loza [mailto:mloza at morphlabs.com]
Sent: Friday, February 27, 2015 3:59 AM
To: Mitchell Chen; openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: Re: [Openstack] Instance can not ping each other

I assume you are using neutron instead of nova-network

On 2/27/15 7:57 PM, Mark Loza wrote:
Hi,

Did you check if the neutron agents are all alive?

# neutron agent-list


On 2/27/15 3:49 PM, Mitchell Chen wrote:
Hi All,

I followed the OpenStack icehouse installation guide to successfully install a five node system, one control node, one neutron and three compute nodes.  Inside of the instance, I can ping other instances’ virtual IP address and outside internet fine. Recently, we decided to move all the nodes to a lab. So, I powered down the five nodes before the move and powered up them afterward. All the physical network connections were exactly the same as they were before the move. But, I can not even ping other instance after the five nodes up again.

I typed “route –n” inside of the instance, no any IP is shown. What’s wrong? Please help how I can get the instance networking back.

Thanks,
Mitchell

************* Email Confidentiality Notice ********************

The information contained in this e-mail message (including any

attachments) may be confidential, proprietary, privileged, or otherwise

exempt from disclosure under applicable laws. It is intended to be

conveyed only to the designated recipient(s). Any use, dissemination,

distribution, printing, retaining or copying of this e-mail (including its

attachments) by unintended recipient(s) is strictly prohibited and may

be unlawful. If you are not an intended recipient of this e-mail, or believe

that you have received this e-mail in error, please notify the sender

immediately (by replying to this e-mail), delete any and all copies of

this e-mail (including any attachments) from your system, and do not

disclose the content of this e-mail to any other person. Thank you!



_______________________________________________

Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Post to     : openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>

Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



************* Email Confidentiality Notice ********************

The information contained in this e-mail message (including any

attachments) may be confidential, proprietary, privileged, or otherwise

exempt from disclosure under applicable laws. It is intended to be

conveyed only to the designated recipient(s). Any use, dissemination,

distribution, printing, retaining or copying of this e-mail (including its

attachments) by unintended recipient(s) is strictly prohibited and may

be unlawful. If you are not an intended recipient of this e-mail, or believe

that you have received this e-mail in error, please notify the sender

immediately (by replying to this e-mail), delete any and all copies of

this e-mail (including any attachments) from your system, and do not

disclose the content of this e-mail to any other person. Thank you!



************* Email Confidentiality Notice ********************

The information contained in this e-mail message (including any

attachments) may be confidential, proprietary, privileged, or otherwise

exempt from disclosure under applicable laws. It is intended to be

conveyed only to the designated recipient(s). Any use, dissemination,

distribution, printing, retaining or copying of this e-mail (including its

attachments) by unintended recipient(s) is strictly prohibited and may

be unlawful. If you are not an intended recipient of this e-mail, or believe

that you have received this e-mail in error, please notify the sender

immediately (by replying to this e-mail), delete any and all copies of

this e-mail (including any attachments) from your system, and do not

disclose the content of this e-mail to any other person. Thank you!

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



!DSPAM:1,54f16872211305095711045!

************* Email Confidentiality Notice ********************

The information contained in this e-mail message (including any

attachments) may be confidential, proprietary, privileged, or otherwise

exempt from disclosure under applicable laws. It is intended to be

conveyed only to the designated recipient(s). Any use, dissemination,

distribution, printing, retaining or copying of this e-mail (including its

attachments) by unintended recipient(s) is strictly prohibited and may

be unlawful. If you are not an intended recipient of this e-mail, or believe

that you have received this e-mail in error, please notify the sender

immediately (by replying to this e-mail), delete any and all copies of

this e-mail (including any attachments) from your system, and do not

disclose the content of this e-mail to any other person. Thank you!


!DSPAM:1,54f16fd1252013253211460!

************* Email Confidentiality Notice ********************

The information contained in this e-mail message (including any

attachments) may be confidential, proprietary, privileged, or otherwise

exempt from disclosure under applicable laws. It is intended to be

conveyed only to the designated recipient(s). Any use, dissemination,

distribution, printing, retaining or copying of this e-mail (including its

attachments) by unintended recipient(s) is strictly prohibited and may

be unlawful. If you are not an intended recipient of this e-mail, or believe

that you have received this e-mail in error, please notify the sender

immediately (by replying to this e-mail), delete any and all copies of

this e-mail (including any attachments) from your system, and do not

disclose the content of this e-mail to any other person. Thank you!



************* Email Confidentiality Notice ********************

The information contained in this e-mail message (including any

attachments) may be confidential, proprietary, privileged, or otherwise

exempt from disclosure under applicable laws. It is intended to be

conveyed only to the designated recipient(s). Any use, dissemination,

distribution, printing, retaining or copying of this e-mail (including its

attachments) by unintended recipient(s) is strictly prohibited and may

be unlawful. If you are not an intended recipient of this e-mail, or believe

that you have received this e-mail in error, please notify the sender

immediately (by replying to this e-mail), delete any and all copies of

this e-mail (including any attachments) from your system, and do not

disclose the content of this e-mail to any other person. Thank you!


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150228/76442dc2/attachment.html>


More information about the Openstack mailing list