[Openstack] Unable to access guests in DevStack on OpenStack environment
Juergen Brendel
juergen at brendel.com
Thu Mar 27 05:25:03 UTC 2014
Hello!
I would be very grateful if someone could please help me to
trouble-shoot a connectivity issue: I cannot ping or SSH into guests I
have created on top of DevStack.
What might be a complicating factor is that I have the DevStack cluster
running on guests that are running on top of a base OpenStack setup, but
most likely, I assume, I'm just making a really silly mistake in my
setup somewhere. I just need to find it. Or maybe, there's just
something odd about my setup that triggers some known issues?
Here's what I'm working with:
1. A base OpenStack install (Havana), using VLANs and a couple of
servers. One controller host and two compute hosts. This is my
base-cluster. It is not based on DevStack, but just an ordinary
OpenStack install.
2. I spin up three guest machines ("nova boot..."). From the
controller host of the base-cluster I can log into those guests
without problem ("ip netns exec .... ssh ..."). These three
guests are my midlevel-cluster. The controller has private IP
address 10.5.5.2, the compute hosts have 10.5.5.5 and 10.5.5.6.
3. I log into the midlevel-cluster hosts and download and install
DevStack ("stable/havana") on them. The stack.sh script runs
without error or problem.
4. After stack.sh has run, I can see that a private and public
network have been created for that DevStack installation. On the
controller I also set some security group rules, to allow ICMP
and SSH to any guest instances that should be created within
DevStack.
5. I now create a guest on that DevStack cluster ("nova boot... "
again). This is my toplevel-guest. I can see ("nova list...")
that the guest has booted and that an IP address has been
assigned, but no matter what I do, I cannot ping or login to
those toplevel-guests. I use the qrouter namespace to attempt
this ("ip netns exec qrouter-...."), but no luck.
Some information, which might be useful for trouble shooting:
On the midlevel-cluster hosts, I use these localrc files to install
DevStack (I'm just looking for simple GRE networking). It's a multi-node
install, so first, here is the localrc for the controller:
# Passwords and tokens
ADMIN_PASSWORD=password
MYSQL_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=tokentoken
# Logging, screen, devstack behavior
API_RATE_LIMIT=False
VERBOSE=True
DEBUG=True
LOGFILE=/home/ubuntu/tempest_run/workspace/stack.sh.log
USE_SCREEN=True
SCREEN_LOGDIR=/home/ubuntu/tempest_run/workspace
RECLONE=Yes
OFFLINE=False
LIBVIRT_TYPE=kvm
# Services
disable_service n-net
disable_service n-cpu
enable_service q-svc
enable_service q-agt
enable_service q-l3
enable_service q-meta
enable_service q-lbaas
enable_service q-dhcp
enable_service tempest
enable_service neutron
# Networking
ENABLE_TENANT_TUNNELS=True
Q_AGENT_EXTRA_AGENT_OPTS=(tunnel_type=gre)
Q_AGENT_EXTRA_OVS_OPTS=(tenant_network_type=gre)
Q_SRV_EXTRA_OPTS=(tenant_network_type=gre)
Q_USE_NAMESPACE=True
Q_USE_SECGROUP=True
Here is the localrc for the compute hosts:
# Passwords and tokens
ADMIN_PASSWORD=password
MYSQL_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=tokentoken
# Logging, screen, devstack behavior
VERBOSE=True
DEBUG=True
LOGFILE=/home/ubuntu/tempest_run/workspace/stack.sh.log
USE_SCREEN=True
SCREEN_LOGDIR=/home/ubuntu/tempest_run/workspace
RECLONE=Yes
OFFLINE=False
LIBVIRT_TYPE=kvm
# Controller connection
HOST_IP=10.5.5.5
SERVICE_HOST=10.5.5.2
MYSQL_HOST=10.5.5.2
RABBIT_HOST=10.5.5.2
Q_HOST=10.5.5.2
GLANCE_HOSTPORT=10.5.5.2:9292
# Services
ENABLED_SERVICES=n-cpu,rabbit,neutron,q-agt
# Networking
ENABLE_TENANT_TUNNELS=True
Q_AGENT_EXTRA_AGENT_OPTS=(tunnel_type=gre)
Q_AGENT_EXTRA_OVS_OPTS=(tenant_network_type=gre)
Q_USE_NAMESPACE=True
Q_USE_SECGROUP=True
After stack.sh has run to completion, I get the following on the
DevStack controller:
$ ip netns qrouter-293c2395-3a05-4ad7-99e5-e5b1ebb80a35
$ neutron net-list
+--------------------------------------+---------+------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+------------------------------------------------------+
| 789cb9ce-9f91-4d0b-9069-6eb5b808bdfc | public | d206940f-5daf-464d-bf28-4dac527aba06 172.24.4.224/28 |
| c8f179ba-6675-49f3-92ba-6e58b38f59c1 | private | b2bc098b-2b87-46a5-bb5e-7a75d9520c17 10.0.0.0/24 |
+--------------------------------------+---------+------------------------------------------------------+
I create a guest instance (toplevel-guest), like so:
$ nova boot --image e09876f9-c755-48a8-ada7-c658f8736a9e --flavor 1 --key-name mykey foobar
With "nova list" I see:
$ nova list
+--------------------------------------+--------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------+--------+------------+-------------+---------------------+
| 8794d3fd-8596-4ddb-bf20-d823b9804f0d | foobar | ACTIVE | - | Running | public=172.24.4.227 |
+--------------------------------------+--------+--------+------------+-------------+---------------------+
The following routers are known to neutron:
$ neutron router-list
+--------------------------------------+---------+-----------------------------------------------------------------------------+
| id | name | external_gateway_info |
+--------------------------------------+---------+-----------------------------------------------------------------------------+
| 293c2395-3a05-4ad7-99e5-e5b1ebb80a35 | router1 | {"network_id": "789cb9ce-9f91-4d0b-9069-6eb5b808bdfc", "enable_snat": true} |
+--------------------------------------+---------+-----------------------------------------------------------------------------+
The following ports are known to neutron:
$ neutron port-list
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
| 22126185-ae99-42d0-8876-bef9a96ff5a1 | | fa:16:3e:9d:be:fb | {"subnet_id": "d206940f-5daf-464d-bf28-4dac527aba06", "ip_address": "172.24.4.227"} |
| 8914b69b-b9f7-4555-bbcb-5af8ae0d340c | | fa:16:3e:0d:fd:d2 | {"subnet_id": "b2bc098b-2b87-46a5-bb5e-7a75d9520c17", "ip_address": "10.0.0.1"} |
| d5f45982-e7fa-49a7-a8b3-b3e7b93a227c | | fa:16:3e:2a:6f:ff | {"subnet_id": "d206940f-5daf-464d-bf28-4dac527aba06", "ip_address": "172.24.4.226"} |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
I can ping the address 172.24.4.226 (presumably the router), but not the address of the guest (172.24.4.227).
$ ping 172.24.4.227
PING 172.24.4.227 (172.24.4.227) 56(84) bytes of data.
From 172.24.4.225 icmp_seq=1 Destination Host Unreachable
From 172.24.4.225 icmp_seq=2 Destination Host Unreachable
From 172.24.4.225 icmp_seq=3 Destination Host Unreachable
Even if I use the qrouter namespace, it still doesn't work:
$ sudo ip netns exec qrouter-293c2395-3a05-4ad7-99e5-e5b1ebb80a35 ping 172.24.4.227
PING 172.24.4.227 (172.24.4.227) 56(84) bytes of data.
From 172.24.4.226 icmp_seq=1 Destination Host Unreachable
From 172.24.4.226 icmp_seq=2 Destination Host Unreachable
From 172.24.4.226 icmp_seq=3 Destination Host Unreachable
In my desparation, I tried security groups in nova as well as neutron.
The nova rules were added like this
$ nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
$ nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
The neutron rules like this:
$ neutron security-group-rule-create --protocol tcp --port-range-min 22 \
--port-range-max 22 --direction ingress default
$ neutron security-group-rule-create --protocol icmp \
--direction ingress foogroup
But in both cases, no luck.
I also tried bringing up the guest NIC on the private network. Still no
luck.
On the DevStack controller host, the OVS config looks like this:
$ sudo ovs-vsctl show
2a2472f2-eebc-4214-8a57-bfc05a21ae26
Bridge br-int
Port br-int
Interface br-int
type: internal
Port "qr-8914b69b-b9"
tag: 1
Interface "qr-8914b69b-b9"
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Port "gre-10.5.5.5"
Interface "gre-10.5.5.5"
type: gre
options: {in_key=flow, local_ip="10.5.5.2", out_key=flow, remote_ip="10.5.5.5"}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Bridge br-ex
Port "qg-d5f45982-e7"
Interface "qg-d5f45982-e7"
type: internal
Port br-ex
Interface br-ex
type: internal
ovs_version: "1.4.6"
On the DevStack compute hosts, it's like this:
$ sudo ovs-vsctl show
9b280f62-130c-4c12-89ea-0bc2fa22156e
Bridge br-int
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port br-int
Interface br-int
type: internal
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Port "gre-10.5.5.2"
Interface "gre-10.5.5.2"
type: gre
options: {in_key=flow, local_ip="10.5.5.5", out_key=flow, remote_ip="10.5.5.2"}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
ovs_version: "1.4.6"
The network interfaces on the DevStack controller are like this:
$ ifconfig -a
br-ex Link encap:Ethernet HWaddr 96:bf:ae:d5:a0:44
inet addr:172.24.4.225 Bcast:0.0.0.0 Mask:255.255.255.240
inet6 addr: fe80::94bf:aeff:fed5:a044/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:468 (468.0 B) TX bytes:468 (468.0 B)
br-int Link encap:Ethernet HWaddr 62:28:61:90:0b:47
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:468 (468.0 B) TX bytes:0 (0.0 B)
br-tun Link encap:Ethernet HWaddr 0e:f3:a7:96:91:43
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 Link encap:Ethernet HWaddr fa:16:3e:dc:7e:82
inet addr:10.5.5.2 Bcast:10.5.5.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fedc:7e82/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:438220 errors:0 dropped:0 overruns:0 frame:0
TX packets:180410 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:580740049 (580.7 MB) TX bytes:14956005 (14.9 MB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:84427 errors:0 dropped:0 overruns:0 frame:0
TX packets:84427 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:51372553 (51.3 MB) TX bytes:51372553 (51.3 MB)
On the compute hosts, they are like this:
$ ifconfig -a
br-int Link encap:Ethernet HWaddr c2:25:18:4b:8b:49
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
br-tun Link encap:Ethernet HWaddr 1e:27:74:ee:a1:45
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 Link encap:Ethernet HWaddr fa:16:3e:07:fd:48
inet addr:10.5.5.5 Bcast:10.5.5.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe07:fd48/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:309885 errors:0 dropped:0 overruns:0 frame:0
TX packets:134774 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:422666082 (422.6 MB) TX bytes:10489797 (10.4 MB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:159 errors:0 dropped:0 overruns:0 frame:0
TX packets:159 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:10580 (10.5 KB) TX bytes:10580 (10.5 KB)
virbr0 Link encap:Ethernet HWaddr 9a:21:30:f2:8a:3b
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
If anyone has any idea what I could possibly be doing wrong, I would be
very grateful for any explanation.
Thank you very much...
Juergen
More information about the Openstack
mailing list