[Openstack] instance cannot access external network (folsom quantum)
ZhiQiang Fan
aji.zqfan at gmail.com
Fri Dec 14 06:07:58 UTC 2012
control node (also act as network node): eth0 192.168.32.18 eth0:0
10.0.0.3 eth0:1(br-ex bridge) 192.168.32.129
compute node: eth0 192.168.32.19 eth0:0 10.0.0.4
fixed ip for instance: 10.0.18.0/24
floating ip for instance: 192.168.32.130-192.168.32.135 range
192.168.32.128/24 gateway 192.168.32.1
quamtum plugin: openvswitch
when instance ping a host in 192.168.32.x, host reply with
destination=10.0.18.x, so i think snat does not act well.
i can ping from 192.168.32.x to instance's floating ip (192.168.32.13x)
more details listed below:
**
information generated by command line
control node:
shell>ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP qlen 1000
link/ether 6c:f0:49:0b:e1:a6 brd ff:ff:ff:ff:ff:ff
inet 192.168.32.18/24 brd 192.168.32.255 scope global eth0
inet 10.0.0.3/24 brd 10.0.0.255 scope global eth0:0
inet6 fe80::6ef0:49ff:fe0b:e1a6/64 scope link
valid_lft forever preferred_lft forever
4: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN
link/ether be:22:4e:37:1f:4e brd ff:ff:ff:ff:ff:ff
5: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UNKNOWN
link/ether 22:5f:e0:e0:97:45 brd ff:ff:ff:ff:ff:ff
inet 192.168.32.129/24 scope global br-ex
9: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN
link/ether ee:9e:44:8e:59:47 brd ff:ff:ff:ff:ff:ff
34: tapafa410e4-d2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UNKNOWN
link/ether fa:16:3e:9a:10:c4 brd ff:ff:ff:ff:ff:ff
inet 10.0.18.2/24 brd 10.0.18.255 scope global tapafa410e4-d2
inet6 fe80::f816:3eff:fe9a:10c4/64 scope link
valid_lft forever preferred_lft forever
35: qr-b17d537e-27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UNKNOWN
link/ether fa:16:3e:cf:28:9f brd ff:ff:ff:ff:ff:ff
inet 10.0.18.1/24 brd 10.0.18.255 scope global qr-b17d537e-27
inet6 fe80::f816:3eff:fecf:289f/64 scope link
valid_lft forever preferred_lft forever
36: qg-1a968e33-e7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UNKNOWN
link/ether fa:16:3e:a8:f3:a0 brd ff:ff:ff:ff:ff:ff
inet 192.168.32.130/24 brd 192.168.32.255 scope global qg-1a968e33-e7
inet6 fe80::f816:3eff:fea8:f3a0/64 scope link
valid_lft forever preferred_lft forever
******************************************************************************************************
shell>route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.32.1 0.0.0.0 UG 0 0 0 eth0
0.0.0.0 192.168.32.1 0.0.0.0 UG 100 0 0 eth0
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.0.18.0 0.0.0.0 255.255.255.0 U 0 0
0 tapafa410e4-d2
10.0.18.0 0.0.0.0 255.255.255.0 U 0 0
0 qr-b17d537e-27
192.168.32.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.32.0 0.0.0.0 255.255.255.0 U 0 0 0 br-ex
192.168.32.0 0.0.0.0 255.255.255.0 U 0 0
0 qg-1a968e33-e7
**************************************************************************************************
shell>ovs-vsctl show
7705db6e-9363-41fb-8d6a-f47ffdfa90a6
Bridge br-int
Port "tapafa410e4-d2"
tag: 13
Interface "tapafa410e4-d2"
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port br-int
Interface br-int
type: internal
Port "qr-b17d537e-27"
tag: 13
Interface "qr-b17d537e-27"
type: internal
Bridge br-tun
Port "gre-2"
Interface "gre-2"
type: gre
options: {in_key=flow, out_key=flow, remote_ip="192.168.32.19"}
Port "gre-4"
Interface "gre-4"
type: gre
options: {in_key=flow, out_key=flow, remote_ip="10.0.0.4"}
Port br-tun
Interface br-tun
type: internal
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "gre-1"
Interface "gre-1"
type: gre
options: {in_key=flow, out_key=flow, remote_ip="192.168.32.18"}
Bridge br-ex
Port "qg-1a968e33-e7"
Interface "qg-1a968e33-e7"
type: internal
Port br-ex
Interface br-ex
type: internal
Port "eth0:1"
Interface "eth0:1"
ovs_version: "1.4.0+build0"
***********************************************************************************************
shell>iptables-save
# Generated by iptables-save v1.4.12 on Fri Dec 14 13:55:36 2012
*nat
:PREROUTING ACCEPT [159:16180]
:INPUT ACCEPT [139:13069]
:OUTPUT ACCEPT [893:55564]
:POSTROUTING ACCEPT [0:0]
:nova-api-OUTPUT - [0:0]
:nova-api-POSTROUTING - [0:0]
:nova-api-PREROUTING - [0:0]
:nova-api-float-snat - [0:0]
:nova-api-snat - [0:0]
:nova-postrouting-bottom - [0:0]
:quantum-l3-agent-OUTPUT - [0:0]
:quantum-l3-agent-POSTROUTING - [0:0]
:quantum-l3-agent-PREROUTING - [0:0]
:quantum-l3-agent-float-snat - [0:0]
:quantum-l3-agent-snat - [0:0]
:quantum-postrouting-bottom - [0:0]
-A PREROUTING -j quantum-l3-agent-PREROUTING
-A PREROUTING -j nova-api-PREROUTING
-A OUTPUT -j quantum-l3-agent-OUTPUT
-A OUTPUT -j nova-api-OUTPUT
-A POSTROUTING -j quantum-l3-agent-POSTROUTING
-A POSTROUTING -j quantum-postrouting-bottom
-A POSTROUTING -j nova-api-POSTROUTING
-A POSTROUTING -j nova-postrouting-bottom
-A nova-api-snat -j nova-api-float-snat
-A nova-postrouting-bottom -j nova-api-snat
-A quantum-l3-agent-POSTROUTING ! -i qg-1a968e33-e7 ! -o
qg-1a968e33-e7 -m conntrack ! --ctstate DNAT -j ACCEPT
-A quantum-l3-agent-POSTROUTING -s 10.0.18.0/24 -d 192.168.32.18/32 -j ACCEPT
-A quantum-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp
--dport 80 -j DNAT --to-destination 192.168.32.18:8775
-A quantum-l3-agent-snat -j quantum-l3-agent-float-snat
-A quantum-l3-agent-snat -s 10.0.18.0/24 -j SNAT --to-source 192.168.32.130
-A quantum-postrouting-bottom -j quantum-l3-agent-snat
COMMIT
# Completed on Fri Dec 14 13:55:36 2012
# Generated by iptables-save v1.4.12 on Fri Dec 14 13:55:36 2012
*filter
:INPUT ACCEPT [132266:27711233]
:FORWARD ACCEPT [3:196]
:OUTPUT ACCEPT [132285:28003590]
:nova-api-FORWARD - [0:0]
:nova-api-INPUT - [0:0]
:nova-api-OUTPUT - [0:0]
:nova-api-local - [0:0]
:nova-filter-top - [0:0]
:quantum-filter-top - [0:0]
:quantum-l3-agent-FORWARD - [0:0]
:quantum-l3-agent-INPUT - [0:0]
:quantum-l3-agent-OUTPUT - [0:0]
:quantum-l3-agent-local - [0:0]
-A INPUT -j quantum-l3-agent-INPUT
-A INPUT -j nova-api-INPUT
-A INPUT -p gre -j ACCEPT
-A FORWARD -j quantum-filter-top
-A FORWARD -j quantum-l3-agent-FORWARD
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-api-FORWARD
-A OUTPUT -j quantum-filter-top
-A OUTPUT -j quantum-l3-agent-OUTPUT
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-api-OUTPUT
-A nova-api-INPUT -d 192.168.32.18/32 -p tcp -m tcp --dport 8775 -j ACCEPT
-A nova-filter-top -j nova-api-local
-A quantum-filter-top -j quantum-l3-agent-local
-A quantum-l3-agent-INPUT -d 192.168.32.18/32 -p tcp -m tcp --dport
8775 -j ACCEPT
COMMIT
# Completed on Fri Dec 14 13:55:36 2012
**
information generated by command line
compute node
shell>ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP qlen 1000
link/ether 90:2b:34:18:e2:e2 brd ff:ff:ff:ff:ff:ff
inet 192.168.32.19/24 brd 192.168.32.255 scope global eth0
inet 10.0.0.4/24 brd 10.0.0.255 scope global eth0:0
inet6 fe80::922b:34ff:fe18:e2e2/64 scope link
valid_lft forever preferred_lft forever
4: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
link/ether d6:af:db:bd:68:4a brd ff:ff:ff:ff:ff:ff
6: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
link/ether 5a:35:be:ae:37:47 brd ff:ff:ff:ff:ff:ff
11: qbrb348eeea-ea: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP
link/ether 6a:88:1f:75:0a:c0 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5cff:17ff:fece:93cc/64 scope link
valid_lft forever preferred_lft forever
12: qvob348eeea-ea: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP qlen 1000
link/ether ae:41:2b:cc:5c:66 brd ff:ff:ff:ff:ff:ff
inet6 fe80::ac41:2bff:fecc:5c66/64 scope link
valid_lft forever preferred_lft forever
13: qvbb348eeea-ea: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast master qbrb348eeea-ea state UP qlen 1000
link/ether 6a:88:1f:75:0a:c0 brd ff:ff:ff:ff:ff:ff
inet6 fe80::6888:1fff:fe75:ac0/64 scope link
valid_lft forever preferred_lft forever
14: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master qbrb348eeea-ea state UNKNOWN qlen 500
link/ether fe:16:3e:5e:ba:01 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fe5e:ba01/64 scope link
valid_lft forever preferred_lft forever
******************************************************************************
shell>route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.32.1 0.0.0.0 UG 100 0 0 eth0
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.32.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
*********************************************************************************************
shell>ovs-vsctl show
4eecf274-60a2-4f2f-aa0d-639357b08557
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Port "gre-3"
Interface "gre-3"
type: gre
options: {in_key=flow, out_key=flow, remote_ip="10.0.0.3"}
Port "gre-2"
Interface "gre-2"
type: gre
options: {in_key=flow, out_key=flow, remote_ip="192.168.32.19"}
Port "gre-1"
Interface "gre-1"
type: gre
options: {in_key=flow, out_key=flow, remote_ip="192.168.32.18"}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Bridge br-int
Port "qvo63414f72-89"
tag: 1
Interface "qvo63414f72-89"
Port "qvo54683c71-42"
tag: 4095
Interface "qvo54683c71-42"
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port br-int
Interface br-int
type: internal
Port "qvo1cb0381f-45"
tag: 4095
Interface "qvo1cb0381f-45"
Port "qvob348eeea-ea"
tag: 2
Interface "qvob348eeea-ea"
ovs_version: "1.4.0+build0"
************************************************************************************
shell>iptables-save
# Generated by iptables-save v1.4.12 on Fri Dec 14 14:01:28 2012
*nat
:PREROUTING ACCEPT [603:70199]
:INPUT ACCEPT [470:49552]
:OUTPUT ACCEPT [279:17279]
:POSTROUTING ACCEPT [352:33179]
:nova-api-metadat-OUTPUT - [0:0]
:nova-api-metadat-POSTROUTING - [0:0]
:nova-api-metadat-PREROUTING - [0:0]
:nova-api-metadat-float-snat - [0:0]
:nova-api-metadat-snat - [0:0]
:nova-compute-OUTPUT - [0:0]
:nova-compute-POSTROUTING - [0:0]
:nova-compute-PREROUTING - [0:0]
:nova-compute-float-snat - [0:0]
:nova-compute-snat - [0:0]
:nova-postrouting-bottom - [0:0]
-A PREROUTING -j nova-compute-PREROUTING
-A PREROUTING -j nova-api-metadat-PREROUTING
-A OUTPUT -j nova-compute-OUTPUT
-A OUTPUT -j nova-api-metadat-OUTPUT
-A POSTROUTING -j nova-compute-POSTROUTING
-A POSTROUTING -j nova-api-metadat-POSTROUTING
-A POSTROUTING -j nova-postrouting-bottom
-A nova-api-metadat-snat -j nova-api-metadat-float-snat
-A nova-compute-snat -j nova-compute-float-snat
-A nova-postrouting-bottom -j nova-compute-snat
-A nova-postrouting-bottom -j nova-api-metadat-snat
COMMIT
# Completed on Fri Dec 14 14:01:28 2012
# Generated by iptables-save v1.4.12 on Fri Dec 14 14:01:28 2012
*filter
:INPUT ACCEPT [13145:5669860]
:FORWARD ACCEPT [255:41943]
:OUTPUT ACCEPT [11296:4412936]
:nova-api-metadat-FORWARD - [0:0]
:nova-api-metadat-INPUT - [0:0]
:nova-api-metadat-OUTPUT - [0:0]
:nova-api-metadat-local - [0:0]
:nova-compute-FORWARD - [0:0]
:nova-compute-INPUT - [0:0]
:nova-compute-OUTPUT - [0:0]
:nova-compute-inst-31 - [0:0]
:nova-compute-local - [0:0]
:nova-compute-provider - [0:0]
:nova-compute-sg-fallback - [0:0]
:nova-filter-top - [0:0]
-A INPUT -j nova-compute-INPUT
-A INPUT -p gre -j ACCEPT
-A INPUT -j nova-api-metadat-INPUT
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-compute-FORWARD
-A FORWARD -j nova-api-metadat-FORWARD
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-compute-OUTPUT
-A OUTPUT -j nova-api-metadat-OUTPUT
-A nova-api-metadat-INPUT -d 192.168.32.19/32 -p tcp -m tcp --dport
8775 -j ACCEPT
-A nova-compute-inst-31 -m state --state INVALID -j DROP
-A nova-compute-inst-31 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A nova-compute-inst-31 -j nova-compute-provider
-A nova-compute-inst-31 -s 10.0.18.2/32 -p udp -m udp --sport 67
--dport 68 -j ACCEPT
-A nova-compute-inst-31 -s 10.0.18.0/24 -j ACCEPT
-A nova-compute-inst-31 -p tcp -m tcp --dport 22 -j ACCEPT
-A nova-compute-inst-31 -p icmp -j ACCEPT
-A nova-compute-inst-31 -j nova-compute-sg-fallback
-A nova-compute-local -d 10.0.18.3/32 -j nova-compute-inst-31
-A nova-compute-sg-fallback -j DROP
-A nova-filter-top -j nova-compute-local
-A nova-filter-top -j nova-api-metadat-local
COMMIT
# Completed on Fri Dec 14 14:01:28 2012
**
information generated by command line
instance
shell>ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP qlen 1000
link/ether fa:16:3e:5e:ba:01 brd ff:ff:ff:ff:ff:ff
inet 10.0.18.3/24 brd 10.0.18.255 scope global eth0
inet6 fe80::f816:3eff:fe5e:ba01/64 scope link tentative flags 08
valid_lft forever preferred_lft forever
******************************************************************************************
shell>route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.18.1 0.0.0.0 UG 0 0 0 eth0
10.0.18.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
*****************************************************************************************
cirros has not installed iptables and i cannot access external network,
On 12/13/12, Gary Kotton <gkotton at redhat.com> wrote:
> On 12/13/2012 12:07 PM, ZhiQiang Fan wrote:
>> i can ping and ssh into instance with private ip and floating ip
>> instance can ping the control node ip, but cannot ping the compute
>> node and any external network
>
> In order to be able to help would it be possible that you provide IP
> addresses and maybe a bit of understanding about your topology.
>
> Basically is there a route from the VM ip address to the IP address of
> the compute node?
>
> In addition to this can you please let us know which plugin you are using?
>
> Thanks
> Gary
>>
>> i have installed quantum in the control node host, and it only got 1
>> nic (same as compute node), and use eth0:0 and eth0:1 to vitualize 2
>> other nic (eth0:0 on compute node)
>>
>> i use tcpdump on control node and compute node to monitor package from
>> instance, actually compute node will reply the icmp package but with
>> destination of instance private ip, since compute node has no route to
>> that network, it failed and no package receive on control node nic.
>> but when i add route via control node, it can reply to insance as
>> expected
>> then i use tcpdump on control node and instance to monitor package to
>> the floating ip, instance got nothing but control node captured the
>> package and reply it instead of instance
>>
>> so i think the problem may be that the control node will not modify
>> the source ip when forwad the icmp package, more exactly, the nat
>> functionality is not enabled?
>>
>> and i try some other command such as "iptables -t nat -A POSTROUTING
>> -o eth0 -j MASQUERADE" but it is not working
>>
>> i'll paste some output if anyone needs
>> thanks
>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack at lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help : https://help.launchpad.net/ListHelp
>
>
More information about the Openstack
mailing list