[Openstack] VM Issues on Grizzly Install on Ubuntu 12.04
Darragh O'Reilly
dara2002-openstack at yahoo.com
Wed May 29 20:53:09 UTC 2013
Hi Farhan,
I was able to reproduce this with curl from the cirros 0.3.1 that supports ssl.
cirros$ curl -L github.com # -L follow redirects
it just hangs and I get these ICMPs on the netnode's physical nic.
20:33:10.811485 IP (tos 0xc0, ttl 63, id 13647, offset 0, flags [none], proto ICMP (1), length 576)
192.168.101.2 > 204.232.175.90: ICMP 192.168.101.2 unreachable - need to frag (mtu 1454), length 556
IP (tos 0x0, ttl 51, id 54729, offset 0, flags [DF], proto TCP (6), length 1500)
204.232.175.90.443 > 192.168.101.2.41237: Flags [.], seq 1:1449, ack 225, win 7, options [nop,nop,TS val 4208725487 ecr 171322], length 1448
So I reduced the mtu from the default 1500 to 1454 on the instance and now 'curl -L github.com' works
cirros$ sudo ip link set mtu 1454 dev eth0
Will need to look into this more. Maybe to do with the GRE tunnels (+~20bytes?) or iptables. Anyway try reducing the mtu for now.
Darragh.
----- Original Message -----
> From: Farhan Patwa <Farhan.Patwa at utsa.edu>
> To: Darragh O'Reilly <dara2002-openstack at yahoo.com>; OpenStack Maillist <openstack at lists.launchpad.net>
> Cc:
> Sent: Wednesday, 29 May 2013, 18:14
> Subject: Re: [Openstack] VM Issues on Grizzly Install on Ubuntu 12.04
>
> Hi Darragh,
> Thank you soo Much! That was it! Now I am able to connect to the VM with
> no issues.
>
> But I am back to another network issue I had when I had Folsom installed
> on the same setup.
> I would really appreciate if you can provide any pointers here.
>
>
> I able to spawn VM get IP, set floating IP and now am trying to do some
> development within the VM.
> I am unable to connect to certain sites and ports:
> git clone https://github.com/openstack-dev/devstack.git - <-- This just
> times out.
>
> ###########################################################################
> #############
> This is what works:
> Wget google.com
> Wget openstack.com
> ###########################################################################
> #############
> This is what hangs and times out:
>
> Wget yahoo.com
> Wget paypal.com
> Wget facebook.com
> Wget github.com
> ubuntu at fpatwa-1:~$ wget github.com
> --2013-05-10 19:08:19-- http://github.com/
> Resolving github.com (github.com)... 204.232.175.90
> Connecting to github.com (github.com)|204.232.175.90|:80... connected.
> HTTP request sent, awaiting response... 301 Moved Permanently
> Location: https://github.com/ [following]
> --2013-05-10 19:08:20-- https://github.com/
> Connecting to github.com (github.com)|204.232.175.90|:443... connected.
>
> ###########################################################################
> #############
>
> The same commands works on the network node.
>
>
> The pattern that I can see is that any SSL website fails (port 443) but
> then something like yahoo fails also and its at port 80.
>
>
> Here are my security rules:
> +-------------+-----------+---------+-----------+--------------+
> | IP Protocol | From Port | To Port | IP Range | Source Group |
> +-------------+-----------+---------+-----------+--------------+
> | icmp | -1 | -1 | 0.0.0.0/0 | |
> | tcp | 1 | 65535 | 0.0.0.0/0 | |
> | tcp | 22 | 22 | 0.0.0.0/0 | |
> | udp | 1 | 65535 | 0.0.0.0/0 | |
> +-------------+-----------+---------+-----------+--------------+
>
>
>
> I have messed around with all kinds of combinations of security rules but
> no luck so far.
>
> Thanks,
>
> -Farhan.
>
>
>
>
> On 5/28/13 3:28 PM, "Darragh O'Reilly"
> <dara2002-openstack at yahoo.com>
> wrote:
>
>> Hi,
>>
>> the ping error "connect: Network is unreachable" means a route
> could not
>> be found.
>>
>> The gateway 10.245.124.253 for the external subnet is not in the subnet
>> CIDR 10.245.124.64/26.
>>
>>
>> So I guess a default route was not setup here:
>> netnode$ ip netns exec <router ns> route -n
>>
>> You will need to create the subnet with a CIDR that includes the gateway
>> ip - something like this:
>> quantum subnet-create <ext-net-id> 10.245.124.192/26 --gateway
>> 10.245.124.253 --enable_dhcp False
>>
>> Darragh.
>>
>>
>> ----- Original Message -----
>>> From: Farhan Patwa <Farhan.Patwa at utsa.edu>
>>> To: Darragh OReilly <darragh.oreilly at yahoo.com>; OpenStack
> Maillist
>>> <openstack at lists.launchpad.net>
>>> Cc:
>>> Sent: Tuesday, 28 May 2013, 19:52
>>> Subject: Re: [Openstack] VM Issues on Grizzly Install on Ubuntu 12.04
>>>
>>> Hi Darragh,
>>> Thanks a lot for your reply and suggestions.
>>> I am not able to ping the gateway ip from the namespace.
>>> Also eth0 is up but br-ex has unknown state?
>>>
>>> #########################################################################
>>> ##
>>> #######################
>>>
>>> root at openstack-2:~# ip link
>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state
> UNKNOWN
>>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> state UP
>>> qlen
>>> 1000
>>> link/ether 78:2b:cb:27:1f:c8 brd ff:ff:ff:ff:ff:ff
>>> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> state UP
>>> qlen
>>> 1000
>>> link/ether 78:2b:cb:27:1f:c9 brd ff:ff:ff:ff:ff:ff
>>> 4: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue
>>> state
>>> UNKNOWN
>>> link/ether f2:3b:f7:1b:b0:46 brd ff:ff:ff:ff:ff:ff
>>> 6: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state
>>> UNKNOWN
>>> link/ether 78:2b:cb:27:1f:c8 brd ff:ff:ff:ff:ff:ff
>>> 32: br-tun: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue
>>> state
>>> UNKNOWN
>>> link/ether 7e:6c:65:0f:c9:43 brd ff:ff:ff:ff:ff:ff
>>>
>>> #########################################################################
>>> ##
>>> #######################
>>>
>>>
>>> Here is the result of the tcpdump as ping is being done:
>>>
>>>
>>> #########################################################################
>>> ##
>>> #######################
>>>
>>> root at openstack-2:~# ip netns exec
>>> qrouter-32f35fb4-f9f1-4817-8818-fff832f73810 ping -c1 10.245.124.253
>>> connect: Network is unreachable
>>>
>>> root at openstack-2:~# tcpdump -nei eth0
>>> tcpdump: WARNING: eth0: no IPv4 address assigned
>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>> decode
>>> listening on eth0, link-type EN10MB (Ethernet), capture size 65535
> bytes
>>> 13:46:31.399055 00:26:88:7a:40:87 > 01:80:c2:00:00:00, 802.3, length
> 60:
>>> LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03:
> STP
>>> 802.1w, Rapid STP, Flags [Proposal], bridge-id
>>> 8000.00:26:88:7a:40:81.8205, length 43
>>> 13:46:33.259195 c2:35:07:e7:b0:10 > ff:ff:ff:ff:ff:ff, ethertype ARP
>>> (0x0806), length 60: Reply 10.245.0.10 is-at c2:35:07:e7:b0:10, length
>>> 46
>>> 13:46:33.313988 00:26:88:7a:40:87 > 01:80:c2:00:00:00, 802.3, length
> 60:
>>> LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03:
> STP
>>> 802.1w, Rapid STP, Flags [Proposal], bridge-id
>>> 8000.00:26:88:7a:40:81.8205, length 43
>>>
>>> #########################################################################
>>> ##
>>> #######################
>>>
>>>
>>>
>>> The other information that you wanted is:
>>>
>>> #########################################################################
>>> ##
>>> #######################
>>>
>>> root at openstack-2:~# ip link
>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state
> UNKNOWN
>>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> state UP
>>> qlen
>>> 1000
>>> link/ether 78:2b:cb:27:1f:c8 brd ff:ff:ff:ff:ff:ff
>>> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> state UP
>>> qlen
>>> 1000
>>> link/ether 78:2b:cb:27:1f:c9 brd ff:ff:ff:ff:ff:ff
>>> 4: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue
>>> state
>>> UNKNOWN
>>> link/ether f2:3b:f7:1b:b0:46 brd ff:ff:ff:ff:ff:ff
>>> 6: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state
>>> UNKNOWN
>>> link/ether 78:2b:cb:27:1f:c8 brd ff:ff:ff:ff:ff:ff
>>> 32: br-tun: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue
>>> state
>>> UNKNOWN
>>> link/ether 7e:6c:65:0f:c9:43 brd ff:ff:ff:ff:ff:ff
>>>
>>> #########################################################################
>>> ##
>>> #######################
>>>
>>> root at openstack-2:~# ip netns exec
>>> qrouter-32f35fb4-f9f1-4817-8818-fff832f73810 ip address
>>> 25: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state
> UNKNOWN
>>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>> inet 127.0.0.1/8 scope host lo
>>> inet6 ::1/128 scope host
>>> valid_lft forever preferred_lft forever
>>> 39: qr-eebfe1cb-0f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
> qdisc
>>> noqueue state UNKNOWN
>>> link/ether fa:16:3e:08:16:19 brd ff:ff:ff:ff:ff:ff
>>> inet 50.50.1.1/24 brd 50.50.1.255 scope global qr-eebfe1cb-0f
>>> inet6 fe80::f816:3eff:fe08:1619/64 scope link
>>> valid_lft forever preferred_lft forever
>>> 40: qg-910fef3b-cb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
> qdisc
>>> noqueue state UNKNOWN
>>> link/ether fa:16:3e:e3:d5:fa brd ff:ff:ff:ff:ff:ff
>>> inet 10.245.124.65/26 brd 10.245.124.127 scope global
> qg-910fef3b-cb
>>> inet 10.245.124.67/32 brd 10.245.124.67 scope global qg-910fef3b-cb
>>> inet6 fe80::f816:3eff:fee3:d5fa/64 scope link
>>> valid_lft forever preferred_lft forever
>>>
>>> #########################################################################
>>> ##
>>> #######################
>>>
>>> root at openstack-2:~# quantum net-show
>>> 37d27ee8-36a9-4cdb-9966-9b5571526b41
>>> +---------------------------+--------------------------------------+
>>> | Field | Value |
>>> +---------------------------+--------------------------------------+
>>> | admin_state_up | True |
>>> | id | 37d27ee8-36a9-4cdb-9966-9b5571526b41 |
>>> | name | ext_net |
>>> | provider:network_type | gre |
>>> | provider:physical_network | |
>>> | provider:segmentation_id | 1 |
>>> | router:external | True |
>>> | shared | True |
>>> | status | ACTIVE |
>>> | subnets | dd6f08f5-bfbd-4bdb-b9e4-c5ca065f3750 |
>>> | tenant_id | 2990df1bd46c4dda915b43558d591a2f |
>>> +---------------------------+--------------------------------------+
>>>
>>> #########################################################################
>>> ##
>>> #######################
>>>
>>>
>>>
>>>
>>> root at openstack-2:~# quantum subnet-show
>>> dd6f08f5-bfbd-4bdb-b9e4-c5ca065f3750
>>>
>>> +------------------+-----------------------------------------------------
>>> +
>>> | Field | Value
>>> |
>>>
>>> +------------------+-----------------------------------------------------
>>> +
>>> | allocation_pools | {"start": "10.245.124.65",
>>> "end": "10.245.124.126"} |
>>> | cidr | 10.245.124.64/26
>>> |
>>> | dns_nameservers | 10.245.0.10
>>> |
>>> | enable_dhcp | False
>>> |
>>> | gateway_ip | 10.245.124.253
>>> |
>>> | host_routes |
>>> |
>>> | id | dd6f08f5-bfbd-4bdb-b9e4-c5ca065f3750
>>> |
>>> | ip_version | 4
>>> |
>>> | name |
>>> |
>>> | network_id | 37d27ee8-36a9-4cdb-9966-9b5571526b41
>>> |
>>> | tenant_id | 2990df1bd46c4dda915b43558d591a2f
>>> |
>>>
>>> +------------------+-----------------------------------------------------
>>> +
>>>
>>> #########################################################################
>>> ##
>>> #######################
>>>
>>>
>>>
>>> Thanks,
>>>
>>> -Farhan.
>>>
>>>
>>>
>>>
>>>
>>> On 5/27/13 4:08 AM, "Darragh OReilly"
>>> <darragh.oreilly at yahoo.com> wrote:
>>>
>>>>
>>>> I'd check the external network config first.
>>>>
>>>> You should be able to ping the external subnet's gateway from
> the
>>>> router
>>>> namespace.
>>>> This gateway should correspond to some real external
> gateway/router.
>>>>
>>>> quantum subnet-show <ext sub id> -c gateway_ip #
> 10.245.124.1 ?
>>>> ip netns exec <router-ns> ping -c1 <ext sub gateway>
>>>>
>>>> If that is not working use tcpdump as you ping. Br-ex is using
> eth0, is
>>>> eth0 up? tcpdump -nei eth0
>>>>
>>>>
>>>> If you are still having problems, post the above output and the
>>>> following:
>>>>
>>>> # network node
>>>> ip link
>>>> ip netns exec <router-ns> ip address
>>>>
>>>> quantum net-show <uuid of external net>
>>>> quantum subnet-show <uuid of external subnet>
>>>>
>>>>
>>>>> ________________________________
>>>>> From: Farhan Patwa <Farhan.Patwa at utsa.edu>
>>>>> To: OpenStack Maillist <openstack at lists.launchpad.net>
>>>>> Sent: Friday, 24 May 2013, 20:28
>>>>> Subject: [Openstack] VM Issues on Grizzly Install on Ubuntu
> 12.04
>>>>>
>>>>>
>>>>>
>>>>> Hello,
>>>>> I followed the following guide to install Grizzly release on
> 3-node
>>>>> setup.
>>>>>
>>>>> http://docs.openstack.org/grizzly/basic-install/apt/content/basic-insta
>>>>> ll
>>>>> _intro.html
>>>>>
>>>>>
>>>>> I am stuck at my last issue with Quantum networking (at least
> that¹s
>>>>> what I think).
>>>>> The VM instance comes up and gets the private IP and the
> metadata.
>>>>> Also I have assigned the floating IP to it but am not able to
> ping
>>>>> either IP except when I use:
>>>>>
>>>>>
>>>>> ip netns exec qrouter-32f35fb4-f9f1-4817-8818-fff832f73810 ping
>>>>> 50.50.1.3 <- fixed IP private network
>>>>> ip netns exec qrouter-32f35fb4-f9f1-4817-8818-fff832f73810 ping
>>>>> 10.24.124.4 <- floating IP external network
>>>>>
>>>>>
>>>>> Based on that I think the security rules are okay
>>>>> The router is tied to the specified tenant and using gateway of
> the
>>>>> external network.
>>>>> I think the issue is routing table or maybe firewall related
> but not
>>>>> sure how to debug this.
>>>>>
>>>>>
>>>>> Some details of my environment are below.
>>>>> Any one have any words of wisdom/guidance?
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>> -Farhan.
>>>>>
>>>>>
>>>>> Management Network: 192.168.0.0/24
>>>>> Data Network: 10.5.5.0/24
>>>>> External Network: 10.245.124.0/24
>>>>>
>>>>>
>>>>> Network Node: (192.168.0.2)
>>>>> ovs-vsctl show
>>>>> ea4fa894-5986-40f2-b10b-55eef2222408
>>>>> Bridge br-tun
>>>>> Port patch-int
>>>>> Interface patch-int
>>>>> type: patch
>>>>> options: {peer=patch-tun}
>>>>> Port "gre-1"
>>>>> Interface "gre-1"
>>>>> type: gre
>>>>> options: {in_key=flow, out_key=flow,
>>>>> remote_ip="192.168.0.3"}
>>>>> Port br-tun
>>>>> Interface br-tun
>>>>> type: internal
>>>>> Bridge br-int
>>>>> Port "tap3fca71a9-c8"
>>>>> tag: 4095
>>>>> Interface "tap3fca71a9-c8"
>>>>> type: internal
>>>>> Port patch-tun
>>>>> Interface patch-tun
>>>>> type: patch
>>>>> options: {peer=patch-int}
>>>>> Port "tap4b8a22a2-9c"
>>>>> tag: 4095
>>>>> Interface "tap4b8a22a2-9c"
>>>>> type: internal
>>>>> Port "tap633ed611-a9"
>>>>> tag: 1
>>>>> Interface "tap633ed611-a9"
>>>>> type: internal
>>>>> Port "qr-eebfe1cb-0f"
>>>>> tag: 1
>>>>> Interface "qr-eebfe1cb-0f"
>>>>> type: internal
>>>>> Port br-int
>>>>> Interface br-int
>>>>> type: internal
>>>>> Bridge br-ex
>>>>> Port "eth0"
>>>>> Interface "eth0"
>>>>> Port br-ex
>>>>> Interface br-ex
>>>>> type: internal
>>>>> Port "qg-910fef3b-cb"
>>>>> Interface "qg-910fef3b-cb"
>>>>> type: internal
>>>>> ovs_version: "1.4.0+build0"
>>>>>
>>>>>
>>>>> Kernel IP routing table
>>>>> Destination Gateway Genmask Flags Metric
> Ref
>>>>> Use
>>>>> Iface
>>>>> 0.0.0.0 192.168.0.253 0.0.0.0 UG 0 0
>>>>> 0
>>>>> eth1
>>>>> 10.5.5.0 0.0.0.0 255.255.255.0 U 0 0
>>>>> 0
>>>>> eth1
>>>>> 10.245.124.0 0.0.0.0 255.255.255.0 U 0 0
>>>>> 0
>>>>> br-ex
>>>>> 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0
>>>>> 0
>>>>> eth1
>>>>>
>>>>>
>>>>> Compute Node: (192.168.0.3)
>>>>> ovs-vsctl show
>>>>> f0fe78a5-dfd0-4f6b-87be-466dac0b4473
>>>>> Bridge br-tun
>>>>> Port patch-int
>>>>> Interface patch-int
>>>>> type: patch
>>>>> options: {peer=patch-tun}
>>>>> Port br-tun
>>>>> Interface br-tun
>>>>> type: internal
>>>>> Port "gre-2"
>>>>> Interface "gre-2"
>>>>> type: gre
>>>>> options: {in_key=flow, out_key=flow,
>>>>> remote_ip="192.168.0.2"}
>>>>> Bridge br-int
>>>>> Port patch-tun
>>>>> Interface patch-tun
>>>>> type: patch
>>>>> options: {peer=patch-int}
>>>>> Port br-int
>>>>> Interface br-int
>>>>> type: internal
>>>>> Port "tap6514a8cc-b2"
>>>>> tag: 1
>>>>> Interface "tap6514a8cc-b2"
>>>>> ovs_version: "1.4.0+build0"
>>>>>
>>>>>
>>>>> Kernel IP routing table
>>>>> Destination Gateway Genmask Flags Metric
> Ref
>>>>> Use
>>>>> Iface
>>>>> 0.0.0.0 192.168.0.253 0.0.0.0 UG 0 0
>>>>> 0
>>>>> eth1
>>>>> 10.5.5.0 0.0.0.0 255.255.255.0 U 0 0
>>>>> 0
>>>>> eth1
>>>>> 10.245.124.0 0.0.0.0 255.255.255.0 U 0 0
>>>>> 0
>>>>> eth0
>>>>> 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0
>>>>> 0
>>>>> eth1
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~openstack
>>>>> Post to : openstack at lists.launchpad.net
>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to : openstack at lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help : https://help.launchpad.net/ListHelp
>>>
>>
>
More information about the Openstack
mailing list