[Openstack] VM Issues on Grizzly Install on Ubuntu 12.04

Rahul Sharma rahulsharmaait at gmail.com
Thu May 30 05:57:14 UTC 2013


Hi Darragh,



Even I am facing the same issue of request getting timed out and even
updates getting hanged up for very long time. I followed your step of
reducing the MTU size from 1500 to 1454 and now everything works fine. I
tried this on Ubuntu instances.



This seems to be an issue with the Grizzly release. I had already started
email-thread earlier for this but was unable to find the root cause. Here
is the link to it:-

https://lists.launchpad.net/openstack/msg23993.html

Thank you for your suggestion of reducing the MTU size as it solved the
problem. You must file a bug for this so that this issue can be tracked.

Thanks and Regards
Rahul Sharma


On Thu, May 30, 2013 at 2:23 AM, Darragh O'Reilly <
dara2002-openstack at yahoo.com> wrote:

> Hi Farhan,
>
> I was able to reproduce this with curl from the cirros 0.3.1 that supports
> ssl.
>
> cirros$ curl -L github.com  # -L follow redirects
>
> it just hangs and I get these ICMPs on the netnode's physical nic.
>
> 20:33:10.811485 IP (tos 0xc0, ttl 63, id 13647, offset 0, flags [none],
> proto ICMP (1), length 576)
>     192.168.101.2 > 204.232.175.90: ICMP 192.168.101.2 unreachable - need
> to frag (mtu 1454), length 556
> IP (tos 0x0, ttl 51, id 54729, offset 0, flags [DF], proto TCP (6), length
> 1500)
>     204.232.175.90.443 > 192.168.101.2.41237: Flags [.], seq 1:1449, ack
> 225, win 7, options [nop,nop,TS val 4208725487 ecr 171322], length 1448
>
> So I reduced the mtu from the default 1500 to 1454 on the instance and now
> 'curl -L github.com' works
>
> cirros$ sudo ip link set mtu 1454 dev eth0
>
> Will need to look into this more. Maybe to do with the GRE tunnels
> (+~20bytes?) or iptables. Anyway try reducing the mtu for now.
>
> Darragh.
>
>
> ----- Original Message -----
> > From: Farhan Patwa <Farhan.Patwa at utsa.edu>
> > To: Darragh O'Reilly <dara2002-openstack at yahoo.com>; OpenStack Maillist
> <openstack at lists.launchpad.net>
> > Cc:
> > Sent: Wednesday, 29 May 2013, 18:14
> > Subject: Re: [Openstack] VM Issues on Grizzly Install on Ubuntu 12.04
> >
> > Hi Darragh,
> > Thank you soo Much! That was it! Now I am able to connect to the VM with
> > no issues.
> >
> > But I am back to another network issue I had when I had Folsom installed
> > on the same setup.
> > I would really appreciate if you can provide any pointers here.
> >
> >
> > I able to spawn VM get IP, set floating IP and now am trying to do some
> > development within the VM.
> > I am unable to connect to certain sites and ports:
> > git clone https://github.com/openstack-dev/devstack.git - <-- This just
> > times out.
> >
> >
> ###########################################################################
> > #############
> > This is what works:
> > Wget google.com
> > Wget openstack.com
> >
> ###########################################################################
> > #############
> > This is what hangs and times out:
> >
> > Wget yahoo.com
> > Wget paypal.com
> > Wget facebook.com
> > Wget github.com
> > ubuntu at fpatwa-1:~$ wget github.com
> > --2013-05-10 19:08:19--  http://github.com/
> > Resolving github.com (github.com)... 204.232.175.90
> > Connecting to github.com (github.com)|204.232.175.90|:80... connected.
> > HTTP request sent, awaiting response... 301 Moved Permanently
> > Location: https://github.com/ [following]
> > --2013-05-10 19:08:20--  https://github.com/
> > Connecting to github.com (github.com)|204.232.175.90|:443... connected.
> >
> >
> ###########################################################################
> > #############
> >
> > The same commands works on the network node.
> >
> >
> > The pattern that I can see is that any SSL website fails (port 443) but
> > then something like yahoo fails also and its at port 80.
> >
> >
> > Here are my security rules:
> > +-------------+-----------+---------+-----------+--------------+
> > | IP Protocol | From Port | To Port | IP Range  | Source Group |
> > +-------------+-----------+---------+-----------+--------------+
> > | icmp        | -1        | -1      | 0.0.0.0/0 |              |
> > | tcp         | 1         | 65535   | 0.0.0.0/0 |              |
> > | tcp         | 22        | 22      | 0.0.0.0/0 |              |
> > | udp         | 1         | 65535   | 0.0.0.0/0 |              |
> > +-------------+-----------+---------+-----------+--------------+
> >
> >
> >
> > I have messed around with all kinds of combinations of security rules but
> > no luck so far.
> >
> > Thanks,
> >
> > -Farhan.
> >
> >
> >
> >
> > On 5/28/13 3:28 PM, "Darragh O'Reilly"
> > <dara2002-openstack at yahoo.com>
> > wrote:
> >
> >> Hi,
> >>
> >> the ping error "connect: Network is unreachable" means a route
> > could not
> >> be found.
> >>
> >> The gateway 10.245.124.253 for the external subnet is not in the subnet
> >> CIDR 10.245.124.64/26.
> >>
> >>
> >> So I guess a default route was not setup here:
> >> netnode$ ip netns exec <router ns> route -n
> >>
> >> You will need to create the subnet with a CIDR that includes the gateway
> >> ip - something like this:
> >> quantum subnet-create <ext-net-id> 10.245.124.192/26 --gateway
> >> 10.245.124.253 --enable_dhcp False
> >>
> >> Darragh.
> >>
> >>
> >> ----- Original Message -----
> >>>  From: Farhan Patwa <Farhan.Patwa at utsa.edu>
> >>>  To: Darragh OReilly <darragh.oreilly at yahoo.com>; OpenStack
> > Maillist
> >>> <openstack at lists.launchpad.net>
> >>>  Cc:
> >>>  Sent: Tuesday, 28 May 2013, 19:52
> >>>  Subject: Re: [Openstack] VM Issues on Grizzly Install on Ubuntu 12.04
> >>>
> >>>  Hi Darragh,
> >>>  Thanks a lot for your reply and suggestions.
> >>>  I am not able to ping the gateway ip from the namespace.
> >>>  Also eth0 is up but br-ex has unknown state?
> >>>
> >>>
> #########################################################################
> >>> ##
> >>>  #######################
> >>>
> >>>  root at openstack-2:~# ip link
> >>>  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state
> > UNKNOWN
> >>>      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> >>>  2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> > state UP
> >>> qlen
> >>>  1000
> >>>      link/ether 78:2b:cb:27:1f:c8 brd ff:ff:ff:ff:ff:ff
> >>>  3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> > state UP
> >>> qlen
> >>>  1000
> >>>      link/ether 78:2b:cb:27:1f:c9 brd ff:ff:ff:ff:ff:ff
> >>>  4: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> > noqueue
> >>> state
> >>>  UNKNOWN
> >>>      link/ether f2:3b:f7:1b:b0:46 brd ff:ff:ff:ff:ff:ff
> >>>  6: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> > noqueue state
> >>>  UNKNOWN
> >>>      link/ether 78:2b:cb:27:1f:c8 brd ff:ff:ff:ff:ff:ff
> >>>  32: br-tun: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> > noqueue
> >>> state
> >>>  UNKNOWN
> >>>      link/ether 7e:6c:65:0f:c9:43 brd ff:ff:ff:ff:ff:ff
> >>>
> >>>
> #########################################################################
> >>> ##
> >>>  #######################
> >>>
> >>>
> >>>  Here is the result of the tcpdump as ping is being done:
> >>>
> >>>
> >>>
> #########################################################################
> >>> ##
> >>>  #######################
> >>>
> >>>  root at openstack-2:~# ip netns exec
> >>>  qrouter-32f35fb4-f9f1-4817-8818-fff832f73810 ping  -c1 10.245.124.253
> >>>  connect: Network is unreachable
> >>>
> >>>  root at openstack-2:~# tcpdump -nei eth0
> >>>  tcpdump: WARNING: eth0: no IPv4 address assigned
> >>>  tcpdump: verbose output suppressed, use -v or -vv for full protocol
> >>> decode
> >>>  listening on eth0, link-type EN10MB (Ethernet), capture size 65535
> > bytes
> >>>  13:46:31.399055 00:26:88:7a:40:87 > 01:80:c2:00:00:00, 802.3, length
> > 60:
> >>>  LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03:
> > STP
> >>>  802.1w, Rapid STP, Flags [Proposal], bridge-id
> >>>  8000.00:26:88:7a:40:81.8205, length 43
> >>>  13:46:33.259195 c2:35:07:e7:b0:10 > ff:ff:ff:ff:ff:ff, ethertype ARP
> >>>  (0x0806), length 60: Reply 10.245.0.10 is-at c2:35:07:e7:b0:10, length
> >>> 46
> >>>  13:46:33.313988 00:26:88:7a:40:87 > 01:80:c2:00:00:00, 802.3, length
> > 60:
> >>>  LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03:
> > STP
> >>>  802.1w, Rapid STP, Flags [Proposal], bridge-id
> >>>  8000.00:26:88:7a:40:81.8205, length 43
> >>>
> >>>
> #########################################################################
> >>> ##
> >>>  #######################
> >>>
> >>>
> >>>
> >>>  The other information that you wanted is:
> >>>
> >>>
> #########################################################################
> >>> ##
> >>>  #######################
> >>>
> >>>  root at openstack-2:~# ip link
> >>>  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state
> > UNKNOWN
> >>>  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> >>>  2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> > state UP
> >>> qlen
> >>>  1000
> >>>  link/ether 78:2b:cb:27:1f:c8 brd ff:ff:ff:ff:ff:ff
> >>>  3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> > state UP
> >>> qlen
> >>>  1000
> >>>  link/ether 78:2b:cb:27:1f:c9 brd ff:ff:ff:ff:ff:ff
> >>>  4: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> > noqueue
> >>> state
> >>>  UNKNOWN
> >>>  link/ether f2:3b:f7:1b:b0:46 brd ff:ff:ff:ff:ff:ff
> >>>  6: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> > noqueue state
> >>>  UNKNOWN
> >>>  link/ether 78:2b:cb:27:1f:c8 brd ff:ff:ff:ff:ff:ff
> >>>  32: br-tun: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> > noqueue
> >>> state
> >>>  UNKNOWN
> >>>  link/ether 7e:6c:65:0f:c9:43 brd ff:ff:ff:ff:ff:ff
> >>>
> >>>
> #########################################################################
> >>> ##
> >>>  #######################
> >>>
> >>>  root at openstack-2:~# ip netns exec
> >>>  qrouter-32f35fb4-f9f1-4817-8818-fff832f73810 ip address
> >>>  25: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state
> > UNKNOWN
> >>>      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> >>>      inet 127.0.0.1/8 scope host lo
> >>>      inet6 ::1/128 scope host
> >>>         valid_lft forever preferred_lft forever
> >>>  39: qr-eebfe1cb-0f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
> > qdisc
> >>>  noqueue state UNKNOWN
> >>>      link/ether fa:16:3e:08:16:19 brd ff:ff:ff:ff:ff:ff
> >>>      inet 50.50.1.1/24 brd 50.50.1.255 scope global qr-eebfe1cb-0f
> >>>      inet6 fe80::f816:3eff:fe08:1619/64 scope link
> >>>         valid_lft forever preferred_lft forever
> >>>  40: qg-910fef3b-cb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
> > qdisc
> >>>  noqueue state UNKNOWN
> >>>      link/ether fa:16:3e:e3:d5:fa brd ff:ff:ff:ff:ff:ff
> >>>      inet 10.245.124.65/26 brd 10.245.124.127 scope global
> > qg-910fef3b-cb
> >>>      inet 10.245.124.67/32 brd 10.245.124.67 scope global
> qg-910fef3b-cb
> >>>      inet6 fe80::f816:3eff:fee3:d5fa/64 scope link
> >>>         valid_lft forever preferred_lft forever
> >>>
> >>>
> #########################################################################
> >>> ##
> >>>  #######################
> >>>
> >>>  root at openstack-2:~# quantum net-show
> >>> 37d27ee8-36a9-4cdb-9966-9b5571526b41
> >>>  +---------------------------+--------------------------------------+
> >>>  | Field                     | Value                                |
> >>>  +---------------------------+--------------------------------------+
> >>>  | admin_state_up            | True                                 |
> >>>  | id                        | 37d27ee8-36a9-4cdb-9966-9b5571526b41 |
> >>>  | name                      | ext_net                              |
> >>>  | provider:network_type     | gre                                  |
> >>>  | provider:physical_network |                                      |
> >>>  | provider:segmentation_id  | 1                                    |
> >>>  | router:external           | True                                 |
> >>>  | shared                    | True                                 |
> >>>  | status                    | ACTIVE                               |
> >>>  | subnets                   | dd6f08f5-bfbd-4bdb-b9e4-c5ca065f3750 |
> >>>  | tenant_id                 | 2990df1bd46c4dda915b43558d591a2f     |
> >>>  +---------------------------+--------------------------------------+
> >>>
> >>>
> #########################################################################
> >>> ##
> >>>  #######################
> >>>
> >>>
> >>>
> >>>
> >>>  root at openstack-2:~# quantum subnet-show
> >>>  dd6f08f5-bfbd-4bdb-b9e4-c5ca065f3750
> >>>
> >>>
> +------------------+-----------------------------------------------------
> >>> +
> >>>  | Field            | Value
> >>>  |
> >>>
> >>>
> +------------------+-----------------------------------------------------
> >>> +
> >>>  | allocation_pools | {"start": "10.245.124.65",
> >>>  "end": "10.245.124.126"} |
> >>>  | cidr             | 10.245.124.64/26
> >>>  |
> >>>  | dns_nameservers  | 10.245.0.10
> >>>  |
> >>>  | enable_dhcp      | False
> >>>  |
> >>>  | gateway_ip       | 10.245.124.253
> >>>  |
> >>>  | host_routes      |
> >>>  |
> >>>  | id               | dd6f08f5-bfbd-4bdb-b9e4-c5ca065f3750
> >>>  |
> >>>  | ip_version       | 4
> >>>  |
> >>>  | name             |
> >>>  |
> >>>  | network_id       | 37d27ee8-36a9-4cdb-9966-9b5571526b41
> >>>  |
> >>>  | tenant_id        | 2990df1bd46c4dda915b43558d591a2f
> >>>  |
> >>>
> >>>
> +------------------+-----------------------------------------------------
> >>> +
> >>>
> >>>
> #########################################################################
> >>> ##
> >>>  #######################
> >>>
> >>>
> >>>
> >>>  Thanks,
> >>>
> >>>  -Farhan.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>  On 5/27/13 4:08 AM, "Darragh OReilly"
> >>>  <darragh.oreilly at yahoo.com> wrote:
> >>>
> >>>>
> >>>>  I'd check the external network config first.
> >>>>
> >>>>  You should be able to ping the external subnet's gateway from
> > the
> >>>> router
> >>>>  namespace.
> >>>>  This gateway should correspond to some real external
> > gateway/router.
> >>>>
> >>>>  quantum subnet-show <ext sub id> -c gateway_ip   #
> > 10.245.124.1 ?
> >>>>  ip netns exec <router-ns> ping -c1 <ext sub gateway>
> >>>>
> >>>>  If that is not working use tcpdump as you ping. Br-ex is using
> > eth0, is
> >>>>  eth0 up? tcpdump -nei eth0
> >>>>
> >>>>
> >>>>  If you are still having problems, post the above output and the
> >>>> following:
> >>>>
> >>>>  # network node
> >>>>  ip link
> >>>>  ip netns exec <router-ns> ip address
> >>>>
> >>>>  quantum net-show <uuid of external net>
> >>>>  quantum subnet-show <uuid of external subnet>
> >>>>
> >>>>
> >>>>>  ________________________________
> >>>>>   From: Farhan Patwa <Farhan.Patwa at utsa.edu>
> >>>>>  To: OpenStack Maillist <openstack at lists.launchpad.net>
> >>>>>  Sent: Friday, 24 May 2013, 20:28
> >>>>>  Subject: [Openstack] VM Issues on Grizzly Install on Ubuntu
> > 12.04
> >>>>>
> >>>>>
> >>>>>
> >>>>>  Hello,
> >>>>>  I followed the following guide to install Grizzly release on
> > 3-node
> >>>>>  setup.
> >>>>>
> >>>>>
> http://docs.openstack.org/grizzly/basic-install/apt/content/basic-insta
> >>>>> ll
> >>>>>  _intro.html
> >>>>>
> >>>>>
> >>>>>  I am stuck at my last issue with Quantum networking (at least
> > that¹s
> >>>>>  what I think).
> >>>>>  The VM instance comes up and gets the private IP and the
> > metadata.
> >>>>>  Also I have assigned the floating IP to it but am not able to
> > ping
> >>>>>  either IP except when I use:
> >>>>>
> >>>>>
> >>>>>  ip netns exec qrouter-32f35fb4-f9f1-4817-8818-fff832f73810 ping
> >>>>>  50.50.1.3       <- fixed IP ­ private network
> >>>>>  ip netns exec qrouter-32f35fb4-f9f1-4817-8818-fff832f73810 ping
> >>>>>  10.24.124.4  <- floating IP ­ external network
> >>>>>
> >>>>>
> >>>>>  Based on that I think the security rules are okay
> >>>>>  The router is tied to the specified tenant and using gateway of
> > the
> >>>>>  external network.
> >>>>>  I  think the issue is routing table or maybe firewall related
> > but not
> >>>>>  sure how to debug this.
> >>>>>
> >>>>>
> >>>>>  Some details of my environment are below.
> >>>>>  Any one have any words of wisdom/guidance?
> >>>>>
> >>>>>
> >>>>>  Thanks,
> >>>>>
> >>>>>
> >>>>>  -Farhan.
> >>>>>
> >>>>>
> >>>>>  Management Network: 192.168.0.0/24
> >>>>>  Data Network: 10.5.5.0/24
> >>>>>  External Network: 10.245.124.0/24
> >>>>>
> >>>>>
> >>>>>  Network Node: (192.168.0.2)
> >>>>>  ovs-vsctl show
> >>>>>  ea4fa894-5986-40f2-b10b-55eef2222408
> >>>>>      Bridge br-tun
> >>>>>          Port patch-int
> >>>>>              Interface patch-int
> >>>>>                  type: patch
> >>>>>                  options: {peer=patch-tun}
> >>>>>          Port "gre-1"
> >>>>>              Interface "gre-1"
> >>>>>                  type: gre
> >>>>>                  options: {in_key=flow, out_key=flow,
> >>>>>  remote_ip="192.168.0.3"}
> >>>>>          Port br-tun
> >>>>>              Interface br-tun
> >>>>>                  type: internal
> >>>>>      Bridge br-int
> >>>>>          Port "tap3fca71a9-c8"
> >>>>>              tag: 4095
> >>>>>              Interface "tap3fca71a9-c8"
> >>>>>                  type: internal
> >>>>>          Port patch-tun
> >>>>>              Interface patch-tun
> >>>>>                  type: patch
> >>>>>                  options: {peer=patch-int}
> >>>>>          Port "tap4b8a22a2-9c"
> >>>>>              tag: 4095
> >>>>>              Interface "tap4b8a22a2-9c"
> >>>>>                  type: internal
> >>>>>          Port "tap633ed611-a9"
> >>>>>              tag: 1
> >>>>>              Interface "tap633ed611-a9"
> >>>>>                  type: internal
> >>>>>          Port "qr-eebfe1cb-0f"
> >>>>>              tag: 1
> >>>>>              Interface "qr-eebfe1cb-0f"
> >>>>>                  type: internal
> >>>>>          Port br-int
> >>>>>              Interface br-int
> >>>>>                  type: internal
> >>>>>      Bridge br-ex
> >>>>>          Port "eth0"
> >>>>>              Interface "eth0"
> >>>>>          Port br-ex
> >>>>>              Interface br-ex
> >>>>>                  type: internal
> >>>>>          Port "qg-910fef3b-cb"
> >>>>>              Interface "qg-910fef3b-cb"
> >>>>>                  type: internal
> >>>>>      ovs_version: "1.4.0+build0"
> >>>>>
> >>>>>
> >>>>>  Kernel IP routing table
> >>>>>  Destination     Gateway         Genmask         Flags Metric
> > Ref
> >>>>> Use
> >>>>>  Iface
> >>>>>  0.0.0.0         192.168.0.253   0.0.0.0         UG    0      0
> >>>>> 0
> >>>>>  eth1
> >>>>>  10.5.5.0        0.0.0.0         255.255.255.0   U     0      0
> >>>>> 0
> >>>>>  eth1
> >>>>>  10.245.124.0    0.0.0.0         255.255.255.0   U     0      0
> >>>>> 0
> >>>>>  br-ex
> >>>>>  192.168.0.0     0.0.0.0         255.255.255.0   U     0      0
> >>>>> 0
> >>>>>  eth1
> >>>>>
> >>>>>
> >>>>>  Compute Node: (192.168.0.3)
> >>>>>  ovs-vsctl show
> >>>>>  f0fe78a5-dfd0-4f6b-87be-466dac0b4473
> >>>>>      Bridge br-tun
> >>>>>          Port patch-int
> >>>>>              Interface patch-int
> >>>>>                  type: patch
> >>>>>                  options: {peer=patch-tun}
> >>>>>          Port br-tun
> >>>>>              Interface br-tun
> >>>>>                  type: internal
> >>>>>          Port "gre-2"
> >>>>>              Interface "gre-2"
> >>>>>                  type: gre
> >>>>>                  options: {in_key=flow, out_key=flow,
> >>>>>  remote_ip="192.168.0.2"}
> >>>>>      Bridge br-int
> >>>>>          Port patch-tun
> >>>>>              Interface patch-tun
> >>>>>                  type: patch
> >>>>>                  options: {peer=patch-int}
> >>>>>          Port br-int
> >>>>>              Interface br-int
> >>>>>                  type: internal
> >>>>>          Port "tap6514a8cc-b2"
> >>>>>              tag: 1
> >>>>>              Interface "tap6514a8cc-b2"
> >>>>>      ovs_version: "1.4.0+build0"
> >>>>>
> >>>>>
> >>>>>  Kernel IP routing table
> >>>>>  Destination     Gateway         Genmask         Flags Metric
> > Ref
> >>>>> Use
> >>>>>  Iface
> >>>>>  0.0.0.0         192.168.0.253   0.0.0.0         UG    0      0
> >>>>> 0
> >>>>>  eth1
> >>>>>  10.5.5.0        0.0.0.0         255.255.255.0   U     0      0
> >>>>> 0
> >>>>>  eth1
> >>>>>  10.245.124.0    0.0.0.0         255.255.255.0   U     0      0
> >>>>> 0
> >>>>>  eth0
> >>>>>  192.168.0.0     0.0.0.0         255.255.255.0   U     0      0
> >>>>> 0
> >>>>>  eth1
> >>>>>
> >>>>>
> >>>>>  _______________________________________________
> >>>>>  Mailing list: https://launchpad.net/~openstack
> >>>>>  Post to     : openstack at lists.launchpad.net
> >>>>>  Unsubscribe : https://launchpad.net/~openstack
> >>>>>  More help   : https://help.launchpad.net/ListHelp
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>>  _______________________________________________
> >>>  Mailing list: https://launchpad.net/~openstack
> >>>  Post to     : openstack at lists.launchpad.net
> >>>  Unsubscribe : https://launchpad.net/~openstack
> >>>  More help   : https://help.launchpad.net/ListHelp
> >>>
> >>
> >
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130530/1ff48c7f/attachment.html>


More information about the Openstack mailing list