[openstack-dev] Implementing VPNaas in Openstack Grizzly release
Ashwini Babureddy
AB0078468 at TechMahindra.com
Tue Jan 14 07:04:01 UTC 2014
I have added all the configurations mentioned below in the mail but getting the same error while executing the command "ipsec-site-connection-create".[ Request Failed: internal server error while processing your request.]
Along with the error, there are few logs seen in vpn_agent.log as shown below:
2014-01-14 20:13:17 ERROR [quantum.agent.l3_agent] Failed synchronizing routers : _sync_routers_task
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 694, in _sync_routers_task
self._process_routers(routers, all_routers=True)
File "/usr/lib/python2.7/dist-packages/quantum/services/vpn/agent.py", line 148, in _process_routers
super(VPNAgent, self)._process_routers(routers, all_routers)
File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 667, in _process_routers
self._router_added(r['id'], r)
File "/usr/lib/python2.7/dist-packages/quantum/services/vpn/agent.py", line 128, in _router_added
super(VPNAgent, self)._router_added(router_id, router)
File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 268, in _router_added
self._create_router_namespace(ri)
File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 246, in _create_router_namespace
ip_wrapper.netns.execute(['sysctl', '-w', 'net.ipv4.ip_forward=1'])
File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/ip_lib.py", line 417, in execute
check_exit_code=check_exit_code)
File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py", line 81, in execute
raise RuntimeError(m)
RuntimeError:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-00aadec6-d405-4ee6-8cd2-16a5d79d682c', 'sysctl', '-w', 'net.ipv4.ip_forward=1']
Exit code: 1
Stdout: ''
Stderr: 'seting the network namespace failed: Invalid argument\n'
And executing the command "quantum -v ipsec-site-connection-create --name ipsec_pqr --vpnservice-id vpnservice_pqr --ikepolicy-id ike_pqr --ipsecpolicy-id ipsec_pqr --peer-address 7.7.0.3 --peer-id 7.7.0.3 --peer-cidr 3.3.0.0/24 --psk secret" gives the following error:
DEBUG: quantumclient.client RESP:{'date': 'Tue, 14 Jan 2014 12:29:21 GMT', 'status': '500', 'content-length': '88', 'content-type': 'application/json; charset=UTF-8'} {"QuantumError": "Request Failed: internal server error while processing your request."}
INFO: quantumclient.client leaving HTTPClient._cs_request
DEBUG: quantumclient.v2_0.client Error message: {"QuantumError": "Request Failed: internal server error while processing your request."}
ERROR: quantumclient.shell Request Failed: internal server error while processing your request.
DEBUG: quantumclient.shell clean_up CreateIPsecSiteConnection
DEBUG: quantumclient.shell got an error: Request Failed: internal server error while processing your request.
INFO: quantumclient.shell Leaving run_subcommand result 1
POST method is being called and then throws this Error:
Inside CreateCommand::get_data(resource : ipsec_site_connection)
Inside CreateCommand::get_data(body : {'ipsec_site_connection': {'psk': 'secret', 'peer_cidrs': ['3.3.0.0/24'], 'ikepolicy_id': u'beb12aae-e982-4186-848e-14f2a7647a90', 'vpnservice_id': u'1ad5bef8-3a57-4be1-9fa1-2a9a4e332e9b', 'peer_address': '7.7.0.3', 'initiator': 'bi-directional', 'name': 'ipsec_pqr', 'admin_state_up': True, 'mtu': '1500', 'ipsecpolicy_id': u'ba7c7456-34c1-4ee7-a9b7-9249d3a4523c', 'peer_id': '7.7.0.3'}})
obj_creator : <function with_params at 0x1b58398>
Inside quantumclient/v2_0/client create_ipsec_site_connection
body : {'ipsec_site_connection': {'psk': 'secret', 'peer_cidrs': ['3.3.0.0/24'], 'ikepolicy_id': u'beb12aae-e982-4186-848e-14f2a7647a90', 'vpnservice_id': u'1ad5bef8-3a57-4be1-9fa1-2a9a4e332e9b', 'peer_address': '7.7.0.3', 'initiator': 'bi-directional', 'name': 'ipsec_pqr', 'admin_state_up': True, 'mtu': '1500', 'ipsecpolicy_id': u'ba7c7456-34c1-4ee7-a9b7-9249d3a4523c', 'peer_id': '7.7.0.3'}}
Inside POST
Inside POST : action -> /vpn/ipsec-site-connections
Inside do_request method info
Inside HTTPClient.do_request, url -> /v2.0/vpn/ipsec-site-connections.json
Inside HTTPClient.do_request, method -> POST
Inside HTTPClient._cs_request
leaving HTTPClient._cs_request
Request Failed: internal server error while processing your request.
Leaving run_subcommand result 1
Could this be a server issue?
Thanks,
Ashwini
From: Paul Michali [mailto:pcm at cisco.com]
Sent: Friday, January 10, 2014 5:40 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Implementing VPNaas in Openstack Grizzly release
Though I mostly use DevStack for debugging, I would think that the same logging information (from q-svc showing service driver side, and from q-vpn showing the device driver side) should be available - probably in the system log. I think you can specify this (and to turn on debug logging) via neutron.conf (DevStack does this for you automatically). Here's the stuff in neutron.conf:
[DEFAULT]
auth_strategy = keystone
allow_overlapping_ips = True
policy_file = /etc/neutron/policy.json
debug = True
verbose = True
service_plugins = neutron.services.vpn.plugin.VPNDriverPlugin
...
One thing to check. Do you have the service driver configured in neutron.conf? If not, then it won't be loaded and it won't respond to the RPC requests from the device driver. Here's the bits:
[service_providers]
# Specify service providers (drivers) for advanced services like loadbalancer, VPN, Firewall.
# Must be in form:
# service_provider=<service_type>:<name>:<driver>[:default]
# List of allowed service type include LOADBALANCER, FIREWALL, VPN
# Combination of <service type> and <name> must be unique; <driver> must also be unique
# this is multiline option, example for default provider:
# service_provider=LOADBALANCER:name:lbaas_plugin_driver_path:default
# example of non-default provider:
# service_provider=FIREWALL:name2:firewall_driver_path
# --- Reference implementations ---
service_provider=LOADBALANCER:Haproxy:neutron.services.loadbalancer.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default
service_provider=VPN:openswan:neutron.services.vpn.service_drivers.ipsec.IPsecVPNDriver:default
...
HTHs,
PCM (Paul Michali)
MAIL pcm at cisco.com<mailto:pcm at cisco.com>
IRC pcm_ (irc.freenode.net<http://irc.freenode.net>)
TW @pmichali
GPG key 4525ECC253E31A83
Fingerprint 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83
On Jan 10, 2014, at 5:14 AM, Ashwini Babureddy <AB0078468 at TechMahindra.com<mailto:AB0078468 at TechMahindra.com>> wrote:
Along with the attached outlook item. Here are the following updates:
With the help of few logs written manually, it is seen that RPC call is waiting on "IPSEC_DRIVER_TOPIC = ipsec_driver" :
2014-01-10 23:35:15 INFO [quantum.openstack.common.rpc.proxy] Inside proxy.RpcProxy : msg [{'args': {'host': 'ubuntu'}, 'method': 'get_vpn_services_on_host'}], topic [ipsec_driver]
2014-01-10 23:35:15 INFO [quantum.openstack.common.rpc.amqp] Making synchronous call on ipsec_driver ...
2014-01-10 23:35:15 INFO [quantum.openstack.common.rpc.amqp] MSG_ID is df6c8e6ff7ca48cca239a84959a61fb7
2014-01-10 23:35:15 INFO [quantum.openstack.common.rpc.amqp] Inside if not case
2014-01-10 23:36:15 ERROR [quantum.agent.l3_agent] Failed synchronizing routers : _sync_routers_task
And queues created :
root at ubuntu:/var/log/quantum# rabbitmqctl list_queues | more
Listing queues ...
consoleauth_fanout_36b3f71c6ea64616a69c0b2e2ed3d7c0 0
l3_agent_fanout_e398e651d8d44303a4826fd2a3faedf4 0
cert_fanout_c04d42ce4ab249f4b050241bbe79caa3 0
q-agent-notifier-tunnel-update_fanout_5f77bce1a09348b0b6927553d0646bc6 0
l3_agent.ubuntu 0
consoleauth.ubuntu 0
cinder-scheduler 0
conductor_fanout_7c10c5e7371340d18af3588dccbc76b3 0
notifications.info 11
cinder-scheduler.ubuntu 0
cinder-scheduler_fanout_5d4e9f0e60c4413fabec3ecf9fce164f 0
q-agent-notifier-port-update_fanout_6043fca215d54263b74c058f33e29a74 0
q-agent-notifier-network-delete_fanout_12e2c4b4670c4455b4192413e8990166 0
conductor.ubuntu 0
dhcp_agent.ubuntu 0
cert.ubuntu 0
ipsec_agent.ubuntu 0
q-plugin 0
dhcp_agent 0
compute 0
scheduler 0
q-agent-notifier-security_group-update_fanout_794f378ea54a495d938fa347f99a6c8d 0
scheduler_fanout_2fc00a8c896a46ecac6f57df6abc8f70 0
l3_agent_fanout_fce1b317fff94b5aaba36dad6a0a9575 0
compute_fanout_ee1f3d0427244a35891c7cf707da1542 0
cert 0
conductor 0
compute.ubuntu 0
5ef01edd61d64f19937d4544b341b0e9 0
scheduler.ubuntu 0
dhcp_agent_fanout_5bd979a41d9a48aaa2f6639bcc079389 0
l3_agent 0
consoleauth 0
...done.
Thanks,
Ashwini
From: Paul Michali [mailto:pcm at cisco.com<http://cisco.com>]
Sent: Thursday, January 09, 2014 6:01 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Implementing VPNaas in Openstack Grizzly release
Just some ideas to look into...
You could look at the q-vpn process log. With devstack, there is a screen-q-vpn.log that might give more info on the failure.
The commands that are working, are ones that strictly deal with the database. The IPSec site connection command is the first one that actually does messaging from the service driver to the device driver. Once at the device driver, it will do an RPC back to the service driver to get the all the information on the services/connections on that host.
Maybe some things are to check that the IPSEC_AGENT_TOPIC is defined (IPSEC_AGENT_TOPIC = 'ipsec_agent'), and that the code is there in the service driver (service_dirvers/ipsec.py) to handle the callback (IPsecVpnDriverCallBack get_vpn_services_on_host). This will call the plugin (VPNPluginRpcDbMixin _get_agent_hosting_vpn_services).
It seems like the RPC from the device driver back to the service driver is timing out, so making sure they are all set up OK, would be useful.
HTHs,
PCM (Paul Michali)
MAIL pcm at cisco.com<mailto:pcm at cisco.com>
IRC pcm_ (irc.freenode.net<http://irc.freenode.net>)
TW @pmichali
GPG key 4525ECC253E31A83
Fingerprint 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83
On Jan 9, 2014, at 5:12 AM, Ashwini Babureddy <AB0078468 at TechMahindra.com<mailto:AB0078468 at TechMahindra.com>> wrote:
Hi,
I am trying to implement VPNaas in openstack grizzly release 2013.1 by taking Havana release as a reference. This is basically a single node set up by following the below link :
https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/master/OpenStack_Grizzly_Install_Guide.rst
Currently all the vpn related files from Havana moved to Grizzly as follows:
* /quantum/services/vpn/*
* /quantum/db/vpn/*
* /quantum/extensions/vpnaas.py
* /etc/quantum/vpn_agent.ini
* /etc/quantum/quantum.conf -> service_plugins = quantum.services.vpn.plugin.VPNPlugin
* /quantumclient/quantum/v2_0/vpn/*
* Installed Openswan
* Made changes in /quantumclient/shell.py
* /usr/bin/quantum-vpn-agent
* /etc/init.d/quantum-plugin-vpn-agent
* /etc/init/quantum-plugin-vpn-agent.conf
Current status:
* Commands running successfully
o Vpn-ikepolicy-create/list/delete
o Vpn-ipsecpolicy-create/list/delete
o Vpn-service-create/list/delete
* Ipsec-site-connection-create command is failing with an HTTP Error. [Request Failed: internal server error while processing your request.]
* /var/log/quantum/vpn-agent.log has logs as follows:
2014-01-09 23:32:30 ERROR [quantum.agent.l3_agent] Failed synchronizing routers : _sync_routers_task
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 694, in _sync_routers_task
self._process_routers(routers, all_routers=True)
File "/usr/lib/python2.7/dist-packages/quantum/services/vpn/agent.py", line 150, in _process_routers
device.sync(self.context, routers)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/lockutils.py", line 242, in inner
retval = f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/quantum/services/vpn/device_drivers/ipsec.py", line 652, in sync
context, self.host)
File "/usr/lib/python2.7/dist-packages/quantum/services/vpn/device_drivers/ipsec.py", line 453, in get_vpn_services_on_host
topic=self.topic)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/proxy.py", line 80, in call
return rpc.call(context, self._get_topic(topic), msg, timeout)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/__init__.py", line 140, in call
return _get_impl().call(CONF, context, topic, msg, timeout)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 798, in call
rpc_amqp.get_connection_pool(conf, Connection))
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 613, in call
rv = list(rv)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 555, in __iter__
self.done()
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 552, in __iter__
self._iterator.next()
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 648, in iterconsume
yield self.ensure(_error_callback, _consume)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 566, in ensure
error_callback(e)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 629, in _error_callback
raise rpc_common.Timeout()
Timeout: Timeout while waiting on RPC response.
2014-01-09 23:32:30 WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 21.531911 sec
Can anyone please help on this issue. Could this issue be due to an incomplete quantum-plugin-vpn-agent [as we have no such standard package].
What else can be done further to make this work?
Thanks,
Ashwini
________________________________
============================================================================================================================
Disclaimer: This message and the information contained herein is proprietary and confidential and subject to the Tech Mahindra policy statement, you may review the policy at http://www.techmahindra.com/Disclaimer.html externally http://tim.techmahindra.com/tim/disclaimer.html internally within TechMahindra.
============================================================================================================================
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
________________________________
============================================================================================================================
Disclaimer: This message and the information contained herein is proprietary and confidential and subject to the Tech Mahindra policy statement, you may review the policy at http://www.techmahindra.com/Disclaimer.html externally http://tim.techmahindra.com/tim/disclaimer.html internally within TechMahindra.
============================================================================================================================
From: Ashwini Babureddy <AB0078468 at TechMahindra.com<mailto:AB0078468 at TechMahindra.com>>
Subject: RE: [openstack-dev] Implementing VPNaas in Openstack Grizzly release
Date: January 10, 2014 2:31:03 AM EST
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Hi Paul,
Thanks for the info provided.
There is IPSEC_AGENT_TOPIC defined and that is imported in service_driver/ipsec.py. Code is present.
As I have not installed Openstack Grizzly set up using devstack, I do not have access to q-vpn process log.
So if the RPC is timing out waiting for the response back from device_driver to service_driver, does this mean that the callback (IPsecVpnDriverCallBack get_vpn_services_on_host) is having an issue.
Currently I am debugging this issue using /var/log/quantum/* logs.
Are there any other debugging tools that I can use in this scenario? Could Eclipse using PyDev be another option for debugging this issue?
Thanks,
Ashwini
From: Paul Michali [mailto:pcm at cisco.com<http://cisco.com>]
Sent: Thursday, January 09, 2014 6:01 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Implementing VPNaas in Openstack Grizzly release
Just some ideas to look into...
You could look at the q-vpn process log. With devstack, there is a screen-q-vpn.log that might give more info on the failure.
The commands that are working, are ones that strictly deal with the database. The IPSec site connection command is the first one that actually does messaging from the service driver to the device driver. Once at the device driver, it will do an RPC back to the service driver to get the all the information on the services/connections on that host.
Maybe some things are to check that the IPSEC_AGENT_TOPIC is defined (IPSEC_AGENT_TOPIC = 'ipsec_agent'), and that the code is there in the service driver (service_dirvers/ipsec.py) to handle the callback (IPsecVpnDriverCallBack get_vpn_services_on_host). This will call the plugin (VPNPluginRpcDbMixin _get_agent_hosting_vpn_services).
It seems like the RPC from the device driver back to the service driver is timing out, so making sure they are all set up OK, would be useful.
HTHs,
PCM (Paul Michali)
MAIL pcm at cisco.com<mailto:pcm at cisco.com>
IRC pcm_ (irc.freenode.net<http://irc.freenode.net>)
TW @pmichali
GPG key 4525ECC253E31A83
Fingerprint 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83
On Jan 9, 2014, at 5:12 AM, Ashwini Babureddy <AB0078468 at TechMahindra.com<mailto:AB0078468 at TechMahindra.com>> wrote:
Hi,
I am trying to implement VPNaas in openstack grizzly release 2013.1 by taking Havana release as a reference. This is basically a single node set up by following the below link :
https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/master/OpenStack_Grizzly_Install_Guide.rst
Currently all the vpn related files from Havana moved to Grizzly as follows:
* /quantum/services/vpn/*
* /quantum/db/vpn/*
* /quantum/extensions/vpnaas.py
* /etc/quantum/vpn_agent.ini
* /etc/quantum/quantum.conf -> service_plugins = quantum.services.vpn.plugin.VPNPlugin
* /quantumclient/quantum/v2_0/vpn/*
* Installed Openswan
* Made changes in /quantumclient/shell.py
* /usr/bin/quantum-vpn-agent
* /etc/init.d/quantum-plugin-vpn-agent
* /etc/init/quantum-plugin-vpn-agent.conf
Current status:
* Commands running successfully
o Vpn-ikepolicy-create/list/delete
o Vpn-ipsecpolicy-create/list/delete
o Vpn-service-create/list/delete
* Ipsec-site-connection-create command is failing with an HTTP Error. [Request Failed: internal server error while processing your request.]
* /var/log/quantum/vpn-agent.log has logs as follows:
2014-01-09 23:32:30 ERROR [quantum.agent.l3_agent] Failed synchronizing routers : _sync_routers_task
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 694, in _sync_routers_task
self._process_routers(routers, all_routers=True)
File "/usr/lib/python2.7/dist-packages/quantum/services/vpn/agent.py", line 150, in _process_routers
device.sync(self.context, routers)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/lockutils.py", line 242, in inner
retval = f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/quantum/services/vpn/device_drivers/ipsec.py", line 652, in sync
context, self.host)
File "/usr/lib/python2.7/dist-packages/quantum/services/vpn/device_drivers/ipsec.py", line 453, in get_vpn_services_on_host
topic=self.topic)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/proxy.py", line 80, in call
return rpc.call(context, self._get_topic(topic), msg, timeout)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/__init__.py", line 140, in call
return _get_impl().call(CONF, context, topic, msg, timeout)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 798, in call
rpc_amqp.get_connection_pool(conf, Connection))
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 613, in call
rv = list(rv)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 555, in __iter__
self.done()
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 552, in __iter__
self._iterator.next()
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 648, in iterconsume
yield self.ensure(_error_callback, _consume)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 566, in ensure
error_callback(e)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 629, in _error_callback
raise rpc_common.Timeout()
Timeout: Timeout while waiting on RPC response.
2014-01-09 23:32:30 WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 21.531911 sec
Can anyone please help on this issue. Could this issue be due to an incomplete quantum-plugin-vpn-agent [as we have no such standard package].
What else can be done further to make this work?
Thanks,
Ashwini
________________________________
============================================================================================================================
Disclaimer: This message and the information contained herein is proprietary and confidential and subject to the Tech Mahindra policy statement, you may review the policy at http://www.techmahindra.com/Disclaimer.html externally http://tim.techmahindra.com/tim/disclaimer.html internally within TechMahindra.
============================================================================================================================
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
============================================================================================================================
Disclaimer: This message and the information contained herein is proprietary and confidential and subject to the Tech Mahindra policy statement, you may review the policy at http://www.techmahindra.com/Disclaimer.html externally http://tim.techmahindra.com/tim/disclaimer.html internally within TechMahindra.
============================================================================================================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140114/a2749086/attachment.html>
More information about the OpenStack-dev
mailing list