Open Stack

Hello Arindam,

While deploying Hadoop cluster Savanna does pretty many API requests to
Nova. Naturally, the number of requests is directly proportional to the
size of the cluster.

On the other hand Nova has a protection agains users abusing API with too
many requests. It is called rate limiting. You need to set limits higher
than they are right now if you want to spin-up a cluster of that size. You
can find details in the Nova docs:
http://docs.openstack.org/grizzly/openstack-compute/admin/content//configuring-compute-API.html

Dmitry


2013/12/5 Arindam Choudhury <arindam at live.com>

> Hi,
>
> When I try to create a big hadoop cluster (21 nodes), sometimes I am
> getting this error:
>
> 2013-12-05 12:17:57.920 29553 ERROR savanna.context [-] Thread
> 'cluster-creating-8d093d9b-c675-4222-b53a-3319d54bc61f' fails with
> exception: 'This request was rate-limited. (HTTP 413)'
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context Traceback (most recent
> call last):
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context   File
> "/root/savanna/savanna/context.py", line 128, in wrapper
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context     func(*args,
> **kwargs)
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context   File
> "/root/savanna/savanna/service/api.py", line 123, in _provision_cluster
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context
> i.create_cluster(cluster)
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context   File
> "/root/savanna/savanna/service/instances.py", line 56, in create_cluster
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context
> _rollback_cluster_creation(cluster, ex)
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context   File
> "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context     self.gen.next()
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context   File
> "/root/savanna/savanna/service/instances.py", line 36, in create_cluster
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context
> _create_instances(cluster)
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context   File
> "/root/savanna/savanna/service/instances.py", line 111, in _create_instances
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context
> _run_instance(cluster, node_group, idx, aa_groups, userdata)
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context   File
> "/root/savanna/savanna/service/instances.py", line 173, in _run_instance
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context
> key_name=cluster.user_keypair_id)
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context   File
> "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/v1_1/servers.py",
> line 658, in create
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context     **boot_kwargs)
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context   File
> "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/base.py",
> line 402, in _boot
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context
> return_raw=return_raw, **kwargs)
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context   File
> "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/base.py",
> line 145, in _create
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context     _resp, body =
> self.api.client.post(url, body=body)
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context   File
> "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py",
> line 232, in post
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context     return
> self._cs_request(url, 'POST', **kwargs)
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context   File
> "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py",
> line 213, in _cs_request
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context     **kwargs)
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context   File
> "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py",
> line 195, in _time_request
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context     resp, body =
> self.request(url, method, **kwargs)
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context   File
> "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py",
> line 189, in request
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context     raise
> exceptions.from_response(resp, body, url, method)
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context OverLimit: This
> request was rate-limited. (HTTP 413)
> 2013-12-05 12:17:57.920 29553 TRACE savanna.context
>
> How I can prevent this to happen? Any help will be highly appreciated.
>
> Regards,
>
> Arindam
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131205/d7c3c6ee/attachment.html>