<div dir="ltr">Hello Arindam,<div><br></div><div>While deploying Hadoop cluster Savanna does pretty many API requests to Nova. Naturally, the number of requests is directly proportional to the size of the cluster.</div><div>
<br></div><div>On the other hand Nova has a protection agains users abusing API with too many requests. It is called rate limiting. You need to set limits higher than they are right now if you want to spin-up a cluster of that size. You can find details in the Nova docs:</div>
<div><a href="http://docs.openstack.org/grizzly/openstack-compute/admin/content//configuring-compute-API.html">http://docs.openstack.org/grizzly/openstack-compute/admin/content//configuring-compute-API.html</a><br></div><div>
<br></div><div>Dmitry</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/12/5 Arindam Choudhury <span dir="ltr"><<a href="mailto:arindam@live.com" target="_blank">arindam@live.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div dir="ltr">Hi,<br><br>When I try to create a big hadoop cluster (21 nodes), sometimes I am getting this error: <br><br>2013-12-05 12:17:57.920 29553 ERROR savanna.context [-] Thread 'cluster-creating-8d093d9b-c675-4222-b53a-3319d54bc61f' fails with exception: 'This request was rate-limited. (HTTP 413)'<br>
2013-12-05 12:17:57.920 29553 TRACE savanna.context Traceback (most recent call last):<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/context.py", line 128, in wrapper<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context func(*args, **kwargs)<br>
2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/api.py", line 123, in _provision_cluster<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context i.create_cluster(cluster)<br>
2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/instances.py", line 56, in create_cluster<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context _rollback_cluster_creation(cluster, ex)<br>
2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context self.gen.next()<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/instances.py", line 36, in create_cluster<br>
2013-12-05 12:17:57.920 29553 TRACE savanna.context _create_instances(cluster)<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/instances.py", line 111, in _create_instances<br>
2013-12-05 12:17:57.920 29553 TRACE savanna.context _run_instance(cluster, node_group, idx, aa_groups, userdata)<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/instances.py", line 173, in _run_instance<br>
2013-12-05 12:17:57.920 29553 TRACE savanna.context key_name=cluster.user_keypair_id)<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/v1_1/servers.py", line 658, in create<br>
2013-12-05 12:17:57.920 29553 TRACE savanna.context **boot_kwargs)<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/base.py", line 402, in _boot<br>
2013-12-05 12:17:57.920 29553 TRACE savanna.context return_raw=return_raw, **kwargs)<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/base.py", line 145, in _create<br>
2013-12-05 12:17:57.920 29553 TRACE savanna.context _resp, body = self.api.client.post(url, body=body)<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py", line 232, in post<br>
2013-12-05 12:17:57.920 29553 TRACE savanna.context return self._cs_request(url, 'POST', **kwargs)<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py", line 213, in _cs_request<br>
2013-12-05 12:17:57.920 29553 TRACE savanna.context **kwargs)<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py", line 195, in _time_request<br>
2013-12-05 12:17:57.920 29553 TRACE savanna.context resp, body = self.request(url, method, **kwargs)<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py", line 189, in request<br>
2013-12-05 12:17:57.920 29553 TRACE savanna.context raise exceptions.from_response(resp, body, url, method)<br>2013-12-05 12:17:57.920 29553 TRACE savanna.context OverLimit: This request was rate-limited. (HTTP 413)<br>
2013-12-05 12:17:57.920 29553 TRACE savanna.context <br><br>How I can prevent this to happen? Any help will be highly appreciated.<br><br>Regards,<br><br>Arindam<br> </div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>