<div dir="ltr">Kesten,<div><br></div><div>The error occurs at the following line:</div><div><a href="https://github.com/stackforge/savanna/blob/master/savanna/service/instances.py#L293">https://github.com/stackforge/savanna/blob/master/savanna/service/instances.py#L293</a><br>
</div><div><br></div><div>The reason is that Savanna detects that one of instances it spawned entered 'Error' state. I.e. the problem is either in your OpenStack or in a way Savanna spawns instances. The instances are spawned by the following piece of code:</div>
<div><a href="https://github.com/stackforge/savanna/blob/master/savanna/service/instances.py#L214">https://github.com/stackforge/savanna/blob/master/savanna/service/instances.py#L214</a><br></div><div><br></div><div>Right now Savanna uses rather dummy policy: if anything goes wrong during cluster start, Savanna completely rolls back startup and terminates spawned instances. If you comment the following line:</div>
<div><a href="https://github.com/stackforge/savanna/blob/master/savanna/service/instances.py#L367">https://github.com/stackforge/savanna/blob/master/savanna/service/instances.py#L367</a><br></div><div><br></div><div>, you should see one of cluster instances in 'Error' state after cluster startup fails.</div>
<div><br></div><div>Dmitry</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/9/17 kesten broughton <span dir="ltr"><<a href="mailto:kesten.broughton@gmail.com" target="_blank">kesten.broughton@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><pre style="font-size:13px;margin-bottom:0px;font-family:'Bitstream Vera Sans Mono',monospace;margin-top:0px;padding:5px 0px">
I have applied the proposed patch for the setattr launch error</pre>
<pre style="font-size:13px;margin-bottom:0px;font-family:'Bitstream Vera Sans Mono',monospace;margin-top:0px;padding:5px 0px"><br></pre><pre style="font-size:13px;margin-bottom:0px;font-family:'Bitstream Vera Sans Mono',monospace;margin-top:0px;padding:5px 0px">
I patch works, but the launch still fails. </pre><pre style="font-size:13px;margin-bottom:0px;font-family:'Bitstream Vera Sans Mono',monospace;margin-top:0px;padding:5px 0px">The stack trace just says the creation failed.</pre>
<pre style="font-size:13px;margin-bottom:0px;font-family:'Bitstream Vera Sans Mono',monospace;margin-top:0px;padding:5px 0px"><br></pre><pre style="font-size:13px;margin-bottom:0px;font-family:'Bitstream Vera Sans Mono',monospace;margin-top:0px;padding:5px 0px">
<pre style="margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace">127.0.0.1 - - [16/Sep/2013 12:21:20] "POST /v1.0/2c8b2627a169458e8ab875690a51eabd/clusters HTTP/1.1" 202 1877 1.441746
2013-09-16 12:21:42.592 47355 WARNING savanna.service.instances [-] Can't start cluster 'cluster-1' (reason: node cluster-1-workers-002 has error status)
2013-09-16 12:21:51.369 47355 ERROR savanna.context [-] Thread 'cluster-creating-057ae8f2-ce41-4508-9696-3affe064178d' fails with exception: 'node cluster-1-workers-002 has error status'
2013-09-16 12:21:51.369 47355 TRACE savanna.context Traceback (most recent call last):
2013-09-16 12:21:51.369 47355 TRACE savanna.context File "/home/stack/savanna-venv/local/lib/python2.7/site-packages/savanna/context.py", line 93, in wrapper
2013-09-16 12:21:51.369 47355 TRACE savanna.context func(*args, **kwargs)
2013-09-16 12:21:51.369 47355 TRACE savanna.context File "/home/stack/savanna-venv/local/lib/python2.7/site-packages/savanna/service/api.py", line 137, in _provision_cluster
2013-09-16 12:21:51.369 47355 TRACE savanna.context i.create_cluster(cluster)
2013-09-16 12:21:51.369 47355 TRACE savanna.context File "/home/stack/savanna-venv/local/lib/python2.7/site-packages/savanna/service/instances.py", line 63, in create_cluster
2013-09-16 12:21:51.369 47355 TRACE savanna.context _rollback_cluster_creation(cluster, ex)
2013-09-16 12:21:51.369 47355 TRACE savanna.context File "/home/stack/savanna-venv/local/lib/python2.7/site-packages/savanna/openstack/common/excutils.py", line 70, in __exit__
2013-09-16 12:21:51.369 47355 TRACE savanna.context six.reraise(self.type_, self.value, self.tb)
2013-09-16 12:21:51.369 47355 TRACE savanna.context File "/home/stack/savanna-venv/local/lib/python2.7/site-packages/savanna/service/instances.py", line 45, in create_cluster
2013-09-16 12:21:51.369 47355 TRACE savanna.context cluster = _await_instances(cluster)</pre></pre><div class="gmail_extra"><div class="gmail_quote"><br></div><div class="gmail_quote">complete details here: <a href="http://paste.openstack.org/show/47126/" target="_blank">http://paste.openstack.org/show/47126/</a><br>
</div><div class="gmail_quote">I can see in horizon that it makes it past the "waiting" log message.</div><div class="gmail_quote"><br></div><div class="gmail_quote">My suspicion is with volumes.attach() below since the output from the cluster_create contains </div>
<div class="gmail_quote"><br></div><div class="gmail_quote"><pre style="font-size:13px;margin-bottom:0px;font-family:'Bitstream Vera Sans Mono',monospace;margin-top:0px;padding:5px 0px">"volume_mount_prefix": "/volumes/disk", </pre>
</div><div class="gmail_quote"><br></div><div class="gmail_quote">I am using the vmware machine as configured in the guide, but </div><div class="gmail_quote">locate /volumes/disk returns nothing.</div><div class="gmail_quote">
<br></div><div class="gmail_quote">--- here is instances.py where the exception is thrown</div><div class="gmail_quote"><br></div><div class="gmail_quote"><div class="gmail_quote">def create_cluster(cluster):</div><div class="gmail_quote">
ctx = context.ctx()</div><div class="gmail_quote"> try:</div><div class="gmail_quote"> # create all instances</div><div class="gmail_quote"> conductor.cluster_update(ctx, cluster, {"status": "Spawning"})</div>
<div class="gmail_quote"> LOG.info(g.format_cluster_status(cluster))</div><div class="gmail_quote"> _create_instances(cluster)</div><div class="gmail_quote"><br></div><div class="gmail_quote"> # wait for all instances are up and accessible</div>
<div class="gmail_quote"> cluster = conductor.cluster_update(ctx, cluster, {"status": "Waiting"})</div><div class="gmail_quote"> LOG.info(g.format_cluster_status(cluster))</div><div class="gmail_quote">
cluster = _await_instances(cluster)</div><div class="gmail_quote"><br></div><div class="gmail_quote"> # attach volumes</div><div class="gmail_quote"> volumes.attach(cluster)</div><div class="gmail_quote">
<br></div><div class="gmail_quote"> # prepare all instances</div><div class="gmail_quote"> cluster = conductor.cluster_update(ctx, cluster,</div><div class="gmail_quote"> {"status": "Preparing"})</div>
<div class="gmail_quote"> LOG.info(g.format_cluster_status(cluster))</div><div class="gmail_quote"><br></div><div class="gmail_quote"> _configure_instances(cluster)</div><div class="gmail_quote"> </div>
<div class="gmail_quote"> except Exception as ex:</div><div class="gmail_quote"> LOG.warn("Can't start cluster '%s' (reason: %s)", <a href="http://cluster.name" target="_blank">cluster.name</a>, ex)</div>
<div>
<br></div><div>tips to debug?</div></div></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>