<div dir="ltr">Anytime you call the nova API and ask it to boot an instance, before it starts handing things off the the scheduler it creates an action in the database. Your scheduler is complaining that it can't find that action. So in your initial email you don't say where your database is. Are you running a mysql host somewhere? Nova-api is going to need to be able to get at the same database as nova-scheduler. I would start poking around there and see if that's your problem.<div>
<br></div><div style>-Mike</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Aug 8, 2013 at 6:06 PM, Juan José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><br></div><div dir="ltr">You are right Wilson, there's no hostiD in the vm description:<div>
<br></div><div><div>+-----------------------------+-----------------------------------------------------------+</div>
<div>| Property | Value |</div>
<div>+-----------------------------+-----------------------------------------------------------+</div><div>| status | BUILD |</div><div>| updated | 2013-08-08T19:23:01Z |</div>
<div>| OS-EXT-STS:task_state | scheduling |</div><div>| key_name | None |</div><div>| image | Ubuntu 12.04.2 LTS (1359ca8d-23a2-40e8-940f-d90b3e68bb39) |</div>
<div>| hostId | |</div><div>| OS-EXT-STS:vm_state | building |</div><div>| flavor | m1.tiny (1) |</div>
<div>| id | b0583cca-63c2-481f-8b94-7aeb2e86641f |</div><div>| security_groups | [{u'name': u'default'}] |</div>
<div>
| user_id | 20390b639d4449c18926dca5e038ec5e |</div><div>| name | prueba11 |</div><div>| created | 2013-08-08T19:19:44Z |</div>
<div>| tenant_id | d1e3aae242f14c488d2225dcbf1e96d6 |</div><div>| OS-DCF:diskConfig | MANUAL |</div><div>| metadata | {} |</div>
<div>| accessIPv4 | |</div><div>| accessIPv6 | |</div><div>| progress | 0 |</div>
<div>| OS-EXT-STS:power_state | 0 |</div><div>| OS-EXT-AZ:availability_zone | nova |</div><div>| config_drive | |</div>
<div>+-----------------------------+-----------------------------------------------------------+</div></div><div><br></div><div>I found this new log in the scheduler, it must be related to it:</div><div><br></div><div><div>
2013-08-08 19:19:46.000 ERROR nova.openstack.common.rpc.amqp [req-b049c969-c411-4b76-9aa5-ad88d714c4ab 20390b639d4449c18926dca5e038ec5e d1e3aae242f14c488d2225dcbf1e96d6] Exception during message handling</div><div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):</div>
<div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data</div><div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)</div>
<div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch</div><div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)</div>
<div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 127, in run_instance</div><div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp 'schedule', *instance_uuids):</div>
<div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 318, in __enter__</div><div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp self.conductor.action_event_start(self.context, event)</div>
<div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 264, in action_event_start</div><div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp return self._manager.action_event_start(context, values)</div>
<div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 1348, in wrapper</div><div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp return func(*args, **kwargs)</div>
<div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 334, in action_event_start</div><div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp evt = self.db.action_event_start(context, values)</div>
<div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 1625, in action_event_start</div><div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp return IMPL.action_event_start(context, values)</div>
<div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 4624, in action_event_start</div><div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp instance_uuid=values['instance_uuid'])</div>
<div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp InstanceActionNotFound: Action for request_id req-b049c969-c411-4b76-9aa5-ad88d714c4ab on instance b0583cca-63c2-481f-8b94-7aeb2e86641f not found</div>
<div>2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp</div></div><div><br></div><div>It's getting me crazy. "InstanceActionNotFound..." does it mean something to you?</div><div><br></div></div>
<div><div>
<div class="gmail_extra"><br><br><div class="gmail_quote">2013/8/8 Mike Wilson <span dir="ltr"><<a href="mailto:geekinutah@gmail.com" target="_blank">geekinutah@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Juan,<div><br></div><div>If your instance show's the task_state as "scheduling" it's possible that your new instance never made it to the scheduler. When you do a nova show <instance_id> does it say which host node it is supposed to be on? If it does then the scheduler probably did its thing but failed the RPC to the compute node, if it doesn't have a host then your problem is that the message never made it from nova-api to nova-scheduler.</div>
<div> </div><div>In any case, it looks like rabbit is not quite setup correctly on your end.</div><div><br></div><div>-Mike</div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Wed, Aug 7, 2013 at 6:55 PM, Juan José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Is there any way i can test nova-conductor and nova-scheduler to be sure they are working like they should? If i list nova-manage service list, everything is fine. I'm running out of ideas hahaha.</div>
<div><div><div class="gmail_extra">
<br><br><div class="gmail_quote">2013/8/7 Juan José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">According to the doc this problem should be related to some service that isn't answering to nova-api. I just have 3 servers in my deployment, so i don't think this is problem related to the amount of messages in the queues. </div>
<div><div>
<div class="gmail_extra"><br><br><div class="gmail_quote">2013/8/7 Juan José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Here i have more information, i tried to boot a vm from the CLI and it doesn't really fail. But when i check the vms status in the dashboard it says "Scheduling" and never changes its state to "running" or "error". </div>
<div><div>
<div class="gmail_extra"><br><br><div class="gmail_quote">2013/8/7 Juan José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>I just finished installing everything, i tried to create my first VM from the dashboard but it doesn't work. After choosing flavor and hitting launch it starts "creating" it but after a few seconds it stops saying: "Error: There was an error submitting the form. Please try again.". The only place where i found something related is in nova.log in my compute node, here is the log:</div>
<div><br></div><div><i>2013-08-07 18:05:55.293 DEBUG nova.openstack.common.rpc.common [req-0cfe760f-2e74-4e92-919c-663ba02c7f2f 20390b639d4449c18926dca5e038ec5e d1e3aae242f14c488d2225dcbf1e96d6] Timed out waiting for RPC response: timed out _error_callback /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py:628<br>
</i></div><div><i>2013-08-07 18:05:55.479 DEBUG nova.quota [req-0cfe760f-2e74-4e92-919c-663ba02c7f2f 20390b639d4449c18926dca5e038ec5e d1e3aae242f14c488d2225dcbf1e96d6] Rolled back reservations ['3e941a2b-2cc6-4f01-8dc1-13dc09369141', '411f6f70-415e-4a21-aa06-3980070d6095', 'd4791eb7-b75a-4ab8-bfdb-5d5cd201e40d'] rollback /usr/lib/python2.7/dist-packages/nova/quota.py:1012</i></div>
<div><i>2013-08-07 18:05:55.480 ERROR nova.api.openstack [req-0cfe760f-2e74-4e92-919c-663ba02c7f2f 20390b639d4449c18926dca5e038ec5e d1e3aae242f14c488d2225dcbf1e96d6] Caught error: Timeout while waiting on RPC response.</i></div>
<div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack Traceback (most recent call last):</i></div><div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 81, in __call__</i></div>
<div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack return req.get_response(self.application)</i></div><div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send</i></div>
<div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack application, catch_exc_info=False)</i></div><div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application</i></div>
<div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack app_iter = application(self.environ, start_response)</i></div><div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__</i></div>
<div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack return resp(environ, start_response)</i></div><div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 450, in __call__</i></div>
<div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack return self.app(env, start_response)</i></div><div><i>...<br></i></div><div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 551, in __iter__<br>
</i></div><div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack self._iterator.next()</i></div><div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 648, in iterconsume</i></div>
<div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack yield self.ensure(_error_callback, _consume)</i></div><div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 566, in ensure</i></div>
<div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack error_callback(e)</i></div><div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 629, in _error_callback</i></div>
<div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack raise rpc_common.Timeout()</i></div><div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack Timeout: Timeout while waiting on RPC response.</i></div>
<div><i>2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack </i></div><div><i>2013-08-07 18:05:55.488 INFO nova.api.openstack [req-0cfe760f-2e74-4e92-919c-663ba02c7f2f 20390b639d4449c18926dca5e038ec5e d1e3aae242f14c488d2225dcbf1e96d6] <a href="http://172.19.136.13:8774/v2/d1e3aae242f14c488d2225dcbf1e96d6/servers" target="_blank">http://172.19.136.13:8774/v2/d1e3aae242f14c488d2225dcbf1e96d6/servers</a> returned with HTTP 500</i></div>
<div><i>2013-08-07 18:05:55.488 DEBUG nova.api.openstack.wsgi [req-0cfe760f-2e74-4e92-919c-663ba02c7f2f 20390b639d4449c18926dca5e038ec5e d1e3aae242f14c488d2225dcbf1e96d6] Returning 500 to user: The server has either erred or is incapable of performing the requested operation. __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1165</i></div>
<div><i>2013-08-07 18:05:55.489 INFO nova.osapi_compute.wsgi.server [req-0cfe760f-2e74-4e92-919c-663ba02c7f2f 20390b639d4449c18926dca5e038ec5e d1e3aae242f14c488d2225dcbf1e96d6] 172.19.136.13 "POST /v2/d1e3aae242f14c488d2225dcbf1e96d6/servers HTTP/1.1" status: 500 len: 335 time: 60.5262640</i></div>
<div><br></div><div>A couple of things about my deployment that may help you help me:</div><div>-One controller node running: nova-conductor, nova-scheduler, keystone, quantum-server, rabbitmq</div><div>-One compute node running: nova-api, nova-compute, glance</div>
<div>-One storage node running cinder</div><div><br></div><div>My ideas:</div><div>-I think it could be a problem related to nova-compute using nova-conductor (i really don't know how to tell nova to use it...), somehow messages from nova-compute doesn't reach nova-conductor on the controller node eventhough that nova-compute is connected to rabbit and so is nova-conductor.</div>
<div>-I haven't found any message like "wrong password for rabbit" on any log file.</div><span><font color="#888888"><div><br></div><div><br></div><div><br></div>-- <br><div dir="ltr">Pavlik Salles Juan José</div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Pavlik Salles Juan José</div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Pavlik Salles Juan José</div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Pavlik Salles Juan José</div>
</div>
</div></div><br></div></div>_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Pavlik Salles Juan José</div>
</div>
</div></div></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Pavlik Salles Juan José</div>
</div>
</div></div><br>_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br></blockquote></div><br></div>