[Openstack-operators] Fwd: Can't start a vm, from dashboard
Juan José Pavlik Salles
jjpavlik at gmail.com
Fri Aug 9 02:13:11 UTC 2013
I really appreciate you help Mike! You saved me lots of hours!!! I didn't
now nova-api needs access to the DB so i never included
sql_connection=mysql://nova:PASSWORD@DB_HOST/nova
in the nova.conf of my compute node. I just added the line and everything
start working!!! This makes me think i should move nova-api to the
controller node instead of the compute node.
Thanks again!!!
2013/8/8 Mike Wilson <geekinutah at gmail.com>
> Anytime you call the nova API and ask it to boot an instance, before it
> starts handing things off the the scheduler it creates an action in the
> database. Your scheduler is complaining that it can't find that action. So
> in your initial email you don't say where your database is. Are you running
> a mysql host somewhere? Nova-api is going to need to be able to get at the
> same database as nova-scheduler. I would start poking around there and see
> if that's your problem.
>
> -Mike
>
>
> On Thu, Aug 8, 2013 at 6:06 PM, Juan José Pavlik Salles <
> jjpavlik at gmail.com> wrote:
>
>>
>> You are right Wilson, there's no hostiD in the vm description:
>>
>>
>> +-----------------------------+-----------------------------------------------------------+
>> | Property | Value
>> |
>>
>> +-----------------------------+-----------------------------------------------------------+
>> | status | BUILD
>> |
>> | updated | 2013-08-08T19:23:01Z
>> |
>> | OS-EXT-STS:task_state | scheduling
>> |
>> | key_name | None
>> |
>> | image | Ubuntu 12.04.2 LTS
>> (1359ca8d-23a2-40e8-940f-d90b3e68bb39) |
>> | hostId |
>> |
>> | OS-EXT-STS:vm_state | building
>> |
>> | flavor | m1.tiny (1)
>> |
>> | id | b0583cca-63c2-481f-8b94-7aeb2e86641f
>> |
>> | security_groups | [{u'name': u'default'}]
>> |
>> | user_id | 20390b639d4449c18926dca5e038ec5e
>> |
>> | name | prueba11
>> |
>> | created | 2013-08-08T19:19:44Z
>> |
>> | tenant_id | d1e3aae242f14c488d2225dcbf1e96d6
>> |
>> | OS-DCF:diskConfig | MANUAL
>> |
>> | metadata | {}
>> |
>> | accessIPv4 |
>> |
>> | accessIPv6 |
>> |
>> | progress | 0
>> |
>> | OS-EXT-STS:power_state | 0
>> |
>> | OS-EXT-AZ:availability_zone | nova
>> |
>> | config_drive |
>> |
>>
>> +-----------------------------+-----------------------------------------------------------+
>>
>> I found this new log in the scheduler, it must be related to it:
>>
>> 2013-08-08 19:19:46.000 ERROR nova.openstack.common.rpc.amqp
>> [req-b049c969-c411-4b76-9aa5-ad88d714c4ab 20390b639d4449c18926dca5e038ec5e
>> d1e3aae242f14c488d2225dcbf1e96d6] Exception during message handling
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp
>> Traceback (most recent call last):
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>> 430, in _process_data
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp
>> rval = self.proxy.dispatch(ctxt, version, method, **args)
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>> line 133, in dispatch
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp
>> return getattr(proxyobj, method)(ctxt, **kwargs)
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 127, in
>> run_instance
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp
>> 'schedule', *instance_uuids):
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 318, in
>> __enter__
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp
>> self.conductor.action_event_start(self.context, event)
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 264, in
>> action_event_start
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp
>> return self._manager.action_event_start(context, values)
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 1348, in wrapper
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp
>> return func(*args, **kwargs)
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 334, in
>> action_event_start
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp
>> evt = self.db.action_event_start(context, values)
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 1625, in
>> action_event_start
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp
>> return IMPL.action_event_start(context, values)
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 4624, in
>> action_event_start
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp
>> instance_uuid=values['instance_uuid'])
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp
>> InstanceActionNotFound: Action for request_id
>> req-b049c969-c411-4b76-9aa5-ad88d714c4ab on instance
>> b0583cca-63c2-481f-8b94-7aeb2e86641f not found
>> 2013-08-08 19:19:46.000 29114 TRACE nova.openstack.common.rpc.amqp
>>
>> It's getting me crazy. "InstanceActionNotFound..." does it mean something
>> to you?
>>
>>
>>
>> 2013/8/8 Mike Wilson <geekinutah at gmail.com>
>>
>>> Juan,
>>>
>>> If your instance show's the task_state as "scheduling" it's possible
>>> that your new instance never made it to the scheduler. When you do a nova
>>> show <instance_id> does it say which host node it is supposed to be on? If
>>> it does then the scheduler probably did its thing but failed the RPC to the
>>> compute node, if it doesn't have a host then your problem is that the
>>> message never made it from nova-api to nova-scheduler.
>>>
>>> In any case, it looks like rabbit is not quite setup correctly on your
>>> end.
>>>
>>> -Mike
>>>
>>>
>>> On Wed, Aug 7, 2013 at 6:55 PM, Juan José Pavlik Salles <
>>> jjpavlik at gmail.com> wrote:
>>>
>>>> Is there any way i can test nova-conductor and nova-scheduler to be
>>>> sure they are working like they should? If i list nova-manage service list,
>>>> everything is fine. I'm running out of ideas hahaha.
>>>>
>>>>
>>>> 2013/8/7 Juan José Pavlik Salles <jjpavlik at gmail.com>
>>>>
>>>>> According to the doc this problem should be related to some service
>>>>> that isn't answering to nova-api. I just have 3 servers in my deployment,
>>>>> so i don't think this is problem related to the amount of messages in the
>>>>> queues.
>>>>>
>>>>>
>>>>> 2013/8/7 Juan José Pavlik Salles <jjpavlik at gmail.com>
>>>>>
>>>>>> Here i have more information, i tried to boot a vm from the CLI and
>>>>>> it doesn't really fail. But when i check the vms status in the dashboard it
>>>>>> says "Scheduling" and never changes its state to "running" or "error".
>>>>>>
>>>>>>
>>>>>> 2013/8/7 Juan José Pavlik Salles <jjpavlik at gmail.com>
>>>>>>
>>>>>>> I just finished installing everything, i tried to create my first VM
>>>>>>> from the dashboard but it doesn't work. After choosing flavor and hitting
>>>>>>> launch it starts "creating" it but after a few seconds it stops saying:
>>>>>>> "Error: There was an error submitting the form. Please try again.". The
>>>>>>> only place where i found something related is in nova.log in my compute
>>>>>>> node, here is the log:
>>>>>>>
>>>>>>> *2013-08-07 18:05:55.293 DEBUG nova.openstack.common.rpc.common
>>>>>>> [req-0cfe760f-2e74-4e92-919c-663ba02c7f2f 20390b639d4449c18926dca5e038ec5e
>>>>>>> d1e3aae242f14c488d2225dcbf1e96d6] Timed out waiting for RPC response: timed
>>>>>>> out _error_callback
>>>>>>> /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py:628
>>>>>>> *
>>>>>>> *2013-08-07 18:05:55.479 DEBUG nova.quota
>>>>>>> [req-0cfe760f-2e74-4e92-919c-663ba02c7f2f 20390b639d4449c18926dca5e038ec5e
>>>>>>> d1e3aae242f14c488d2225dcbf1e96d6] Rolled back reservations
>>>>>>> ['3e941a2b-2cc6-4f01-8dc1-13dc09369141',
>>>>>>> '411f6f70-415e-4a21-aa06-3980070d6095',
>>>>>>> 'd4791eb7-b75a-4ab8-bfdb-5d5cd201e40d'] rollback
>>>>>>> /usr/lib/python2.7/dist-packages/nova/quota.py:1012*
>>>>>>> *2013-08-07 18:05:55.480 ERROR nova.api.openstack
>>>>>>> [req-0cfe760f-2e74-4e92-919c-663ba02c7f2f 20390b639d4449c18926dca5e038ec5e
>>>>>>> d1e3aae242f14c488d2225dcbf1e96d6] Caught error: Timeout while waiting on
>>>>>>> RPC response.*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack Traceback
>>>>>>> (most recent call last):*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 81,
>>>>>>> in __call__*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack return
>>>>>>> req.get_response(self.application)*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File
>>>>>>> "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
>>>>>>> *
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack
>>>>>>> application, catch_exc_info=False)*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File
>>>>>>> "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in
>>>>>>> call_application*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack
>>>>>>> app_iter = application(self.environ, start_response)*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File
>>>>>>> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
>>>>>>> *
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack return
>>>>>>> resp(environ, start_response)*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File
>>>>>>> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
>>>>>>> line 450, in __call__*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack return
>>>>>>> self.app(env, start_response)*
>>>>>>> *...
>>>>>>> *
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>> 551, in __iter__
>>>>>>> *
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack
>>>>>>> self._iterator.next()*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>>>>>>> line 648, in iterconsume*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack yield
>>>>>>> self.ensure(_error_callback, _consume)*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>>>>>>> line 566, in ensure*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack
>>>>>>> error_callback(e)*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>>>>>>> line 629, in _error_callback*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack raise
>>>>>>> rpc_common.Timeout()*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack Timeout:
>>>>>>> Timeout while waiting on RPC response.*
>>>>>>> *2013-08-07 18:05:55.480 29278 TRACE nova.api.openstack *
>>>>>>> *2013-08-07 18:05:55.488 INFO nova.api.openstack
>>>>>>> [req-0cfe760f-2e74-4e92-919c-663ba02c7f2f 20390b639d4449c18926dca5e038ec5e
>>>>>>> d1e3aae242f14c488d2225dcbf1e96d6]
>>>>>>> http://172.19.136.13:8774/v2/d1e3aae242f14c488d2225dcbf1e96d6/serversreturned with HTTP 500
>>>>>>> *
>>>>>>> *2013-08-07 18:05:55.488 DEBUG nova.api.openstack.wsgi
>>>>>>> [req-0cfe760f-2e74-4e92-919c-663ba02c7f2f 20390b639d4449c18926dca5e038ec5e
>>>>>>> d1e3aae242f14c488d2225dcbf1e96d6] Returning 500 to user: The server has
>>>>>>> either erred or is incapable of performing the requested operation.
>>>>>>> __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1165
>>>>>>> *
>>>>>>> *2013-08-07 18:05:55.489 INFO nova.osapi_compute.wsgi.server
>>>>>>> [req-0cfe760f-2e74-4e92-919c-663ba02c7f2f 20390b639d4449c18926dca5e038ec5e
>>>>>>> d1e3aae242f14c488d2225dcbf1e96d6] 172.19.136.13 "POST
>>>>>>> /v2/d1e3aae242f14c488d2225dcbf1e96d6/servers HTTP/1.1" status: 500 len: 335
>>>>>>> time: 60.5262640*
>>>>>>>
>>>>>>> A couple of things about my deployment that may help you help me:
>>>>>>> -One controller node running: nova-conductor, nova-scheduler,
>>>>>>> keystone, quantum-server, rabbitmq
>>>>>>> -One compute node running: nova-api, nova-compute, glance
>>>>>>> -One storage node running cinder
>>>>>>>
>>>>>>> My ideas:
>>>>>>> -I think it could be a problem related to nova-compute using
>>>>>>> nova-conductor (i really don't know how to tell nova to use it...), somehow
>>>>>>> messages from nova-compute doesn't reach nova-conductor on the controller
>>>>>>> node eventhough that nova-compute is connected to rabbit and so is
>>>>>>> nova-conductor.
>>>>>>> -I haven't found any message like "wrong password for rabbit" on any
>>>>>>> log file.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pavlik Salles Juan José
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Pavlik Salles Juan José
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Pavlik Salles Juan José
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Pavlik Salles Juan José
>>>>
>>>> _______________________________________________
>>>> OpenStack-operators mailing list
>>>> OpenStack-operators at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>>
>>>>
>>>
>>
>>
>> --
>> Pavlik Salles Juan José
>>
>>
>>
>> --
>> Pavlik Salles Juan José
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>
--
Pavlik Salles Juan José
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20130808/c7464ce4/attachment.html>
More information about the OpenStack-operators
mailing list