Open Stack

Ha, yeah have come across this too in our tempest runs. Haven’t had a chance to look into it yet. But I think you are on the right track. Just need to modify the tests.

Cheers,
Sam



> On 20 May 2016, at 1:25 AM, Mike Dorman <mdorman at godaddy.com> wrote:
> 
> We are struggling a bit with the "Send up AZ and instance_name in system metadata” patch.  It works fine under normal circumstances, but if there is a scheduler error (No valid host was found), then there is a failure when transitioning the instance to an error state, and the instance gets stuck in scheduling forever:
> 
> 2016-05-19 08:15:07.677 24589 WARNING nova.scheduler.utils [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA f48e57277a7a484290ba9afdc49a21a9 - - -] [instance: 71009301-7377-49f7-9f9e-f4e718bd5c83] Setting instance to ERROR state.
> 2016-05-19 08:15:07.679 24589 DEBUG nova.objects.instance [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA f48e57277a7a484290ba9afdc49a21a9 - - -] Lazy-loading `system_metadata' on Instance uuid 71009301-7377-49f7-9f9e-f4e718bd5c83 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:873
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA f48e57277a7a484290ba9afdc49a21a9 - - -] Exception during message handling: Object action obj_load_attr failed because: attribute id not lazy-loadable
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 752, in build_instances
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     exc, request_spec)
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 625, in _set_vm_state_and_notify
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     ex, request_spec, self.db)
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 111, in set_vm_state_and_notify
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     instance.save()
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 213, in wrapper
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     return fn(self, *args, **kwargs)
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 615, in save
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     self._ensure_cells_system_metadata()
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 555, in _ensure_cells_system_metadata
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     self.system_metadata[u'instance_name'] = self.name
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 224, in name
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     base_name = CONF.instance_name_template % self.id
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 66, in getter
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     self.obj_load_attr(name)
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 864, in obj_load_attr
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     reason='attribute %s not lazy-loadable' % attrname)
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher ObjectActionError: Object action obj_load_attr failed because: attribute id not lazy-loadable
> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher
> 
> 
> The problem is that _ensure_cells_system_metadata is referencing self.name before the object is actually loaded from the database, so only uuid and I think vm_state are set at that point.  The name accessor method references self.id, which (for some reason) is not lazy loadable.
> 
> So I tried moving the _ensure_cells_system_metadata later in the save method, after the object is loaded from the database (here: https://github.com/openstack/nova/blob/stable/liberty/nova/objects/instance.py#L676 )  That seems to work in practice, but it causes some of the tox tests to fail:  https://gist.github.com/misterdorm/cc7dfd235ebcc2a23009b9115b58e4d5
> 
> Anyways, I’m at a bit of a loss here and curious if anybody might have some better insights.
> 
> Thanks,
> Mike
> 
> 
> 
> From:  Sam Morrison <sorrison at gmail.com>
> Date:  Wednesday, May 4, 2016 at 6:23 PM
> To:  Mike Dorman <mdorman at godaddy.com>
> Cc:  OpenStack Operators <openstack-operators at lists.openstack.org>
> Subject:  Re: [Openstack-operators] Nova cells patches against Liberty
> 
> 
> Hi Mike,
> 
> I’ve also been working on these and have some updated patches at:
> 
> https://github.com/NeCTAR-RC/nova/commits/stable/liberty-cellsv1
> 
> There are a couple of patches that you have in your tree that need updating for Liberty. Mainly around supporting the v2.1 API and more things moved to objects. I have also written some tests for some more of them too. I haven’t tested all of these functionally
> yet but they pass all tox tests.
> 
> Cheers,
> Sam
> 
> 
> 
> 
> On 5 May 2016, at 4:19 AM, Mike Dorman <mdorman at godaddy.com> wrote:
> 
> I went ahead and pulled out the Nova cells patches we’re running against Liberty so that others can use them if so desired.
> 
> https://github.com/godaddy/openstack-nova-patches
> 
> Usual disclaimers apply here, your mileage may vary, these may not work as expected in your environment, etc.  We have tested these at a basic level (unit tests), but are not running these for Liberty in real production yet.
> 
> Mike
> 
> 
> 
> 
> 
> 
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> 
> 
> 
> 
> 
>