Open Stack

Hey Mike,

I have updated our patch for this https://github.com/NeCTAR-RC/nova/commit/debd88652bd9f32f91ab68dd1a3979a7cdc9ec80

Is that what you’re doing too?

Cheers,
Sam


> On 20 May 2016, at 9:26 AM, Sam Morrison <sorrison at gmail.com> wrote:
> 
> Ha, yeah have come across this too in our tempest runs. Haven’t had a chance to look into it yet. But I think you are on the right track. Just need to modify the tests.
> 
> Cheers,
> Sam
> 
> 
> 
>> On 20 May 2016, at 1:25 AM, Mike Dorman <mdorman at godaddy.com> wrote:
>> 
>> We are struggling a bit with the "Send up AZ and instance_name in system metadata” patch.  It works fine under normal circumstances, but if there is a scheduler error (No valid host was found), then there is a failure when transitioning the instance to an error state, and the instance gets stuck in scheduling forever:
>> 
>> 2016-05-19 08:15:07.677 24589 WARNING nova.scheduler.utils [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA f48e57277a7a484290ba9afdc49a21a9 - - -] [instance: 71009301-7377-49f7-9f9e-f4e718bd5c83] Setting instance to ERROR state.
>> 2016-05-19 08:15:07.679 24589 DEBUG nova.objects.instance [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA f48e57277a7a484290ba9afdc49a21a9 - - -] Lazy-loading `system_metadata' on Instance uuid 71009301-7377-49f7-9f9e-f4e718bd5c83 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:873
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA f48e57277a7a484290ba9afdc49a21a9 - - -] Exception during message handling: Object action obj_load_attr failed because: attribute id not lazy-loadable
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 752, in build_instances
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     exc, request_spec)
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 625, in _set_vm_state_and_notify
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     ex, request_spec, self.db)
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 111, in set_vm_state_and_notify
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     instance.save()
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 213, in wrapper
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     return fn(self, *args, **kwargs)
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 615, in save
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     self._ensure_cells_system_metadata()
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 555, in _ensure_cells_system_metadata
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     self.system_metadata[u'instance_name'] = self.name
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 224, in name
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     base_name = CONF.instance_name_template % self.id
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 66, in getter
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     self.obj_load_attr(name)
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 864, in obj_load_attr
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     reason='attribute %s not lazy-loadable' % attrname)
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher ObjectActionError: Object action obj_load_attr failed because: attribute id not lazy-loadable
>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher
>> 
>> 
>> The problem is that _ensure_cells_system_metadata is referencing self.name before the object is actually loaded from the database, so only uuid and I think vm_state are set at that point.  The name accessor method references self.id, which (for some reason) is not lazy loadable.
>> 
>> So I tried moving the _ensure_cells_system_metadata later in the save method, after the object is loaded from the database (here: https://github.com/openstack/nova/blob/stable/liberty/nova/objects/instance.py#L676 )  That seems to work in practice, but it causes some of the tox tests to fail:  https://gist.github.com/misterdorm/cc7dfd235ebcc2a23009b9115b58e4d5
>> 
>> Anyways, I’m at a bit of a loss here and curious if anybody might have some better insights.
>> 
>> Thanks,
>> Mike
>> 
>> 
>> 
>> From:  Sam Morrison <sorrison at gmail.com>
>> Date:  Wednesday, May 4, 2016 at 6:23 PM
>> To:  Mike Dorman <mdorman at godaddy.com>
>> Cc:  OpenStack Operators <openstack-operators at lists.openstack.org>
>> Subject:  Re: [Openstack-operators] Nova cells patches against Liberty
>> 
>> 
>> Hi Mike,
>> 
>> I’ve also been working on these and have some updated patches at:
>> 
>> https://github.com/NeCTAR-RC/nova/commits/stable/liberty-cellsv1
>> 
>> There are a couple of patches that you have in your tree that need updating for Liberty. Mainly around supporting the v2.1 API and more things moved to objects. I have also written some tests for some more of them too. I haven’t tested all of these functionally
>> yet but they pass all tox tests.
>> 
>> Cheers,
>> Sam
>> 
>> 
>> 
>> 
>> On 5 May 2016, at 4:19 AM, Mike Dorman <mdorman at godaddy.com> wrote:
>> 
>> I went ahead and pulled out the Nova cells patches we’re running against Liberty so that others can use them if so desired.
>> 
>> https://github.com/godaddy/openstack-nova-patches
>> 
>> Usual disclaimers apply here, your mileage may vary, these may not work as expected in your environment, etc.  We have tested these at a basic level (unit tests), but are not running these for Liberty in real production yet.
>> 
>> Mike
>> 
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> 
>> 
>> 
>> 
>> 
>> 
>