Open Stack

Hi Sam,

I never did get the tests working right, but your updates to them look good.  The functional piece of the patch is exactly the same as what we’re doing, though.  We’ve had it running in our lab environments for a couple weeks, and I’m beginning the production upgrades today.  Everything else nova-wise has been working well for us.

Thanks!
Mike



On 6/1/16, 9:50 PM, "Sam Morrison" <sorrison at gmail.com> wrote:

>Hey Mike,
>
>I have updated our patch for this https://github.com/NeCTAR-RC/nova/commit/debd88652bd9f32f91ab68dd1a3979a7cdc9ec80
>
>Is that what you’re doing too?
>
>Cheers,
>Sam
>
>
>> On 20 May 2016, at 9:26 AM, Sam Morrison <sorrison at gmail.com> wrote:
>> 
>> Ha, yeah have come across this too in our tempest runs. Haven’t had a chance to look into it yet. But I think you are on the right track. Just need to modify the tests.
>> 
>> Cheers,
>> Sam
>> 
>> 
>> 
>>> On 20 May 2016, at 1:25 AM, Mike Dorman <mdorman at godaddy.com> wrote:
>>> 
>>> We are struggling a bit with the "Send up AZ and instance_name in system metadata” patch.  It works fine under normal circumstances, but if there is a scheduler error (No valid host was found), then there is a failure when transitioning the instance to an error state, and the instance gets stuck in scheduling forever:
>>> 
>>> 2016-05-19 08:15:07.677 24589 WARNING nova.scheduler.utils [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA f48e57277a7a484290ba9afdc49a21a9 - - -] [instance: 71009301-7377-49f7-9f9e-f4e718bd5c83] Setting instance to ERROR state.
>>> 2016-05-19 08:15:07.679 24589 DEBUG nova.objects.instance [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA f48e57277a7a484290ba9afdc49a21a9 - - -] Lazy-loading `system_metadata' on Instance uuid 71009301-7377-49f7-9f9e-f4e718bd5c83 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:873
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher [req-b9e5b8b4-10bd-44d5-98d0-d89db70698bc 02b775qlx5jA f48e57277a7a484290ba9afdc49a21a9 - - -] Exception during message handling: Object action obj_load_attr failed because: attribute id not lazy-loadable
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 752, in build_instances
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     exc, request_spec)
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 625, in _set_vm_state_and_notify
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     ex, request_spec, self.db)
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 111, in set_vm_state_and_notify
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     instance.save()
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 213, in wrapper
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     return fn(self, *args, **kwargs)
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 615, in save
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     self._ensure_cells_system_metadata()
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 555, in _ensure_cells_system_metadata
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     self.system_metadata[u'instance_name'] = self.name
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 224, in name
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     base_name = CONF.instance_name_template % self.id
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 66, in getter
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     self.obj_load_attr(name)
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 864, in obj_load_attr
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher     reason='attribute %s not lazy-loadable' % attrname)
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher ObjectActionError: Object action obj_load_attr failed because: attribute id not lazy-loadable
>>> 2016-05-19 08:15:07.741 24589 ERROR oslo_messaging.rpc.dispatcher
>>> 
>>> 
>>> The problem is that _ensure_cells_system_metadata is referencing self.name before the object is actually loaded from the database, so only uuid and I think vm_state are set at that point.  The name accessor method references self.id, which (for some reason) is not lazy loadable.
>>> 
>>> So I tried moving the _ensure_cells_system_metadata later in the save method, after the object is loaded from the database (here: https://github.com/openstack/nova/blob/stable/liberty/nova/objects/instance.py#L676 )  That seems to work in practice, but it causes some of the tox tests to fail:  https://gist.github.com/misterdorm/cc7dfd235ebcc2a23009b9115b58e4d5
>>> 
>>> Anyways, I’m at a bit of a loss here and curious if anybody might have some better insights.
>>> 
>>> Thanks,
>>> Mike
>>> 
>>> 
>>> 
>>> From:  Sam Morrison <sorrison at gmail.com>
>>> Date:  Wednesday, May 4, 2016 at 6:23 PM
>>> To:  Mike Dorman <mdorman at godaddy.com>
>>> Cc:  OpenStack Operators <openstack-operators at lists.openstack.org>
>>> Subject:  Re: [Openstack-operators] Nova cells patches against Liberty
>>> 
>>> 
>>> Hi Mike,
>>> 
>>> I’ve also been working on these and have some updated patches at:
>>> 
>>> https://github.com/NeCTAR-RC/nova/commits/stable/liberty-cellsv1
>>> 
>>> There are a couple of patches that you have in your tree that need updating for Liberty. Mainly around supporting the v2.1 API and more things moved to objects. I have also written some tests for some more of them too. I haven’t tested all of these functionally
>>> yet but they pass all tox tests.
>>> 
>>> Cheers,
>>> Sam
>>> 
>>> 
>>> 
>>> 
>>> On 5 May 2016, at 4:19 AM, Mike Dorman <mdorman at godaddy.com> wrote:
>>> 
>>> I went ahead and pulled out the Nova cells patches we’re running against Liberty so that others can use them if so desired.
>>> 
>>> https://github.com/godaddy/openstack-nova-patches
>>> 
>>> Usual disclaimers apply here, your mileage may vary, these may not work as expected in your environment, etc.  We have tested these at a basic level (unit tests), but are not running these for Liberty in real production yet.
>>> 
>>> Mike
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> OpenStack-operators mailing list
>>> OpenStack-operators at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>