Open Stack

On 11/21/2015 3:47 PM, Belmiro Moreira wrote:
> Hi,
> We are about to upgrade nova to kilo using cells and we noticed
> the resize/migrate functionality is not working properly.
>
> The instance is correctly resized/migrated but fails to
> “confirm resize” with the following trace:
>
> 2015-11-21 22:40:49.804 26786 ERROR nova.api.openstack.wsgi
> [req-67f6a225-602c-426c-a33a-02c3f5e3093d belmiro
> 40c7729b-c8e5-432f-81d5-a41543c8bf13] Exception handling resource:
> 'NoneType' object has no attribute '__getitem__'
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi Traceback
> (most recent call last):
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821,
> in _process_stack
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi
> action_result = self.dispatch(meth, request, action_args)
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911,
> in dispatch
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     return
> method(req=request, **action_args)
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line
> 823, in _action_confirm_resize
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi
> self.compute_api.confirm_resize(context, instance)
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 233, in wrapped
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     return
> func(self, context, target, *args, **kwargs)
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 222, in inner
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     return
> function(self, context, instance, *args, **kwargs)
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 250, in
> _wrapped
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     return
> fn(self, context, instance, *args, **kwargs)
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 203, in inner
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     return
> f(self, context, instance, *args, **kw)
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2452, in
> confirm_resize
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     deltas =
> self._downsize_quota_delta(context, instance)
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2537, in
> _downsize_quota_delta
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     return
> API._resize_quota_delta(context, new_flavor, old_flavor, 1, -1)
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2504, in
> _resize_quota_delta
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     if
> compare * _quota_delta('vcpus') > 0:
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2501, in
> _quota_delta
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     return
> sense * (new_flavor[resource] - old_flavor[resource])
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi TypeError:
> 'NoneType' object has no attribute '__getitem__'
> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi
>
>
>
> I can't reproduce in master so this must have been fixed meanwhile,
> however I can't identify the patch.
>
> Is someone experiencing the same problem? and know the commit that
> fixed it?
>
> thanks,
> Belmiro
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

I'm not seeing anything very different in that specific utility code 
path in liberty or mitaka, so I'd have to assume there is something 
going on in the online flavor migration when lazy-loading the flavor 
from the instance object such that the old or new flavor is None.

Are you rolling to kilo 2015.1.0 or the latest release 2015.1.2?

Are there any debug log messages before the error/trace where it says 
it's lazy-loading the flavor and doing anything there? If you know the 
instance in question, you could check the instance_system_metadata and 
instance_extra tables to see if the old/new flavor information has been 
migrated successfully.

I don't think you'd have this problem in master because the online 
flavor migration was dropped in liberty (there is a DB migration that 
stops you from continuing if you haven't manually migrated the flavors yet).

-- 

Thanks,

Matt Riedemann