Open Stack

Hi Matt,
I'm rolling kilo 2015.1.1

After resizing an instance (state: verify_resize) I have the following in
the instance_extra table:

API cell DB:
{"new": null, "old": null, "cur": {"nova_object.version": "1.1",
"nova_object.changes": ["root_gb", "name", "ephemeral_gb", "memory_mb",
"vcpus", "extra_specs", "swap", "rxtx_factor", "flavorid", "vcpu_weight",
"id"], "nova_object.name": "Flavor", "nova_object.data": {"root_gb": 20,
"name": "m1.small", "ephemeral_gb": 0, "memory_mb": 2048, "vcpus": 1,
"extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "flavorid": "2",
"vcpu_weight": null, "id": 5}, "nova_object.namespace": "nova"}}

and in the compute cell DB:
{"new": {"nova_object.version": "1.1", "nova_object.changes": ["deleted",
"ephemeral_gb", "updated_at", "disabled", "extra_specs", "rxtx_factor",
"is_public", "flavorid", "id", "root_gb", "deleted_at", "name",
"created_at", "memory_mb", "vcpus", "swap", "vcpu_weight"], "
nova_object.name": "Flavor", "nova_object.data": {"disabled": false,
"root_gb": 40, "name": "m1.medium", "flavorid": "3", "deleted": false,
"created_at": null, "ephemeral_gb": 0, "updated_at": null, "memory_mb":
4096, "vcpus": 2, "extra_specs": {}, "swap": 0, "rxtx_factor": 1.0,
"is_public": true, "deleted_at": null, "vcpu_weight": 0, "id": 1},
"nova_object.namespace": "nova"}, "old": {"nova_object.version": "1.1",
"nova_object.changes": ["root_gb", "name", "ephemeral_gb", "memory_mb",
"vcpus", "extra_specs", "swap", "rxtx_factor", "flavorid", "vcpu_weight",
"id"], "nova_object.name": "Flavor", "nova_object.data": {"root_gb": 20,
"name": "m1.small", "ephemeral_gb": 0, "memory_mb": 2048, "vcpus": 1,
"extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "flavorid": "2",
"vcpu_weight": null, "id": 5}, "nova_object.namespace": "nova"}, "cur":
{"nova_object.version": "1.1", "nova_object.changes": ["deleted",
"ephemeral_gb", "updated_at", "disabled", "extra_specs", "rxtx_factor",
"is_public", "flavorid", "id", "root_gb", "deleted_at", "name",
"created_at", "memory_mb", "vcpus", "swap", "vcpu_weight"], "
nova_object.name": "Flavor", "nova_object.data": {"disabled": false,
"root_gb": 40, "name": "m1.medium", "flavorid": "3", "deleted": false,
"created_at": null, "ephemeral_gb": 0, "updated_at": null, "memory_mb":
4096, "vcpus": 2, "extra_specs": {}, "swap": 0, "rxtx_factor": 1.0,
"is_public": true, "deleted_at": null, "vcpu_weight": 0, "id": 1},
"nova_object.namespace": "nova"}}


The quota is calculated in the API cell and the new flavor information is
missing there.
I don't see any relevant/related logs using DEBUG.

thanks,
Belmiro


On Sun, Nov 22, 2015 at 1:02 AM, Matt Riedemann <mriedem at linux.vnet.ibm.com>
wrote:

>
>
> On 11/21/2015 3:47 PM, Belmiro Moreira wrote:
>
>> Hi,
>> We are about to upgrade nova to kilo using cells and we noticed
>> the resize/migrate functionality is not working properly.
>>
>> The instance is correctly resized/migrated but fails to
>> “confirm resize” with the following trace:
>>
>> 2015-11-21 22:40:49.804 26786 ERROR nova.api.openstack.wsgi
>> [req-67f6a225-602c-426c-a33a-02c3f5e3093d belmiro
>> 40c7729b-c8e5-432f-81d5-a41543c8bf13] Exception handling resource:
>> 'NoneType' object has no attribute '__getitem__'
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi Traceback
>> (most recent call last):
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
>> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821,
>> in _process_stack
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi
>> action_result = self.dispatch(meth, request, action_args)
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
>> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911,
>> in dispatch
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     return
>> method(req=request, **action_args)
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
>> "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py",
>> line
>> 823, in _action_confirm_resize
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi
>> self.compute_api.confirm_resize(context, instance)
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 233, in
>> wrapped
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     return
>> func(self, context, target, *args, **kwargs)
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 222, in inner
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     return
>> function(self, context, instance, *args, **kwargs)
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 250, in
>> _wrapped
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     return
>> fn(self, context, instance, *args, **kwargs)
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 203, in inner
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     return
>> f(self, context, instance, *args, **kw)
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2452, in
>> confirm_resize
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     deltas =
>> self._downsize_quota_delta(context, instance)
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2537, in
>> _downsize_quota_delta
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     return
>> API._resize_quota_delta(context, new_flavor, old_flavor, 1, -1)
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2504, in
>> _resize_quota_delta
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     if
>> compare * _quota_delta('vcpus') > 0:
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi   File
>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2501, in
>> _quota_delta
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi     return
>> sense * (new_flavor[resource] - old_flavor[resource])
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi TypeError:
>> 'NoneType' object has no attribute '__getitem__'
>> 2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi
>>
>>
>>
>> I can't reproduce in master so this must have been fixed meanwhile,
>> however I can't identify the patch.
>>
>> Is someone experiencing the same problem? and know the commit that
>> fixed it?
>>
>> thanks,
>> Belmiro
>>
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
> I'm not seeing anything very different in that specific utility code path
> in liberty or mitaka, so I'd have to assume there is something going on in
> the online flavor migration when lazy-loading the flavor from the instance
> object such that the old or new flavor is None.
>
> Are you rolling to kilo 2015.1.0 or the latest release 2015.1.2?
>
> Are there any debug log messages before the error/trace where it says it's
> lazy-loading the flavor and doing anything there? If you know the instance
> in question, you could check the instance_system_metadata and
> instance_extra tables to see if the old/new flavor information has been
> migrated successfully.
>
> I don't think you'd have this problem in master because the online flavor
> migration was dropped in liberty (there is a DB migration that stops you
> from continuing if you haven't manually migrated the flavors yet).
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20151124/b228d23f/attachment.html>