<div dir="ltr">Are all of your instance_types tables identical across the cell/API DBs? And also marked is_public?<div><br></div><div><div><br></div><div>-AH </div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 23, 2015 at 6:25 PM, Belmiro Moreira <span dir="ltr"><<a href="mailto:moreira.belmiro.email.lists@gmail.com" target="_blank">moreira.belmiro.email.lists@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Matt,<div>I'm rolling kilo 2015.1.1</div><div><br></div><div>After resizing an instance (state: verify_resize) I have the following in the instance_extra table:</div><div><br></div><div>API cell DB:</div><div>{"new": null, "old": null, "cur": {"nova_object.version": "1.1", "nova_object.changes": ["root_gb", "name", "ephemeral_gb", "memory_mb", "vcpus", "extra_specs", "swap", "rxtx_factor", "flavorid", "vcpu_weight", "id"], "<a href="http://nova_object.name" target="_blank">nova_object.name</a>": "Flavor", "nova_object.data": {"root_gb": 20, "name": "m1.small", "ephemeral_gb": 0, "memory_mb": 2048, "vcpus": 1, "extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "flavorid": "2", "vcpu_weight": null, "id": 5}, "nova_object.namespace": "nova"}}<br></div><div><br></div><div>and in the compute cell DB:</div><div>{"new": {"nova_object.version": "1.1", "nova_object.changes": ["deleted", "ephemeral_gb", "updated_at", "disabled", "extra_specs", "rxtx_factor", "is_public", "flavorid", "id", "root_gb", "deleted_at", "name", "created_at", "memory_mb", "vcpus", "swap", "vcpu_weight"], "<a href="http://nova_object.name" target="_blank">nova_object.name</a>": "Flavor", "nova_object.data": {"disabled": false, "root_gb": 40, "name": "m1.medium", "flavorid": "3", "deleted": false, "created_at": null, "ephemeral_gb": 0, "updated_at": null, "memory_mb": 4096, "vcpus": 2, "extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "is_public": true, "deleted_at": null, "vcpu_weight": 0, "id": 1}, "nova_object.namespace": "nova"}, "old": {"nova_object.version": "1.1", "nova_object.changes": ["root_gb", "name", "ephemeral_gb", "memory_mb", "vcpus", "extra_specs", "swap", "rxtx_factor", "flavorid", "vcpu_weight", "id"], "<a href="http://nova_object.name" target="_blank">nova_object.name</a>": "Flavor", "nova_object.data": {"root_gb": 20, "name": "m1.small", "ephemeral_gb": 0, "memory_mb": 2048, "vcpus": 1, "extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "flavorid": "2", "vcpu_weight": null, "id": 5}, "nova_object.namespace": "nova"}, "cur": {"nova_object.version": "1.1", "nova_object.changes": ["deleted", "ephemeral_gb", "updated_at", "disabled", "extra_specs", "rxtx_factor", "is_public", "flavorid", "id", "root_gb", "deleted_at", "name", "created_at", "memory_mb", "vcpus", "swap", "vcpu_weight"], "<a href="http://nova_object.name" target="_blank">nova_object.name</a>": "Flavor", "nova_object.data": {"disabled": false, "root_gb": 40, "name": "m1.medium", "flavorid": "3", "deleted": false, "created_at": null, "ephemeral_gb": 0, "updated_at": null, "memory_mb": 4096, "vcpus": 2, "extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "is_public": true, "deleted_at": null, "vcpu_weight": 0, "id": 1}, "nova_object.namespace": "nova"}}<br></div><div><br></div><div><br></div><div>The quota is calculated in the API cell and the new flavor information is missing there.</div><div>I don't see any relevant/related logs using DEBUG.</div><div><br></div><div>thanks,</div><div>Belmiro</div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Nov 22, 2015 at 1:02 AM, Matt Riedemann <span dir="ltr"><<a href="mailto:mriedem@linux.vnet.ibm.com" target="_blank">mriedem@linux.vnet.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><br>
<br>
On 11/21/2015 3:47 PM, Belmiro Moreira wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>
Hi,<br>
We are about to upgrade nova to kilo using cells and we noticed<br>
the resize/migrate functionality is not working properly.<br>
<br>
The instance is correctly resized/migrated but fails to<br>
“confirm resize” with the following trace:<br>
<br>
2015-11-21 22:40:49.804 26786 ERROR nova.api.openstack.wsgi<br>
[req-67f6a225-602c-426c-a33a-02c3f5e3093d belmiro<br>
40c7729b-c8e5-432f-81d5-a41543c8bf13] Exception handling resource:<br>
'NoneType' object has no attribute '__getitem__'<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi Traceback<br>
(most recent call last):<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi File<br>
"/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821,<br>
in _process_stack<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi<br>
action_result = self.dispatch(meth, request, action_args)<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi File<br>
"/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911,<br>
in dispatch<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return<br>
method(req=request, **action_args)<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi File<br>
"/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line<br>
823, in _action_confirm_resize<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi<br>
self.compute_api.confirm_resize(context, instance)<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi File<br>
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 233, in wrapped<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return<br>
func(self, context, target, *args, **kwargs)<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi File<br>
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 222, in inner<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return<br>
function(self, context, instance, *args, **kwargs)<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi File<br>
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 250, in<br>
_wrapped<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return<br>
fn(self, context, instance, *args, **kwargs)<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi File<br>
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 203, in inner<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return<br>
f(self, context, instance, *args, **kw)<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi File<br>
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2452, in<br>
confirm_resize<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi deltas =<br>
self._downsize_quota_delta(context, instance)<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi File<br>
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2537, in<br>
_downsize_quota_delta<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return<br>
API._resize_quota_delta(context, new_flavor, old_flavor, 1, -1)<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi File<br>
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2504, in<br>
_resize_quota_delta<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi if<br>
compare * _quota_delta('vcpus') > 0:<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi File<br>
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2501, in<br>
_quota_delta<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi return<br>
sense * (new_flavor[resource] - old_flavor[resource])<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi TypeError:<br>
'NoneType' object has no attribute '__getitem__'<br>
2015-11-21 22:40:49.804 26786 TRACE nova.api.openstack.wsgi<br>
<br>
<br>
<br>
I can't reproduce in master so this must have been fixed meanwhile,<br>
however I can't identify the patch.<br>
<br>
Is someone experiencing the same problem? and know the commit that<br>
fixed it?<br>
<br>
thanks,<br>
Belmiro<br>
<br>
<br></div></div>
_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br>
</blockquote>
<br>
I'm not seeing anything very different in that specific utility code path in liberty or mitaka, so I'd have to assume there is something going on in the online flavor migration when lazy-loading the flavor from the instance object such that the old or new flavor is None.<br>
<br>
Are you rolling to kilo 2015.1.0 or the latest release 2015.1.2?<br>
<br>
Are there any debug log messages before the error/trace where it says it's lazy-loading the flavor and doing anything there? If you know the instance in question, you could check the instance_system_metadata and instance_extra tables to see if the old/new flavor information has been migrated successfully.<br>
<br>
I don't think you'd have this problem in master because the online flavor migration was dropped in liberty (there is a DB migration that stops you from continuing if you haven't manually migrated the flavors yet).<span><font color="#888888"><br>
<br>
-- <br>
<br>
Thanks,<br>
<br>
Matt Riedemann<br>
<br>
<br>
_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
</font></span></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br></blockquote></div><br></div>