Open Stack

Hi all,

In kilo (yeah I know it's an old release, but still :)), I was getting a nova errors for DetachedInstanceError on instance.get_by_uuid().

2016-10-05 18:30:53.791 103618 ERROR nova.api.metadata.handler [req-5aa7b422-d6c0-424e-b40f-cea79d3a3963 - - - - -] Failed to get metadata for instance id: 859aba9c-7620-4cbf-a5aa-f6f29c320980
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler Traceback (most recent call last):
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.7/site-packages/nova/api/metadata/handler.py", line 220, in _handle_instance_id_request
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler     remote_address)
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.7/site-packages/nova/api/metadata/handler.py", line 106, in get_metadata_by_instance_id
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler     instance_id, address)
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.7/site-packages/nova/api/metadata/base.py", line 531, in get_metadata_by_instance_id
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler     instance = objects.Instance.get_by_uuid(ctxt, instance_id)
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.7/site-packages/nova/objects/base.py", line 163, in wrapper
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler     result = fn(cls, context, *args, **kwargs)
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 654, in get_by_uuid
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler     expected_attrs)
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 608, in _from_db_object
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler     instance._maybe_migrate_flavor(db_inst, expected_attrs)
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 535, in _maybe_migrate_flavor
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler     if instance_extra.get('flavor') is not None:
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 60, in get
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler     return getattr(self, key, default)
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 239, in __get__
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler     return self.impl.get(instance_state(instance), dict_)
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 591, in get
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler     value = self.callable_(state, passive)
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/strategies.py", line 279, in _load_for_state
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler     (orm_util.state_str(state), self.key)
2016-10-05 18:30:53.791 103618 TRACE nova.api.metadata.handler DetachedInstanceError: Parent instance <InstanceExtra at 0x6f43390> is not bound to a Session; deferred load operation of attribute 'flavor' cannot proceed

Now I've fixed it by adding a "joinedload" in the database query (in instance_get_by_uuid()) on 'extra'.

So this fixes the issue, what I'd like to understand is why in most cases (like 99.9% of the time), the code was still working without the "joinedload" ?  What could explain that the children object was loaded but at some point dropped from the session ?

I tried to reproduce the issue in mitaka (sorry didn't had a newton setup), but it appears I can't reproduce it there ...  I was hoping to reproduce it and make sure there is no latent bug in nova.

Has anybody seen this issue before or something similar ?

Thanks for the help,
/ludovic
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161007/d5fe0f47/attachment.html>