<html><head><base href="x-msg://315/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">All of the workers are single-threaded, so I'm not sure that scoped sessions are really necessary.<div><br></div><div>We did however decide that objects from the db layer are supposed to be simple dictionaries. We currently allow nested dictionaries to optimize joined objects. Unfortunately we never switched to sanitizing data from sqlalchemy, and instead we make the sqlalchemy objects provide a dictionary-like interface and pass the object itself.</div><div><br></div><div>The issue that you're seeing is because network wasn't properly 'joinedload'ed in the initial query, and because the data is not sanitized, sqlalchemy tries to joinedload, but the session has been terminated. If we had sanitized data, we would get a more useful error like a key error when network is accessed. The current solution is to add the proper joinedload.</div><div><br></div><div>One of the goals of the nova-database team is to do the necessary data sanitization and to remove as many of the joinedloads as possible (hopefully all of them).</div><div><br></div><div>Vish</div><div><br><div><div><div>On Oct 31, 2011, at 12:25 PM, Day, Phil wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; ">Hi Folks,<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; ">We’ve been looking into a problem which looks a lot like:<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><a href="https://bugs.launchpad.net/nova/+bug/855660" style="color: blue; text-decoration: underline; ">https://bugs.launchpad.net/nova/+bug/855660</a><o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; ">2011-10-21 14:13:31,035 ERROR nova.api [5bd52130-d46f-4702-b06b-9ca5045473d7 smokeuser smokeproject] Unexpected error raised: Parent instance <FixedIp at 0x4e74490> is not bound to a Session; lazy load operation of attribute 'network' cannot proceed<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: Traceback (most recent call last):<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/ec2/__init__.py", line 363, in __call__<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: result = api_request.invoke(context)<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/ec2/apirequest.py", line 90, in invoke<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: result = method(context, **args)<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/ec2/cloud.py", line 1195, in describe_instances<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: instance_id=instance_id)<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/ec2/cloud.py", line 1204, in _format_describe_instances<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: return {'reservationSet': self._format_instances(context, **kwargs)}<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/ec2/cloud.py", line 1309, in _format_instances<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: if fixed['network'] and use_v6:<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/models.py", line 76, in __getitem__<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: return getattr(self, key)<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 163, in __get__<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: instance_dict(instance))<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 383, in get<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: value = callable_(passive=passive)<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 595, in __call__<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: (mapperutil.state_str(state), self.key)<span class="Apple-converted-space"> </span><br>(nova.api): TRACE: DetachedInstanceError: Parent instance <FixedIp at 0x4e74490> is not bound to a Session; lazy load operation of attribute 'network' cannot proceed<span class="Apple-converted-space"> </span><br>(nova.api): TRACE:<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; ">As far as we can see the problem seems to be related to some conflict between multiple threads in the same API server instance and lazy loading of some part of the object.<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; ">Looking at the sqlalchemy documentation it seems to strongly suggest that when used from multi-threaded WSGI applications that scoped_sessions should be used (I’m not clear on the details but it seems that this effectively makes lazy load operations thread safe). However whilst this fixes the problem it has a bad effect on the unit tests – in particular it seems to upset all of the DB migration code used in the unit tests.<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; ">So does anyone know if there was an explicit decision / reason not to use scoped_sessions in Nova ?<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; ">Thanks,<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; ">Phil<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; ">PS: The other possible fix we’ve found is to change sqlalchemy/models.py so that the associations are explicitly set to use eager load – which also seems to fix the problem but feels like a more clumsy way to go about it. Any thoughts on that would also be appreciated ?<o:p></o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><o:p> </o:p></div></div>_______________________________________________<br>Mailing list:<span class="Apple-converted-space"> </span><a href="https://launchpad.net/~openstack" style="color: blue; text-decoration: underline; ">https://launchpad.net/~openstack</a><br>Post to :<span class="Apple-converted-space"> </span><a href="mailto:openstack@lists.launchpad.net" style="color: blue; text-decoration: underline; ">openstack@lists.launchpad.net</a><br>Unsubscribe :<span class="Apple-converted-space"> </span><a href="https://launchpad.net/~openstack" style="color: blue; text-decoration: underline; ">https://launchpad.net/~openstack</a><br>More help :<span class="Apple-converted-space"> </span><a href="https://help.launchpad.net/ListHelp" style="color: blue; text-decoration: underline; ">https://help.launchpad.net/ListHelp</a><br></div></span></blockquote></div><br></div></div></body></html>