<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"><meta name="Generator" content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang="EN-US" link="blue" vlink="purple"><div class="WordSection1"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Alex et al,</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I’ve succeeded to reproduce this problem by issuing multiple concurrent euca-allocate-address & euca-release-address calls.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Here is one of traces from nova-api.log:</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">2011-09-23 22:07:24,355 ERROR nova.api [3a5ae61d-4e6a-4701-9180-ef6c49a76c61 diego nubeblog] Unexpected error raised: ResourceClosedError This result object does not return rows. It has been closed automatically.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">[u'Traceback (most recent call last):\n', u' File "/mnt/share/s-cloud-Vlad/nova/rpc/impl_kombu.py", line 620, in _process_data\n rval = node_func(context=ctxt, **node_args)\n', u' File "/mnt/share/s-cloud-Vlad/nova/network/manager.py", line 279, in allocate_floating_ip\n project_id)\n', u' File "/mnt/share/s-cloud-Vlad/nova/db/api.py", line 232, in floating_ip_allocate_address\n return IMPL.floating_ip_allocate_address(context, project_id)\n', u' File "/mnt/share/s-cloud-Vlad/nova/db/sqlalchemy/api.py", line 119, in wrapper\n return f(*args, **kwargs)\n', u' File "/mnt/share/s-cloud-Vlad/nova/db/sqlalchemy/api.py", line 500, in floating_ip_allocate_address\n with_lockmode(\'update\').\\\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1535, in first\n ret = list(self[0:1])\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1444, in __getitem__\n return list(res)\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1708, in instances\n fetch = cursor.fetchall()\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2493, in fetchall\n l = self.process_rows(self._fetchall_impl())\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2462, in _fetchall_impl\n self._non_result()\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2467, in _non_result\n "This result object does not return rows. "\n', u'ResourceClosedError: This result object does not return rows. It has been closed automatically.\n']</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: Traceback (most recent call last):</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: File "/mnt/share/s-cloud-Vlad/nova/api/ec2/__init__.py", line 398, in __call__</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: result = api_request.invoke(context)</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: File "/mnt/share/s-cloud-Vlad/nova/api/ec2/apirequest.py", line 78, in invoke</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: result = method(context, **args)</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: File "/mnt/share/s-cloud-Vlad/nova/api/ec2/cloud.py", line 1302, in allocate_address</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: public_ip = self.network_api.allocate_floating_ip(context)</span></p><p class="MsoNormal">
<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: File "/mnt/share/s-cloud-Vlad/nova/network/api.py", line 61, in allocate_floating_ip</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: 'args': {'project_id': context.project_id}})</span></p><p class="MsoNormal">
<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: File "/mnt/share/s-cloud-Vlad/nova/rpc/__init__.py", line 45, in call</span></p><p class="MsoNormal">
<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: return get_impl().call(context, topic, msg)</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: File "/mnt/share/s-cloud-Vlad/nova/rpc/impl_kombu.py", line 739, in call</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: rv = list(rv)</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: File "/mnt/share/s-cloud-Vlad/nova/rpc/impl_kombu.py", line 703, in __iter__</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: raise result</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(nova.api): TRACE: RemoteError: ResourceClosedError This result object does not return rows. It has been closed automatically.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The problem seems to be related to .with_lockmode('update') functionality of sqlalchemy. It seems to raise ResourceClosedError exception if several threads are trying to perform the same operation on the DB instead of waiting on it.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Regards,</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">-Vladimir</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p><div><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal">
<b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Alex Lyakas [mailto:<a href="mailto:alex@zadarastorage.com">alex@zadarastorage.com</a>] <br>
<b>Sent:</b> Wednesday, September 21, 2011 5:27 AM<br><b>To:</b> <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br><b>Cc:</b> Yair Hershko; Vladimir Popovsky<br><b>Subject:</b> volume_allocate_iscsi_target in db/sqlalchemy/api.py hits ResourceClosedError</span></p>
</div></div><p class="MsoNormal"> </p><div><div><div><div><p class="MsoNormal"><span style="font-family:"Calibri","sans-serif";color:black">Greetings everybody,</span></p></div></div><div><div><p class="MsoNormal">
<span style="font-family:"Calibri","sans-serif";color:black">in one of our tests we occasionally hit </span></p></div><div><p class="MsoNormal"><span style="font-family:"Calibri","sans-serif";color:black">ResourceClosedError when executing the volume_allocate_iscsi_target:</span></p>
</div></div><div><div><p class="MsoNormal"><span style="font-family:"Calibri","sans-serif";color:black"> </span></p></div></div><div><div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Courier New";color:black">2011-09-20 04:25:20,515 DEBUG nova.volume.driver [-] volume_allocate_iscsi_target failed for volume volume-0000114c from (pid=23701) create_export /mnt/share/s-cloud-Vlad/nova/volume/driver.py:959</span><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"></span></p>
</div><div><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"> </span></p></div><div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Courier New";color:black">2011-09-20 04:25:20,515 WARNING nova.volume.manager [-] Volume creation failed with traceback: <traceback object at 0x3c55878></span><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"></span></p>
</div><div><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"> </span></p></div><div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Courier New";color:black">2011-09-20 04:25:20,643 ERROR nova.rpc [-] Exception during message handling</span><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"></span></p>
</div><div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Courier New";color:black">(nova.rpc): TRACE: Traceback (most recent call last):</span><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"></span></p>
</div><div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Courier New";color:black">(nova.rpc): TRACE: File "/mnt/share/s-cloud-Vlad/nova/rpc/impl_kombu.py", line 620, in _process_data</span><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"></span></p>
</div><div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Courier New";color:black">(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)</span><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"></span></p>
</div><div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Courier New";color:black">(nova.rpc): TRACE: File "/mnt/share/s-cloud-Vlad/nova/volume/manager.py", line 140, in create_volume</span><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"></span></p>
</div><div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Courier New";color:black">(nova.rpc): TRACE: raise exc_info</span><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"></span></p>
</div><div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Courier New";color:black">(nova.rpc): TRACE: ResourceClosedError</span><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"></span></p>
</div><div><p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Courier New";color:black">(nova.rpc): TRACE: </span><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"></span></p>
</div><div><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"> </span></p></div><div><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black">In one of the tests it happened at the same time on three different compute nodes. </span></p>
</div><div><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black">Can anybody pls advise how to investigate the root cause of this problem? The MySQL server logs do not show anything useful at the time of the problem.</span></p>
</div><div><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"> </span></p></div><div><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black">Thanks,</span></p>
</div><div><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"> Alex.</span></p></div><div><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"> </span></p>
</div><div><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"> </span></p></div><div><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"> </span></p>
</div></div></div></div></div></body></html>