<div dir="ltr">Thanks! I was able to reproduce and I've proposed a fix to both master and grizzly.<br><div><br></div><div>master- <a href="https://review.openstack.org/#/c/25831/">https://review.openstack.org/#/c/25831/</a><br>
</div><div>grizzly- <a href="https://review.openstack.org/#/c/25833/">https://review.openstack.org/#/c/25833/</a><br></div><div><br></div><div style>If you can verify this patch on your system, that would be really helpful.</div>
</div><div class="gmail_extra"><br clear="all"><div><div><br></div>-Dolph</div>
<br><br><div class="gmail_quote">On Mon, Apr 1, 2013 at 1:57 PM, Miller, Mark M (EB SW Cloud - R&D - Corvallis) <span dir="ltr"><<a href="mailto:mark.m.miller@hp.com" target="_blank">mark.m.miller@hp.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Dolph,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">[token]<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">driver = keystone.token.backends.sql.Token<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"># Amount of time a token should remain valid (in seconds)<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">expiration = 999986400<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I get the same results for either UUID or PKI tokens.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Mark<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Dolph Mathews [mailto:<a href="mailto:dolph.mathews@gmail.com" target="_blank">dolph.mathews@gmail.com</a>]
<br>
<b>Sent:</b> Monday, April 01, 2013 11:56 AM</span></p><div class="im"><br>
<b>To:</b> Miller, Mark M (EB SW Cloud - R&D - Corvallis)<br>
<b>Cc:</b> <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
<b>Subject:</b> Re: [Openstack] Keystone Too Many Connections Problem<u></u><u></u></div><p></p><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
<p class="MsoNormal"><u></u> <u></u></p>
</font></span><div><span class="HOEnZb"><font color="#888888">
<p class="MsoNormal">Mark,<u></u><u></u></p></font></span><div><div class="h5">
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Which token driver are you using?<u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">-Dolph<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Mon, Apr 1, 2013 at 10:49 AM, Dolph Mathews <<a href="mailto:dolph.mathews@gmail.com" target="_blank">dolph.mathews@gmail.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">I also opened bug 1162857, which produces a slightly different symptom with a different configuration (sqlite), but may share the same underlying cause.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888"><br clear="all">
<span><u></u><u></u></span></span></p>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">-Dolph<u></u><u></u></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Mon, Apr 1, 2013 at 10:26 AM, Dolph Mathews <<a href="mailto:dolph.mathews@gmail.com" target="_blank">dolph.mathews@gmail.com</a>> wrote:<u></u><u></u></p>
<div>
<p class="MsoNormal">I'm attempting to reproduce this now (haven't been successful with PKI + sqlite on disk), but in the meantime I opened a bug on this issue for tracking:<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> <a href="https://bugs.launchpad.net/keystone/+bug/1162845" target="_blank">
https://bugs.launchpad.net/keystone/+bug/1162845</a><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-Dolph<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">On Fri, Mar 29, 2013 at 1:35 PM, Miller, Mark M (EB SW Cloud - R&D - Corvallis) <<a href="mailto:mark.m.miller@hp.com" target="_blank">mark.m.miller@hp.com</a>> wrote:<u></u><u></u></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal">Hi,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I have a loop that gets PKI tokens from Keystone (RC2). After about 116 loops I get the following error and have to reset Keystone to continue. Has anyone else run into this?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Mark<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">(keystone.auth.controllers): 2013-03-29 11:14:01,274 ERROR (OperationalError) (1040, 'Too many connections') None None</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">Traceback (most recent call last):</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/controllers.py", line 286, in authenticate_for_token</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> self.authenticate(context, auth_info, auth_context)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/controllers.py", line 352, in authenticate</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> auth_context)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py", line 108, in authenticate</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> user_info = UserAuthInfo(context, auth_payload)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py", line 36, in __init__</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> self._validate_and_normalize_auth_data(auth_payload)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py", line 87, in _validate_and_normalize_auth_data</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> domain_ref = self._lookup_domain(user_info['domain'])</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py", line 60, in _lookup_domain</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> context=self.context, domain_name=domain_name)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/common/manager.py", line 47, in _wrapper</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> return f(*args, **kw)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/common/sql/core.py", line 272, in wrapper</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> return method(*args, **kwargs)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/identity/backends/sql.py", line 589, in get_domain_by_name</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> ref = session.query(Domain).filter_by(name=domain_name).one()</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2184, in one</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> ret = list(self)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2227, in __iter__</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> return self._execute_and_instances(context)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2240, in _execute_and_instances</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> close_with_result=True)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2231, in _connection_from_session</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> **kw)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 777, in connection</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> close_with_result=close_with_result)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 783, in _connection_for_bind</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> return engine.contextual_connect(**kwargs)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2489, in contextual_connect</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> self.pool.connect(),</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 236, in connect</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> return _ConnectionFairy(self).checkout()</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 401, in __init__</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> rec = self._connection_record = pool._do_get()</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 746, in _do_get</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> con = self._create_connection()</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 189, in _create_connection</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> return _ConnectionRecord(self)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 282, in __init__</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> self.connection = self.__connect()</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 344, in __connect</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> connection = self.__pool._creator()</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/strategies.py", line 80, in connect</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> return dialect.connect(*cargs, **cparams)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py", line 281, in connect</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> return self.dbapi.connect(*cargs, **cparams)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> return Connection(*args, **kwargs)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> super(Connection, self).__init__(*args, **kwargs2)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">OperationalError: (OperationalError) (1040, 'Too many connections') None None</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">(access): 2013-03-29 11:14:01,278 INFO 15.253.58.148 - - [29/Mar/2013:18:14:01 +0000] "POST
<a href="http://15.253.58.165:35357/v3/auth/tokens" target="_blank">http://15.253.58.165:35357/v3/auth/tokens</a> HTTP/1.0" 401 125</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">(eventlet.wsgi.server): 2013-03-29 11:14:01,279 DEBUG 15.253.58.148 - - [29/Mar/2013 11:14:01] "POST /v3/auth/tokens HTTP/1.1" 401 290 0.017322</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
</div>
</blockquote></div><br></div>