<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">We were using the eventlet db pool class.<div><br></div><div>You can see the code here:</div><div><br></div><div><a href="https://github.com/openstack/nova/blob/2011.3/nova/db/sqlalchemy/session.py">https://github.com/openstack/nova/blob/2011.3/nova/db/sqlalchemy/session.py</a></div><div><br></div><div>there were a few bugs about it, but here is one:</div><div><br></div><div><a href="https://bugs.launchpad.net/nova/+bug/838581">https://bugs.launchpad.net/nova/+bug/838581</a></div><div><br></div><div>Vish</div><div><br><div><div>On Mar 1, 2012, at 6:11 PM, Kapil Thangavelu wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div><br>I actually didn't mean to send the parent.. but since we're here. <br><br>Additional threads/pool make the problem potentially worse, they are the origin <br>of the greenlet switching issue that was the start of this thread. ie. with the <br>monkey patching, most stdlib socket, thread module usage by python code in the <br>non-main threads will potentially attempt a greenlet trampoline across threads <br>at worst (ie. error), or unintentional per thread hubs at best. <br><br>Since the mysqldb is an extension (no python code) wrapping it in eventlet <br>thread pool should work in theory. What where the problems with it last time it <br>was attempted?<br><br>cheers,<br><br>-kapil<br><br><br>Excerpts from Devin Carlen's message of 2012-03-01 20:38:20 -0500:<br><blockquote type="cite">As long as we allocate a thread in the eventlet thread pool for the number of mysql connections we want to actually maintain in our connection pool, we shouldn't have problems getting the results we want even with the blocking mysql c drivers. <br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Devin<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On Thursday, March 1, 2012 at 5:23 PM, Kapil Thangavelu wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">The standard python postgresql driver (psycopg2) does have an async mode. There <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">are non db api compliant async mysql drivers for gevent.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Excerpts from Vishvananda Ishaya's message of 2012-03-01 15:36:43 -0500:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Yes it does. We actually tried to use a pool at diablo release and it was very broken. There was discussion about moving over to a pure-python mysql library, but it hasn't been tried yet.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Vish<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">On Mar 1, 2012, at 11:45 AM, Yun Mao wrote:<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">There are plenty eventlet discussion recently but I'll stick my<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">question to this thread, although it's pretty much a separate<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">question. :)<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">How is MySQL access handled in eventlet? Presumably it's external C<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">library so it's not going to be monkey patched. Does that make every<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">db access call a blocking call? Thanks,<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Yun<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">On Wed, Feb 29, 2012 at 9:18 PM, Johannes Erdfelt <<a href="mailto:johannes@erdfelt.com">johannes@erdfelt.com</a> (<a href="mailto:johannes@erdfelt.com">mailto:johannes@erdfelt.com</a>)> wrote:<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">On Wed, Feb 29, 2012, Yun Mao <<a href="mailto:yunmao@gmail.com">yunmao@gmail.com</a> (<a href="mailto:yunmao@gmail.com">mailto:yunmao@gmail.com</a>)> wrote:<br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Thanks for the explanation. Let me see if I understand this.<br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">1. Eventlet will never have this problem if there is only 1 OS thread<br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">-- let's call it main thread.<br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">In fact, that's exactly what Python calls it :)<br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">2. In Nova, there is only 1 OS thread unless you use xenapi and/or the<br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">virt/firewall driver.<br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">3. The python logging module uses locks. Because of the monkey patch,<br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">those locks are actually eventlet or "green" locks and may trigger a<br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">green thread context switch.<br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Based on 1-3, does it make sense to say that in the other OS threads<br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">(i.e. not main thread), if logging (plus other pure python library<br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">code involving locking) is never used, and we do not run a eventlet<br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">hub at all, we should never see this problem?<br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">That should be correct. I'd have to double check all of the monkey<br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">patching that eventlet does to make sure there aren't other cases where<br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">you may inadvertently use eventlet primitives across real threads.<br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">JE<br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Mailing list: <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a> (<a href="mailto:openstack@lists.launchpad.net">mailto:openstack@lists.launchpad.net</a>)<br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Unsubscribe : <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">More help : <a href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a><br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Mailing list: <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a> (<a href="mailto:openstack@lists.launchpad.net">mailto:openstack@lists.launchpad.net</a>)<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Unsubscribe : <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">More help : <a href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Mailing list: <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a> (<a href="mailto:openstack@lists.launchpad.net">mailto:openstack@lists.launchpad.net</a>)<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Unsubscribe : <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">More help : <a href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></div></blockquote></div><br></div></body></html>