<HTML>
<HEAD>
<TITLE>Re: [Openstack] Memory leaks from greenthreads</TITLE>
</HEAD>
<BODY>
<FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>So there is the question here of thread safe (which I don’t think python is going toward?) and then making it multiprocess safe (<a href="http://docs.python.org/library/multiprocessing.html">http://docs.python.org/library/multiprocessing.html</a>).<BR>
<BR>
I’m not sure if one is easier than the other (but the multiprocess model forces queues, object exchange via queues as a passing model, which might be good/great from a forced design pattern point of view). Hacks like sleep(0) seem pretty “hacky” to me. <BR>
<BR>
But this might be a good long-term thing to talk about (its a big change).<BR>
<BR>
On 3/1/12 2:51 AM, "Day, Phil" <<a href="philip.day@hp.com">philip.day@hp.com</a>> wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>> Has there been any thinking around only using eventlet/greenlet for webserver endpoints and using something like multiprocessing for everything else?<BR>
<BR>
<BR>
I was already beginning to think that this would be a good blueprint/discussion topic for the design summit ;-)<BR>
<BR>
We’ve seen a number of issues with the eventlet approach in the computer & network manager where a long running activity (such as updating all security groups, creating and uploading a snapshot) will block any other activities. Whilst it’s possible to work round the first of these types of issues by planting sleep(0) statements in the loop, snapshot upload is still a problem.<BR>
<BR>
Of course making everything thread safe isn’t going to be trivial, although there is lock code in place for things like iptables I suspect that there are a whole bunch of other timing / concurrency issues that we’ll find once we move to a full threaded model. <BR>
<BR>
Phil<BR>
<BR>
<BR>
<BR>
<BR>
</SPAN><FONT SIZE="2"><SPAN STYLE='font-size:10pt'><B>From:</B> <a href="openstack-bounces+philip.day=hp.com@lists.launchpad.net">openstack-bounces+philip.day=hp.com@lists.launchpad.net</a> [<a href="mailto:openstack-bounces+philip.day=hp.com@lists.launchpad.net">mailto:openstack-bounces+philip.day=hp.com@lists.launchpad.net</a>] <B>On Behalf Of </B>Joshua Harlow<BR>
<B>Sent:</B> 29 February 2012 21:26<BR>
<B>To:</B> Vishvananda Ishaya; openstack<BR>
<B>Subject:</B> Re: [Openstack] Memory leaks from greenthreads<BR>
</SPAN></FONT></FONT><FONT FACE="Times New Roman"><SPAN STYLE='font-size:12pt'> <BR>
</SPAN></FONT><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Cool.<BR>
<BR>
Just a thought I was having, that others might want to chime in on.<BR>
<BR>
Has there been any thinking around only using eventlet/greenlet for webserver endpoints and using something like multiprocessing for everything else?<BR>
<BR>
I know its a fundamental change, but it would force people to think about how to break up there code into something that would work with a message passing architecture (this is already happening with nova + rabbitmq). Nova is a good example, but my thought was to go even further and have anything that needs to run for a long time (ie a equivalent of a nova manager) that is shared inside a application also be a separate “process” with a queue for message passing. Then maybe eventlet/greenlet isn’t needed at all? This would force good interfaces, and we wouldn’t have to worry about missing a <I>monkey patch</I>. Maybe the python people plan for multiprocess to replace eventlet/greenlet in the end anyway???<BR>
<BR>
Thoughts?<BR>
<BR>
On 2/29/12 12:48 PM, "Vishvananda Ishaya" <<a href="vishvananda@gmail.com">vishvananda@gmail.com</a>> wrote:<BR>
Hello Everyone,<BR>
<BR>
We have had a memory leak due to an interaction with eventlet for a while that Johannes has just made a fix for.<BR>
<BR>
bug:<BR>
<a href="https://bugs.launchpad.net/nova/+bug/903199">https://bugs.launchpad.net/nova/+bug/903199</a><BR>
<BR>
fix:<BR>
<a href="https://bitbucket.org/which_linden/eventlet/pull-request/10/monkey-patch-threadingcurrent_thread-as">https://bitbucket.org/which_linden/eventlet/pull-request/10/monkey-patch-threadingcurrent_thread-as</a><BR>
<BR>
Unfortuantely, I don' t think we have a decent workaround for nova while that patch is upstreamed. I wanted to make sure that all of the distros are aware of it in case they want to carry an eventlet patch to prevent the slow memory leak.<BR>
<BR>
Vish<BR>
<BR>
</SPAN></FONT></BLOCKQUOTE>
</BODY>
</HTML>