<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Thanks I added it to the commit.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Could you share your uwsgi config as well.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best Regards, Erik Olof Gunnar Andersson<o:p></o:p></p>
<p class="MsoNormal">Technical Lead, Senior Cloud Engineer<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Ionut Biru <ionut@fleio.com> <br>
<b>Sent:</b> Tuesday, January 5, 2021 1:51 AM<br>
<b>To:</b> Erik Olof Gunnar Andersson <eandersson@blizzard.com><br>
<b>Cc:</b> Spyros Trigazis <strigazi@gmail.com>; feilong <feilong@catalyst.net.nz>; openstack-discuss <openstack-discuss@lists.openstack.org><br>
<b>Subject:</b> Re: [magnum][api] Error system library fopen too many open files with magnum-auto-healer<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Here is my config. maybe something is fishy.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I did have around 300 messages in the queue in <a href="https://urldefense.com/v3/__http:/notification.info__;!!Ci6f514n9QsL8ck!zXau4TQ7lpYxxCmShvD-QtwfISyXyajq11TeBMle6hAdw3N9NdP7PuG5YgqgOhdO4A$">
notification.info</a> and notification.err and I purged them.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://urldefense.com/v3/__https:/paste.xinu.at/woMt/__;!!Ci6f514n9QsL8ck!zXau4TQ7lpYxxCmShvD-QtwfISyXyajq11TeBMle6hAdw3N9NdP7PuG5YgrG1_F7_w$">https://paste.xinu.at/woMt/</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Jan 5, 2021 at 11:23 AM Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com">eandersson@blizzard.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Yea - tested locally as well and wasn't able to reproduce it either. I changed the health service job to run every second and maxed out at about 42 connections to RabbitMQ with two conductor workers.<br>
<br>
/etc/magnum/magnun.conf<br>
<br>
[conductor] <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">workers = 2<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="gmail-m_8552743414295496124divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Spyros Trigazis <<a href="mailto:strigazi@gmail.com" target="_blank">strigazi@gmail.com</a>><br>
<b>Sent:</b> Tuesday, January 5, 2021 12:59 AM<br>
<b>To:</b> Ionut Biru <<a href="mailto:ionut@fleio.com" target="_blank">ionut@fleio.com</a>><br>
<b>Cc:</b> Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com" target="_blank">eandersson@blizzard.com</a>>; feilong <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>>; openstack-discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [magnum][api] Error system library fopen too many open files with magnum-auto-healer</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Jan 5, 2021 at 9:36 AM Ionut Biru <<a href="mailto:ionut@fleio.com" target="_blank">ionut@fleio.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Hi, <o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
I tried with process=1 and it reached 1016 connections to rabbitmq.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">lsof<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://urldefense.com/v3/__https:/paste.xinu.at/jGg/__;!!Ci6f514n9QsL8ck!w-sy8zu-TkPMcmlD3ZhyxEiBTRWikibrBZOfumXkqKodtdcI4FD236uNMmjynMvIcA$" target="_blank">https://paste.xinu.at/jGg/</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">i think it goes into error when it reaches 1024 file descriptors.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'm out of ideas of how to resolve this. I only have 3 clusters available and it's kinda weird and It doesn't scale.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">No issues here with 100s of clusters. Not sure what doesn't scale.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">* Maybe your rabbit is flooded with notifications that are not consumed? <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">* You can use way more than 1024 file descriptors, maybe 2^10?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Spyros<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Mon, Jan 4, 2021 at 9:53 PM Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com" target="_blank">eandersson@blizzard.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p>Sure looks like RabbitMQ. How many workers do have you configured?<o:p></o:p></p>
<p> <o:p></o:p></p>
<p>Could you try to changing the uwsgi configuration to <span style="color:#24292E;background:white">
workers=1 (or processes=1) </span>and then see if it goes beyond 30 connections to amqp.<o:p></o:p></p>
<p> <o:p></o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p><b>From:</b> Ionut Biru <<a href="mailto:ionut@fleio.com" target="_blank">ionut@fleio.com</a>>
<br>
<b>Sent:</b> Monday, January 4, 2021 4:07 AM<br>
<b>To:</b> Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com" target="_blank">eandersson@blizzard.com</a>><br>
<b>Cc:</b> feilong <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>>; openstack-discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [magnum][api] Error system library fopen too many open files with magnum-auto-healer<o:p></o:p></p>
</div>
<p> <o:p></o:p></p>
<div>
<p>Hi Erik,<o:p></o:p></p>
<div>
<p> <o:p></o:p></p>
</div>
<div>
<p>Here is lsof of one uwsgi api. <a href="https://urldefense.com/v3/__https:/paste.xinu.at/5YUWf/__;!!Ci6f514n9QsL8ck!wv_wzG-Ntk0gd3ReOupQl-iXIcWpPR3genCqeKNY5JCKZDWxQHSqqa-uxxgUFFhz0Q$" target="_blank">https://paste.xinu.at/5YUWf/</a><o:p></o:p></p>
</div>
<div>
<p> <o:p></o:p></p>
</div>
<div>
<p>I have kubernetes 12.0.1 installed in env.<o:p></o:p></p>
</div>
<div>
<p> <o:p></o:p></p>
</div>
</div>
<p> <o:p></o:p></p>
<div>
<div>
<p>On Sun, Jan 3, 2021 at 3:06 AM Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com" target="_blank">eandersson@blizzard.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p><span style="font-size:12.0pt;color:black">Maybe something similar to this?<br>
<a href="https://urldefense.com/v3/__https:/github.com/kubernetes-client/python/issues/1158__;!!Ci6f514n9QsL8ck!wv_wzG-Ntk0gd3ReOupQl-iXIcWpPR3genCqeKNY5JCKZDWxQHSqqa-uxxgAtzJkNg$" target="_blank">https://github.com/kubernetes-client/python/issues/1158</a><br>
<br>
What does lsof say?</span><o:p></o:p></p>
<p><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<div>
<div>
<p> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Ionut Biru - <a href="https://urldefense.com/v3/__https:/fleio.com__;!!Ci6f514n9QsL8ck!w-sy8zu-TkPMcmlD3ZhyxEiBTRWikibrBZOfumXkqKodtdcI4FD236uNMmit-G0eng$" target="_blank">
https://fleio.com</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Ionut Biru - <a href="https://urldefense.com/v3/__https:/fleio.com__;!!Ci6f514n9QsL8ck!zXau4TQ7lpYxxCmShvD-QtwfISyXyajq11TeBMle6hAdw3N9NdP7PuG5Ygp-5WUmyw$" target="_blank">
https://fleio.com</a><o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>