<div dir="ltr">Hi Erik, <div><br></div><div>Thanks a lot for the patch. Indeed 769471 fixes my problem at first glance.</div><div><br>I'll let it run for a couple of days. </div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 6, 2021 at 12:23 PM Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com">eandersson@blizzard.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I pushed a couple of patches that you can try out.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
This is the most likely culprit.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<a href="https://review.opendev.org/c/openstack/magnum/+/769471" id="gmail-m_-5182837653570759985LPlnk664958" target="_blank">https://review.opendev.org/c/openstack/magnum/+/769471</a> - <span style="color:rgb(33,33,33);font-family:"Open Sans",Roboto,-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:16.002px;background-color:rgb(255,255,255);display:inline">Re-use
rpc client</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(33,33,33);font-family:"Open Sans",Roboto,-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:16.002px;background-color:rgb(255,255,255);display:inline"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(33,33,33);font-family:"Open Sans",Roboto,-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:16.002px;background-color:rgb(255,255,255);display:inline">I
also created this one, but doubt this is an issue as the implementation here is the same as I use in Designate</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(33,33,33);font-family:"Open Sans",Roboto,-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:16.002px;background-color:rgb(255,255,255);display:inline"><a href="https://review.opendev.org/c/openstack/magnum/+/769457" id="gmail-m_-5182837653570759985LPlnk698045" target="_blank">https://review.opendev.org/c/openstack/magnum/+/769457</a> - <span style="background-color:rgb(255,255,255);display:inline">[WIP]
Singleton notifier</span><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(33,33,33);font-family:"Open Sans",Roboto,-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:16.002px;background-color:rgb(255,255,255);display:inline"><span style="background-color:rgb(255,255,255);display:inline"><br>
</span></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(33,33,33);font-family:"Open Sans",Roboto,-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:16.002px;background-color:rgb(255,255,255);display:inline"><span style="background-color:rgb(255,255,255);display:inline">Finally
I also created a PR to add magnum-api testing using uwsgi.</span></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(33,33,33);font-family:"Open Sans",Roboto,-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:16.002px;background-color:rgb(255,255,255);display:inline"><span style="background-color:rgb(255,255,255);display:inline"><a href="https://review.opendev.org/c/openstack/magnum/+/769450" id="gmail-m_-5182837653570759985LPlnk882175" target="_blank">https://review.opendev.org/c/openstack/magnum/+/769450</a><br>
<br>
Let me know if any of these patches help!</span></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(33,33,33);font-family:"Open Sans",Roboto,-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:16.002px;background-color:rgb(255,255,255);display:inline"><br>
</span></div>
<div id="gmail-m_-5182837653570759985appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-5182837653570759985divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Ionut Biru <<a href="mailto:ionut@fleio.com" target="_blank">ionut@fleio.com</a>><br>
<b>Sent:</b> Tuesday, January 5, 2021 8:36 AM<br>
<b>To:</b> Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com" target="_blank">eandersson@blizzard.com</a>><br>
<b>Cc:</b> Spyros Trigazis <<a href="mailto:strigazi@gmail.com" target="_blank">strigazi@gmail.com</a>>; feilong <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>>; openstack-discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [magnum][api] Error system library fopen too many open files with magnum-auto-healer</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi,
<div><br>
</div>
<div>I found this story: <a href="https://urldefense.com/v3/__https://storyboard.openstack.org/*!/story/2008308__;Iw!!Ci6f514n9QsL8ck!1f5rl4Hhpd13WKbYo8oADBrjfiG2BvU4omHN8zT_EtCcWSC4JoI9JJkg_A3rIZPB1g$" target="_blank">https://storyboard.openstack.org/#!/story/2008308</a>
regarding disabling cluster update notifications in rabbitmq.</div>
<div><br>
</div>
<div>I think this will help me.</div>
</div>
<br>
<div>
<div dir="ltr">On Tue, Jan 5, 2021 at 12:21 PM Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com" target="_blank">eandersson@blizzard.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Sorry, being repetitive here, but maybe try adding this to your magnum config as well. If you have A LOT of cores it could add up to a crazy amount of connections.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255);display:inline">[conductor]</span>
<div style="margin:0px;background-color:rgb(255,255,255)">workers = 2</div>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_-5182837653570759985x_gmail-m_1589104128920787310appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-5182837653570759985x_gmail-m_1589104128920787310divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Ionut Biru <<a href="mailto:ionut@fleio.com" target="_blank">ionut@fleio.com</a>><br>
<b>Sent:</b> Tuesday, January 5, 2021 1:50 AM<br>
<b>To:</b> Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com" target="_blank">eandersson@blizzard.com</a>><br>
<b>Cc:</b> Spyros Trigazis <<a href="mailto:strigazi@gmail.com" target="_blank">strigazi@gmail.com</a>>; feilong <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>>; openstack-discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [magnum][api] Error system library fopen too many open files with magnum-auto-healer</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi,
<div><br>
</div>
<div>Here is my config. maybe something is fishy.</div>
<div><br>
</div>
<div>I did have around 300 messages in the queue in <a href="https://urldefense.com/v3/__http://notification.info__;!!Ci6f514n9QsL8ck!zXau4TQ7lpYxxCmShvD-QtwfISyXyajq11TeBMle6hAdw3N9NdP7PuG5YgqgOhdO4A$" target="_blank">
notification.info</a> and notification.err and I purged them.</div>
<div><br>
</div>
<div><a href="https://urldefense.com/v3/__https://paste.xinu.at/woMt/__;!!Ci6f514n9QsL8ck!zXau4TQ7lpYxxCmShvD-QtwfISyXyajq11TeBMle6hAdw3N9NdP7PuG5YgrG1_F7_w$" target="_blank">https://paste.xinu.at/woMt/</a><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Tue, Jan 5, 2021 at 11:23 AM Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com" target="_blank">eandersson@blizzard.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Yea - tested locally as well and wasn't able to reproduce it either. I changed the health service job to run every second and maxed out at about 42 connections to RabbitMQ with two conductor workers.<br>
<br>
/etc/magnum/magnun.conf<br>
<br>
[conductor]
<div>workers = 2</div>
<br>
<br>
</div>
<div id="gmail-m_-5182837653570759985x_gmail-m_1589104128920787310x_gmail-m_8552743414295496124appendonsend">
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-5182837653570759985x_gmail-m_1589104128920787310x_gmail-m_8552743414295496124divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Spyros Trigazis <<a href="mailto:strigazi@gmail.com" target="_blank">strigazi@gmail.com</a>><br>
<b>Sent:</b> Tuesday, January 5, 2021 12:59 AM<br>
<b>To:</b> Ionut Biru <<a href="mailto:ionut@fleio.com" target="_blank">ionut@fleio.com</a>><br>
<b>Cc:</b> Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com" target="_blank">eandersson@blizzard.com</a>>; feilong <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>>; openstack-discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [magnum][api] Error system library fopen too many open files with magnum-auto-healer</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div>
<div dir="ltr">On Tue, Jan 5, 2021 at 9:36 AM Ionut Biru <<a href="mailto:ionut@fleio.com" target="_blank">ionut@fleio.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
I tried with process=1 and it reached 1016 connections to rabbitmq.</div>
<div>lsof</div>
<div><a href="https://urldefense.com/v3/__https://paste.xinu.at/jGg/__;!!Ci6f514n9QsL8ck!w-sy8zu-TkPMcmlD3ZhyxEiBTRWikibrBZOfumXkqKodtdcI4FD236uNMmjynMvIcA$" target="_blank">https://paste.xinu.at/jGg/</a><br>
</div>
<div><br>
</div>
<div>i think it goes into error when it reaches 1024 file descriptors.</div>
<div><br>
</div>
<div>I'm out of ideas of how to resolve this. I only have 3 clusters available and it's kinda weird and It doesn't scale.</div>
</div>
</blockquote>
<div><br>
</div>
<div>No issues here with 100s of clusters. Not sure what doesn't scale.</div>
<div><br>
</div>
<div>* Maybe your rabbit is flooded with notifications that are not consumed? </div>
<div>* You can use way more than 1024 file descriptors, maybe 2^10?</div>
<div><br>
</div>
<div>Spyros</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<div>
<div dir="ltr">On Mon, Jan 4, 2021 at 9:53 PM Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com" target="_blank">eandersson@blizzard.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div>
<p>Sure looks like RabbitMQ. How many workers do have you configured?<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>Could you try to changing the uwsgi configuration to <span style="color:rgb(36,41,46);background:white">
workers=1 (or processes=1) </span>and then see if it goes beyond 30 connections to amqp.<u></u><u></u></p>
<p><u></u> <u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p><b>From:</b> Ionut Biru <<a href="mailto:ionut@fleio.com" target="_blank">ionut@fleio.com</a>>
<br>
<b>Sent:</b> Monday, January 4, 2021 4:07 AM<br>
<b>To:</b> Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com" target="_blank">eandersson@blizzard.com</a>><br>
<b>Cc:</b> feilong <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>>; openstack-discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [magnum][api] Error system library fopen too many open files with magnum-auto-healer<u></u><u></u></p>
</div>
<p><u></u> <u></u></p>
<div>
<p>Hi Erik,<u></u><u></u></p>
<div>
<p><u></u> <u></u></p>
</div>
<div>
<p>Here is lsof of one uwsgi api. <a href="https://urldefense.com/v3/__https:/paste.xinu.at/5YUWf/__;!!Ci6f514n9QsL8ck!wv_wzG-Ntk0gd3ReOupQl-iXIcWpPR3genCqeKNY5JCKZDWxQHSqqa-uxxgUFFhz0Q$" target="_blank">https://paste.xinu.at/5YUWf/</a><u></u><u></u></p>
</div>
<div>
<p><u></u> <u></u></p>
</div>
<div>
<p>I have kubernetes 12.0.1 installed in env.<u></u><u></u></p>
</div>
<div>
<p><u></u> <u></u></p>
</div>
</div>
<p><u></u> <u></u></p>
<div>
<div>
<p>On Sun, Jan 3, 2021 at 3:06 AM Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com" target="_blank">eandersson@blizzard.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p><span style="font-size:12pt;color:black">Maybe something similar to this?<br>
<a href="https://urldefense.com/v3/__https:/github.com/kubernetes-client/python/issues/1158__;!!Ci6f514n9QsL8ck!wv_wzG-Ntk0gd3ReOupQl-iXIcWpPR3genCqeKNY5JCKZDWxQHSqqa-uxxgAtzJkNg$" target="_blank">https://github.com/kubernetes-client/python/issues/1158</a><br>
<br>
What does lsof say?<u></u><u></u></span></p>
<p><span style="font-size:12pt;color:black"><u></u> <u></u></span></p>
</div>
</div>
</blockquote>
</div>
<div>
<div>
<p><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">Ionut Biru - <a href="https://urldefense.com/v3/__https://fleio.com__;!!Ci6f514n9QsL8ck!w-sy8zu-TkPMcmlD3ZhyxEiBTRWikibrBZOfumXkqKodtdcI4FD236uNMmit-G0eng$" target="_blank">
https://fleio.com</a><br>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">Ionut Biru - <a href="https://urldefense.com/v3/__https://fleio.com__;!!Ci6f514n9QsL8ck!zXau4TQ7lpYxxCmShvD-QtwfISyXyajq11TeBMle6hAdw3N9NdP7PuG5Ygp-5WUmyw$" target="_blank">
https://fleio.com</a><br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">Ionut Biru - <a href="https://urldefense.com/v3/__https://fleio.com__;!!Ci6f514n9QsL8ck!1f5rl4Hhpd13WKbYo8oADBrjfiG2BvU4omHN8zT_EtCcWSC4JoI9JJkg_A1pHO7VEQ$" target="_blank">
https://fleio.com</a><br>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Ionut Biru - <a href="https://fleio.com" target="_blank">https://fleio.com</a><br></div></div>