<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Microsoft JhengHei";
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"\@Microsoft JhengHei";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Sure looks like RabbitMQ. How many workers do have you configured?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Could you try to change it to <span style="font-size:9.0pt;font-family:Consolas;color:#24292E;background:white">
workers=1 (or processes=1) </span>and then see if it goes beyond 30 connections to amqp.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best Regards, Erik Olof Gunnar Andersson<o:p></o:p></p>
<p class="MsoNormal">Technical Lead, Senior Cloud Engineer<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Ionut Biru <ionut@fleio.com> <br>
<b>Sent:</b> Monday, January 4, 2021 4:07 AM<br>
<b>To:</b> Erik Olof Gunnar Andersson <eandersson@blizzard.com><br>
<b>Cc:</b> feilong <feilong@catalyst.net.nz>; openstack-discuss <openstack-discuss@lists.openstack.org><br>
<b>Subject:</b> Re: [magnum][api] Error system library fopen too many open files with magnum-auto-healer<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi Erik,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Here is lsof of one uwsgi api. <a href="https://urldefense.com/v3/__https:/paste.xinu.at/5YUWf/__;!!Ci6f514n9QsL8ck!wv_wzG-Ntk0gd3ReOupQl-iXIcWpPR3genCqeKNY5JCKZDWxQHSqqa-uxxgUFFhz0Q$">https://paste.xinu.at/5YUWf/</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I have kubernetes 12.0.1 installed in env.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Sun, Jan 3, 2021 at 3:06 AM Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com">eandersson@blizzard.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Maybe something similar to this?<br>
<a href="https://urldefense.com/v3/__https:/github.com/kubernetes-client/python/issues/1158__;!!Ci6f514n9QsL8ck!wv_wzG-Ntk0gd3ReOupQl-iXIcWpPR3genCqeKNY5JCKZDWxQHSqqa-uxxgAtzJkNg$" target="_blank">https://github.com/kubernetes-client/python/issues/1158</a><br>
<br>
What does lsof say?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="gmail-m_2047728040938236488divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com" target="_blank">eandersson@blizzard.com</a>><br>
<b>Sent:</b> Saturday, January 2, 2021 4:54 PM<br>
<b>To:</b> Ionut Biru <<a href="mailto:ionut@fleio.com" target="_blank">ionut@fleio.com</a>>; feilong <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>><br>
<b>Cc:</b> openstack-discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [magnum][api] Error system library fopen too many open files with magnum-auto-healer</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Are you sure you aren't just looking at the connection pool expanding? Each worker has a max number of connections it can use. Maybe look at lowering </span><span style="font-size:9.0pt;font-family:Consolas;color:#032F62;background:white">rpc_conn_pool_size. </span><span style="font-size:12.0pt;color:black;background:white">By
default I believe each worker might create a pool of up to 30 connections.</span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black;background:white"><br>
Looking at the code it could also be have something to do with the k8s client. Since it creates a new instance each time it does an health check. What version of the k8s client do you have installed?</span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="gmail-m_2047728040938236488x_divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Ionut Biru <<a href="mailto:ionut@fleio.com" target="_blank">ionut@fleio.com</a>><br>
<b>Sent:</b> Tuesday, December 29, 2020 2:20 PM<br>
<b>To:</b> feilong <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>><br>
<b>Cc:</b> openstack-discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [magnum][api] Error system library fopen too many open files with magnum-auto-healer</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Hi, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Not sure if my suspicion is true but I think for each update a new notifier is prepared and used without closing the connection but my understanding of oslo is nonexistent. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://urldefense.com/v3/__https:/opendev.org/openstack/magnum/src/branch/master/magnum/conductor/utils.py*L147__;Iw!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnKwDSl9vw$" target="_blank">https://opendev.org/openstack/magnum/src/branch/master/magnum/conductor/utils.py#L147</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://urldefense.com/v3/__https:/opendev.org/openstack/magnum/src/branch/master/magnum/common/rpc.py*L173__;Iw!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnJa8-cGbA$" target="_blank">https://opendev.org/openstack/magnum/src/branch/master/magnum/common/rpc.py#L173</a><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Dec 29, 2020 at 11:52 PM Ionut Biru <<a href="mailto:ionut@fleio.com" target="_blank">ionut@fleio.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Hi Feilong, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I found out that each time the update_health_status periodic task is run, a new connection(for each uwsgi) is made to rabbitmq.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">root@ctrl1cj-magnum-container-7a7a412a:~# netstat -npt | grep 5672 | wc -l<br>
229<br>
root@ctrl1cj-magnum-container-7a7a412a:~# netstat -npt | grep 5672 | wc -l<br>
234<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">root@ctrl1cj-magnum-container-7a7a412a:~# netstat -npt | grep 5672 | wc -l<br>
238<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">root@ctrl1cj-magnum-container-7a7a412a:~# netstat -npt | grep 5672 | wc -l<br>
241<br>
root@ctrl1cj-magnum-container-7a7a412a:~# netstat -npt | grep 5672 | wc -l<br>
244<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Not sure <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Dec 29 21:51:22 ctrl1cj-magnum-container-7a7a412a magnum-conductor[262800]: 2020-12-29 21:51:22.024 262800 DEBUG magnum.service.periodic [req-3b495326-cf80-481e-b3c6-c741f05b7f0e - - - - -]
<br>
Dec 29 21:51:22 ctrl1cj-magnum-container-7a7a412a magnum-conductor[262800]: 2020-12-29 21:51:22.024 262800 DEBUG oslo_service.periodic_task [-] Running periodic task MagnumPeriodicTasks.sync<br>
Dec 29 21:51:16 ctrl1cj-magnum-container-7a7a412a magnum-conductor[262804]: 2020-12-29 21:51:16.462 262804 DEBUG magnum.conductor.handlers.cluster_conductor [req-284ac12b-d76a-4e50-8e74-5bfb<br>
Dec 29 21:51:15 ctrl1cj-magnum-container-7a7a412a magnum-conductor[262800]: 2020-12-29 21:51:15.573 262800 DEBUG magnum.service.periodic [-] Status for cluster 118 updated to HEALTHY ({'api'<br>
Dec 29 21:51:15 ctrl1cj-magnum-container-7a7a412a magnum-conductor[262805]: 2020-12-29 21:51:15.572 262805 DEBUG magnum.conductor.handlers.cluster_conductor [req-3fc29ee9-4051-42e7-ae19-3a49<br>
Dec 29 21:51:15 ctrl1cj-magnum-container-7a7a412a magnum-conductor[262800]: 2020-12-29 21:51:15.572 262800 DEBUG magnum.service.periodic [-] Status for cluster 121 updated to HEALTHY ({'api'<br>
Dec 29 21:51:15 ctrl1cj-magnum-container-7a7a412a magnum-conductor[262800]: 2020-12-29 21:51:15.572 262800 DEBUG magnum.service.periodic [-] Status for cluster 122 updated to HEALTHY ({'api'<br>
Dec 29 21:51:15 ctrl1cj-magnum-container-7a7a412a magnum-conductor[262800]: 2020-12-29 21:51:15.553 262800 DEBUG magnum.service.periodic [-] Updating health status for cluster 122 update_hea<br>
Dec 29 21:51:15 ctrl1cj-magnum-container-7a7a412a magnum-conductor[262800]: 2020-12-29 21:51:15.544 262800 DEBUG magnum.service.periodic [-] Updating health status for cluster 121 update_hea<br>
Dec 29 21:51:15 ctrl1cj-magnum-container-7a7a412a magnum-conductor[262800]: 2020-12-29 21:51:15.535 262800 DEBUG magnum.service.periodic [-] Updating health status for cluster 118 update_hea<br>
Dec 29 21:51:15 ctrl1cj-magnum-container-7a7a412a magnum-conductor[262800]: 2020-12-29 21:51:15.494 262800 DEBUG magnum.service.periodic [req-405b1fed-0b8a-4a60-b6ae-834f548b21d1 - - - <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">2020-12-29 21:51:14.082 [info] <0.953.1293> accepting AMQP connection <0.953.1293> (<a href="https://urldefense.com/v3/__http:/172.29.93.14:48474__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnJ0aOLwIQ$" target="_blank">172.29.93.14:48474</a>
-> <a href="https://urldefense.com/v3/__http:/172.29.95.38:5672__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnKyfYp2-Q$" target="_blank">
172.29.95.38:5672</a>)<br>
2020-12-29 21:51:14.083 [info] <0.953.1293> Connection <0.953.1293> (<a href="https://urldefense.com/v3/__http:/172.29.93.14:48474__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnJ0aOLwIQ$" target="_blank">172.29.93.14:48474</a>
-> <a href="https://urldefense.com/v3/__http:/172.29.95.38:5672__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnKyfYp2-Q$" target="_blank">
172.29.95.38:5672</a>) has a client-provided name: uwsgi:262739:f86c0570-8739-4b74-8102-76b5357acd71<br>
2020-12-29 21:51:14.084 [info] <0.953.1293> connection <0.953.1293> (<a href="https://urldefense.com/v3/__http:/172.29.93.14:48474__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnJ0aOLwIQ$" target="_blank">172.29.93.14:48474</a>
-> <a href="https://urldefense.com/v3/__http:/172.29.95.38:5672__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnKyfYp2-Q$" target="_blank">
172.29.95.38:5672</a> - uwsgi:262739:f86c0570-8739-4b74-8102-76b5357acd71): user 'magnum' authenticated and granted access to vhost '/magnum'<br>
2020-12-29 21:51:15.560 [info] <0.1656.1283> accepting AMQP connection <0.1656.1283> (<a href="https://urldefense.com/v3/__http:/172.29.93.14:48548__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnLsbQ8hVw$" target="_blank">172.29.93.14:48548</a>
-> <a href="https://urldefense.com/v3/__http:/172.29.95.38:5672__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnKyfYp2-Q$" target="_blank">
172.29.95.38:5672</a>)<br>
2020-12-29 21:51:15.561 [info] <0.1656.1283> Connection <0.1656.1283> (<a href="https://urldefense.com/v3/__http:/172.29.93.14:48548__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnLsbQ8hVw$" target="_blank">172.29.93.14:48548</a>
-> <a href="https://urldefense.com/v3/__http:/172.29.95.38:5672__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnKyfYp2-Q$" target="_blank">
172.29.95.38:5672</a>) has a client-provided name: uwsgi:262744:2c9792ab-9198-493a-970c-f6ccfd9947d3<br>
2020-12-29 21:51:15.561 [info] <0.1656.1283> connection <0.1656.1283> (<a href="https://urldefense.com/v3/__http:/172.29.93.14:48548__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnLsbQ8hVw$" target="_blank">172.29.93.14:48548</a>
-> <a href="https://urldefense.com/v3/__http:/172.29.95.38:5672__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnKyfYp2-Q$" target="_blank">
172.29.95.38:5672</a> - uwsgi:262744:2c9792ab-9198-493a-970c-f6ccfd9947d3): user 'magnum' authenticated and granted access to vhost '/magnum'<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Dec 22, 2020 at 4:12 AM feilong <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p>Hi Ionut,<o:p></o:p></p>
<p>I didn't see this before on our production. Magnum auto healer just simply sends a POST request to Magnum api to update the health status. So I would suggest write a small script or even use curl to see if you can reproduce this firstly.
<o:p></o:p></p>
<p><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 19/12/20 2:27 am, Ionut Biru wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Hi again, <o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
I failed to mention that is stable/victoria with couples of patches from review. Ignore the fact that in logs it shows the 19.1.4 version in venv path.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Dec 18, 2020 at 3:22 PM Ionut Biru <<a href="mailto:ionut@fleio.com" target="_blank">ionut@fleio.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Hi guys, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I have an issue with magnum api returning an error after a while:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><code><span style="font-size:10.0pt">Server-side error: "[('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]"</span></code><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Log file: <a href="https://urldefense.com/v3/__https:/paste.xinu.at/6djE/__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnJkPZX-_Q$" target="_blank">https://paste.xinu.at/6djE/</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
This started to appear after I enabled the template auto_healing_controller = magnum-auto-healer, magnum_auto_healer_tag = v1.19.0.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Currently, I only have 4 clusters.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">After that the API is in error state and doesn't work unless I restart it.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Ionut Biru - <a href="https://urldefense.com/v3/__https:/fleio.com__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnKfpV6EIg$" target="_blank">
https://fleio.com</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Ionut Biru - <a href="https://urldefense.com/v3/__https:/fleio.com__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnKfpV6EIg$" target="_blank">
https://fleio.com</a><o:p></o:p></p>
</div>
</div>
</blockquote>
<pre>-- <o:p></o:p></pre>
<pre>Cheers & Best regards,<o:p></o:p></pre>
<pre>Feilong Wang (<span style="font-family:"MS Gothic"">王</span><span style="font-family:"Microsoft JhengHei",sans-serif">飞龙</span>)<o:p></o:p></pre>
<pre>------------------------------------------------------<o:p></o:p></pre>
<pre>Senior Cloud Software Engineer<o:p></o:p></pre>
<pre>Tel: +64-48032246<o:p></o:p></pre>
<pre>Email: <a href="mailto:flwang@catalyst.net.nz" target="_blank">flwang@catalyst.net.nz</a><o:p></o:p></pre>
<pre>Catalyst IT Limited<o:p></o:p></pre>
<pre>Level 6, Catalyst House, 150 Willis Street, Wellington<o:p></o:p></pre>
<pre>------------------------------------------------------ <o:p></o:p></pre>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Ionut Biru - <a href="https://urldefense.com/v3/__https:/fleio.com__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnKfpV6EIg$" target="_blank">
https://fleio.com</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Ionut Biru - <a href="https://urldefense.com/v3/__https:/fleio.com__;!!Ci6f514n9QsL8ck!3b_NgWO8HXsOoUOdTUZp4KEzKcx9zpWomeb2yGJ4RRqkS1QI159_zwjwVnKfpV6EIg$" target="_blank">
https://fleio.com</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Ionut Biru - <a href="https://urldefense.com/v3/__https:/fleio.com__;!!Ci6f514n9QsL8ck!wv_wzG-Ntk0gd3ReOupQl-iXIcWpPR3genCqeKNY5JCKZDWxQHSqqa-uxxg3AiT38g$" target="_blank">
https://fleio.com</a><o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>