<div><div dir="auto">Hi Lingxian Kong,</div></div><div dir="auto"><br></div><div dir="auto">That’s actually very interesting as I’ve come to the same conclusion this morning during my investigation and was starting to think about a fix, which it seems you already made!</div><div dir="auto"><br></div><div dir="auto">Is there a reason why it didn’t was backported to rocky?</div><div dir="auto"><br></div><div dir="auto">Very helpful, many many thanks to you you clearly spare me hours of works! I’ll get a review of your patch and test it on our lab.</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le mar. 4 juin 2019 à 11:06, Gaël THEROND <<a href="mailto:gael.therond@gmail.com">gael.therond@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div dir="auto">Hi Felix,</div></div><div dir="auto"><br></div><div dir="auto">« Glad » you had the same issue before, and yes of course I looked at the HM<span class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(51,51,51)"></span> logs <span class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(51,51,51)">which is were I</span> actually <span class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(51,51,51)">found out</span> that this event was triggered by octavia (Beside the DB data that validated that) <span class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(51,51,51)">here is my log trace related to this event, It doesn't really shows major issue IMHO.<br><br>Here is the stacktrace that our octavia service archived for our both controllers servers, with the initial loadbalancer creation trace (Worker.log) and both controllers triggered task (Health-Manager.log). <br><br><a href="http://paste.openstack.org/show/7z5aZYu12Ttoae3AOhwF/" target="_blank">http://paste.openstack.org/show/7z5aZYu12Ttoae3AOhwF/</a><br></span></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(51,51,51)">I well may have miss something in it, but I don't see something strange on from my point of view.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(51,51,51)">Feel free to tell me if you spot something weird.</div><br></div></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le mar. 4 juin 2019 à 10:38, Felix Hüttner <felix.huettner@mail.schwarz> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="DE">
<div class="m_-4850122681777510811gmail-m_-8545333611937500375m_-6515720819355056471WordSection1">
<p class="MsoNormal"><span>Hi Gael,<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">we had a similar issue in the past.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">You could check the octiava healthmanager log (should be on the same node where the worker is running).
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">This component monitors the status of the Amphorae and restarts them if they don’t trigger a callback after a specific time. This might also happen if there is some connection issue
between the two components.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">But normally it should at least restart the LB with new Amphorae…<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Hope that helps<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Felix<u></u><u></u></span></p></div></div><div lang="DE"><div class="m_-4850122681777510811gmail-m_-8545333611937500375m_-6515720819355056471WordSection1">
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Gaël THEROND <<a href="mailto:gael.therond@gmail.com" target="_blank">gael.therond@gmail.com</a>>
<br>
<b>Sent:</b> Tuesday, June 4, 2019 9:44 AM<br>
<b>To:</b> Openstack <<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a>><br>
<b>Subject:</b> [OCTAVIA][ROCKY] - MASTER & BACKUP instances unexpectedly deleted by octavia<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal">Hi guys,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I’ve a weird situation here.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I smoothly operate a large scale multi-region Octavia service using the default amphora driver which imply the use of nova instances as loadbalancers.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Everything is running really well and our customers (K8s and traditional users) are really happy with the solution so far.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">However, yesterday one of those customers using the loadbalancer in front of their ElasticSearch cluster poked me because this loadbalancer suddenly passed from ONLINE/OK to ONLINE/ERROR, meaning the amphoras were no longer available but
yet the anchor/member/pool and listeners settings were still existing.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">So I investigated and found out that the loadbalancer amphoras have been destroyed by the octavia user.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">The weird part is, both the master and the backup instance have been destroyed at the same moment by the octavia service user.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Is there specific circumstances where the octavia service could decide to delete the instances but not the anchor/members/pool ?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">It’s worrying me a bit as there is no clear way to trace why does Octavia did take this action.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I digged within the nova and Octavia DB in order to correlate the action but except than validating my investigation it doesn’t really help as there are no clue of why the octavia service did trigger the deletion.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If someone have any clue or tips to give me I’ll be more than happy to discuss this situation.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Cheers guys!<u></u><u></u></p>
</div>
</div></div><div lang="DE"><div class="m_-4850122681777510811gmail-m_-8545333611937500375m_-6515720819355056471WordSection1"></div>
Hinweise zum Datenschutz finden Sie <a href="https://www.datenschutz.schwarz" target="_blank">hier</a>.
</div>
</blockquote></div></div>
</blockquote></div></div>