<div dir="ltr"><div id="gmail-magicdomid2" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">Hey all!</span></div><div id="gmail-magicdomid3" class="gmail-ace-line"><br></div><div id="gmail-magicdomid4" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">Here is a </span><span class="gmail-author-a-z69zz79z34z79zm1yz74zwz65zz80zyz69z72">thread</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> to coordinate all the teams (oslo, nova, stable, requirements) </span><span class="gmail-author-a-z69zz79z34z79zm1yz74zwz65zz80zyz69z72">working on</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> the update of </span><span class="gmail-author-a-z69zz79z34z79zm1yz74zwz65zz80zyz69z72">the </span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">oslo.service constraint in the</span><span class="gmail-author-a-z69zz79z34z79zm1yz74zwz65zz80zyz69z72"> Rocky</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> requirements.</span></div><div id="gmail-magicdomid5" class="gmail-ace-line"><br></div><div id="gmail-magicdomid6" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"># Summary</span></div><div id="gmail-magicdomid7" class="gmail-ace-line"><br></div><div id="gmail-magicdomid8" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">Usage of threading event with eventlet caused inefficient code (causing many useless system calls and  high CPU usage).</span></div><div id="gmail-magicdomid9" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">This issue was already fixed on oslo.service master and we also want to fix it in stable/rocky.</span></div><div id="gmail-magicdomid10" class="gmail-ace-line"><br></div><div id="gmail-magicdomid11" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">Our main issue is how to fix the high CPU usage on stable/rocky without break the nova CI.</span></div><div id="gmail-magicdomid12" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> </span></div><div id="gmail-magicdomid231" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">Indeed, we already have backport</span><span class="gmail-author-a-z69zz79z34z79zm1yz74zwz65zz80zyz69z72">ed</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> the eventlet related fix to oslo.service but this fix requires also a nova update to avoid nova CI errors due to threading removal on oslo.service that introduce the nova CI errors.</span></div><div id="gmail-magicdomid14" class="gmail-ace-line"><br></div><div id="gmail-magicdomid15" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">A fix was proposed and merged on oslo.service master to introduce a new feature (fixture) that avoid the nova CI errors, but</span></div><div id="gmail-magicdomid229" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">backport</span><span class="gmail-author-a-z69zz79z34z79zm1yz74zwz65zz80zyz69z72">ing</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> the master fix to Rocky introduce</span><span class="gmail-author-a-z69zz79z34z79zm1yz74zwz65zz80zyz69z72">s</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> a new feature into a stable branch so this is also an issue.</span></div><div id="gmail-magicdomid17" class="gmail-ace-line"><br></div><div id="gmail-magicdomid18" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">So we need to discuss with all the teams to find a proper solution.</span></div><div id="gmail-magicdomid19" class="gmail-ace-line"><br></div><div id="gmail-magicdomid20" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"># History</span></div><div id="gmail-magicdomid21" class="gmail-ace-line"><br></div><div id="gmail-magicdomid240" class="gmail-ace-line"><span class="gmail-author-a-z69zz79z34z79zm1yz74zwz65zz80zyz69z72">A f</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">ew weeks ago this issue was opened on oslo.service (</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z gmail-url"><a href="https://bugs.launchpad.net/oslo.service/+bug/1798774)">https://bugs.launchpad.net/oslo.service/+bug/1798774)</a></span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> and it was fixed by this submited patch</span><span class="gmail-author-a-z69zz79z34z79zm1yz74zwz65zz80zyz69z72"> on the master branch</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> ( </span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z gmail-url"><a href="https://review.openstack.org/#/c/611807/">https://review.openstack.org/#/c/611807/</a></span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> ).</span></div><div id="gmail-magicdomid23" class="gmail-ace-line"><br></div><div id="gmail-magicdomid239" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">This change use the proper event primitive to fix the performance issue.</span></div><div id="gmail-magicdomid25" class="gmail-ace-line"><br></div><div id="gmail-magicdomid26" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">A new version of oslo.service was released (1.32.1)</span></div><div id="gmail-magicdomid27" class="gmail-ace-line"><br></div><div id="gmail-magicdomid241" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">Since these changes was introduced into oslo.service master, nova facing some issues into the master CI process, due to the threading changes, and they was fixed by these patches ( </span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z gmail-url"><a href="https://review.openstack.org/#/c/615724/">https://review.openstack.org/#/c/615724/</a></span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">, </span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z gmail-url"><a href="https://review.openstack.org/#/c/617989/">https://review.openstack.org/#/c/617989/</a></span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> ) into master.</span></div><div id="gmail-magicdomid29" class="gmail-ace-line"><br></div><div id="gmail-magicdomid243" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">Few weeks ago I have backport to oslo.service some changes ( </span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z gmail-url"><a href="https://review.openstack.org/#/c/614489/">https://review.openstack.org/#/c/614489/</a></span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> ) from master to stable/rocky to also fix the problem in the rocky release.</span></div><div id="gmail-magicdomid31" class="gmail-ace-line"><br></div><div id="gmail-magicdomid245" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">When this backport was merged we have created a new release of oslo.service (1.31.6) ( </span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z gmail-url"><a href="https://review.openstack.org/#/c/616505/">https://review.openstack.org/#/c/616505/</a></span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> ) (stable/rocky version).</span></div><div id="gmail-magicdomid33" class="gmail-ace-line"><br></div><div id="gmail-magicdomid34" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">Then the openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI  </span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z gmail-url"><a href="https://review.openstack.org/#/c/618834/">https://review.openstack.org/#/c/618834/</a></span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> so this patch is currently blocked to avoid nova CI error.</span></div><div id="gmail-magicdomid35" class="gmail-ace-line"><br></div><div id="gmail-magicdomid36" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"># Issue</span></div><div id="gmail-magicdomid37" class="gmail-ace-line"><br></div><div id="gmail-magicdomid38" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">Since the oslo.services threading changes w</span><span class="gmail-author-a-z69zz79z34z79zm1yz74zwz65zz80zyz69z72">ere</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> backported to rocky we risk to  faces the same issues inside the nova rocky CI if we update the requirements.</span></div><div id="gmail-magicdomid39" class="gmail-ace-line"><br></div><div id="gmail-magicdomid369" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">In parallel in oslo.service we have start</span><span class="gmail-author-a-z69zz79z34z79zm1yz74zwz65zz80zyz69z72">ed</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> to backport a new patch who introduces fixture  ( </span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z gmail-url"><a href="https://review.openstack.org/#/c/617989/">https://review.openstack.org/#/c/617989/</a></span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> ) from master to rocky, and also we start to backport on nova rocky branch ( </span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z gmail-url"><a href="https://review.openstack.org/619019">https://review.openstack.org/619019</a></span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">, </span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z gmail-url"><a href="https://review.openstack.org/619022">https://review.openstack.org/619022</a></span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> ) patches who use oslo.service.fixture and who solve the nova CI issue. The patch on oslo.service exposes a public oslo_service.fixture.SleepFixture for this purpose. It can be maintained opaquely as internals change without affecting its consumers. </span></div><div id="gmail-magicdomid41" class="gmail-ace-line"><br></div><div id="gmail-magicdomid42" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">The main problem is that the patch bring a new functionality to a stable branch (oslo.service rocky) but this patch help to fix the nova issue.</span></div><div id="gmail-magicdomid43" class="gmail-ace-line"><br></div><div id="gmail-magicdomid44" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">Also openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI  </span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z gmail-url"><a href="https://review.openstack.org/#/c/618834/">https://review.openstack.org/#/c/618834/</a></span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> since the oslo service 1.31.6 is incompatible with novas stable rocky unittest due to the threading changes.</span></div><div id="gmail-magicdomid45" class="gmail-ace-line"><br></div><div id="gmail-magicdomid46" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"># Questions and proposed solutions</span></div><div id="gmail-magicdomid47" class="gmail-ace-line"><br></div><div id="gmail-magicdomid48" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">This thread try to summarize the current situation.</span></div><div id="gmail-magicdomid49" class="gmail-ace-line"><br></div><div id="gmail-magicdomid50" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix.</span></div><div id="gmail-magicdomid51" class="gmail-ace-line"><br></div><div id="gmail-magicdomid52" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z gmail-url"><a href="https://review.openstack.org/#/c/617989/)">https://review.openstack.org/#/c/617989/)</a></span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> ?</span></div><div id="gmail-magicdomid53" class="gmail-ace-line"><br></div><div id="gmail-magicdomid54" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example: </span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z gmail-url"><a href="https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compute_mgr.py)">https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compute_mgr.py)</a></span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> ?  </span></div><div id="gmail-magicdomid235" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">This is only a fix on the nova side and </span><span class="gmail-author-a-z69zz79z34z79zm1yz74zwz65zz80zyz69z72">it</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> allow</span><span class="gmail-author-a-z69zz79z34z79zm1yz74zwz65zz80zyz69z72">s</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> us to update oslo.service requirements and allow</span><span class="gmail-author-a-z69zz79z34z79zm1yz74zwz65zz80zyz69z72">s</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> us to fix the high CPU usage issue. I've submit this patch (</span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z gmail-url"><a href="https://review.openstack.org/619246)">https://review.openstack.org/619246)</a></span><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z"> who implement the description above.</span></div><div id="gmail-magicdomid56" class="gmail-ace-line"><br></div><div id="gmail-magicdomid118" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">Personaly I think we need to find an another approach like the mocking remplacement (c.f 2).</span></div><div id="gmail-magicdomid120" class="gmail-ace-line"><br></div><div id="gmail-magicdomid185" class="gmail-ace-line"><span class="gmail-author-a-o0z122zo9qz74zz66zz69zz66zgdxz89zz75zz65z">We need to decide which way we use and to discuss about other solutions.</span></div><div><br>-- <br><div dir="ltr" class="m_5390074029165786051gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Hervé Beraud</div><div>Senior Software Engineer<br></div><div>Red Hat - Openstack Oslo</div><div>irc: hberaud</div><div>-----BEGIN PGP SIGNATURE-----<br><br>wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+<br>Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+<br>RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP<br>F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G<br>5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g<br>glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw<br>m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ<br>hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0<br>qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y<br>F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3<br>B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O<br>v6rDpkeNksZ9fFSyoY2o<br>=ECSj<br>-----END PGP SIGNATURE-----<br><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>