<html><head></head><body><div class="ydpdf48d069yahoo-style-wrap" style="font-family: times new roman, new york, times, serif; font-size: 16px;"><div></div>
<div><br></div></div><div id="ydp96ad3225yahoo_quoted_0845201174" class="ydp96ad3225yahoo_quoted"><div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div><div id="ydp96ad3225yiv7683476856"><div><div class="ydp96ad3225yiv7683476856ydp7906eb98yahoo-style-wrap" style="font-family:times new roman, new york, times, serif;font-size:16px;"><div></div>
<div dir="ltr">Hi neutron team,</div><div dir="ltr"><br clear="none"></div><div dir="ltr">We've been troubleshooting an issue with neutron's dhcp-client for sometime now.</div><div dir="ltr"><br clear="none"></div><div dir="ltr"><br clear="none"></div><div dir="ltr">We were previously on neutron 12.0.5 and observed that upon reloading 5-8 baremetals simultaneously almost </div><div dir="ltr">always led to a few baremetals failing PXE boot during provisioning and/or cleaning.</div><div dir="ltr"><br clear="none"></div><div dir="ltr"><div><span style="font-size:11.0pt;font-family:Calibri, sans-serif;">This was traced to a
fixed bug.<br clear="none">
<a shape="rect" href="https://bugs.launchpad.net/neutron/+bug/1760047" rel="nofollow" target="_blank">https://bugs.launchpad.net/neutron/+bug/1760047</a><br clear="none">
which was applied to Queens in April 2019.<br clear="none">
<a shape="rect" href="https://review.opendev.org/#/c/649580/" rel="nofollow" target="_blank">https://review.opendev.org/#/c/649580/</a><br clear="none">
<br clear="none">
</span></div><br clear="none"></div><div dir="ltr"><div><span style="font-size:11.0pt;font-family:Calibri, sans-serif;">We patched the above fix but found out the problem was not resolved.<br clear="none">
The fix gets rid of the semaphores by serializing the multiple messages into a
Priority Queue.<br clear="none">
The Priority Queue then drains the messages serially one by one making sure not
to yield during the processing<br clear="none">
of each message. All in all this just seems like a more elegant way of
getting rid of the semaphores but does not really <br clear="none">
fix the issue at hand.</span></div><div><span style="font-size:11.0pt;font-family:Calibri, sans-serif;"><br clear="none"></span></div><div><span style="font-size:11.0pt;font-family:Calibri, sans-serif;"><br clear="none"></span></div><div><span style="font-size:11.0pt;font-family:Calibri, sans-serif;"><br clear="none"></span></div><div dir="ltr"><span style="font-size:11.0pt;font-family:Calibri, sans-serif;">Below are the logs from dhcp-agent in neutron release 12.0.5. As can be seen the semaphore locks all threads for almost 6 seconds.</span></div><div dir="ltr"><span style="font-size:11.0pt;font-family:Calibri, sans-serif;">While the below has been fixed using </span></div><div dir="ltr"><div><a shape="rect" href="https://review.opendev.org/#/c/649580/" style="color:rgb(51, 143, 233);text-decoration-line:underline;font-family:Calibri, sans-serif;font-size:14.6667px;" rel="nofollow" target="_blank">https://review.opendev.org/#/c/649580/</a><br clear="none" style="color:rgb(0, 0, 0);font-family:Calibri, sans-serif;font-size:14.6667px;"></div><div dir="ltr">the underlying problem has not been fixed. The semaphore has been removed but instead the message is being serialized and does not yield</div><div dir="ltr">resulting in PXE boot failures on the baremetal nodes.</div><div dir="ltr"><br clear="none"></div><div dir="ltr">Any pointers would be appreciated.</div><div dir="ltr"><br clear="none"></div><div dir="ltr">thanks,</div><div class="ydp96ad3225yiv7683476856yqt8434578522" id="ydp96ad3225yiv7683476856yqtfd41175"><div dir="ltr">Fred</div></div><div><br clear="none"></div><div><br clear="none"></div></div><div dir="ltr"><p class="ydp96ad3225yiv7683476856ydp9856525MsoNormal" style="margin-left:.5in;">2019-10-03 18:07:37.454 318956
DEBUG oslo_concurrency.lockutils [req-eac79995-3846-46e6-b946-c5b5ccdb7aa5
8941137e383548bda725e74a93b2f865 19f6fb7446dc47dd88c63cf03c1cce94 - - -] <span style="background:yellow;">Acquired semaphore
"dhcp-agent-network-lock-077aa2d1-605c-48ec-842d-7dd6767bfd01" lock
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212</span></p>
<p class="ydp96ad3225yiv7683476856ydp9856525MsoNormal" style="margin-left:.5in;">2019-10-03 18:07:37.455 318956
DEBUG neutron.agent.dhcp.agent [req-eac79995-3846-46e6-b946-c5b5ccdb7aa5 8941137e383548bda725e74a93b2f865
19f6fb7446dc47dd88c63cf03c1cce94 - - -] Calling driver for network:
077aa2d1-605c-48ec-842d-7dd6767bfd01 action: reload_allocations call_driver
/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py:135</p>
<p class="ydp96ad3225yiv7683476856ydp9856525MsoNormal" style="margin-left:.5in;">2019-10-03 18:07:37.456 318956
DEBUG neutron.agent.linux.utils [req-eac79995-3846-46e6-b946-c5b5ccdb7aa5
8941137e383548bda725e74a93b2f865 19f6fb7446dc47dd88c63cf03c1cce94 - - -]
Running command (rootwrap daemon): ['ip', 'netns', 'exec',
'qdhcp-077aa2d1-605c-48ec-842d-7dd6767bfd01', 'dhcp_release', 'ns-8387b854-d1',
'10.33.27.77', '9c:71:3a:cb:7c:43', '01:9c:71:3a:cb:7c:43']
execute_rootwrap_daemon
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:108</p>
<p class="ydp96ad3225yiv7683476856ydp9856525MsoNormal" style="margin-left:.5in;">2019-10-03 18:07:38.101 318956
DEBUG neutron.agent.linux.utils [req-eac79995-3846-46e6-b946-c5b5ccdb7aa5
8941137e383548bda725e74a93b2f865 19f6fb7446dc47dd88c63cf03c1cce94 - - -]
Running command (rootwrap daemon): ['ip', 'netns', 'exec',
'qdhcp-077aa2d1-605c-48ec-842d-7dd6767bfd01', 'dhcp_release', 'ns-8387b854-d1',
'10.33.27.75', '9c:71:3a:cb:7b:fb'] execute_rootwrap_daemon
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:108</p>
<p class="ydp96ad3225yiv7683476856ydp9856525MsoNormal" style="margin-left:.5in;">2019-10-03 18:07:38.717 318956
DEBUG neutron.agent.linux.utils [req-eac79995-3846-46e6-b946-c5b5ccdb7aa5
8941137e383548bda725e74a93b2f865 19f6fb7446dc47dd88c63cf03c1cce94 - - -]
Running command (rootwrap daemon): ['ip', 'netns', 'exec',
'qdhcp-077aa2d1-605c-48ec-842d-7dd6767bfd01', 'dhcp_release', 'ns-8387b854-d1',
'10.33.27.75', '9c:71:3a:cb:7b:fb', 'ff:3a:cb:7b:fb:00:04:8a:ef:2f:58:b4:20:45:03:80:27:0f:15:84:a4:70:7b']
execute_rootwrap_daemon
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:108</p>
<p class="ydp96ad3225yiv7683476856ydp9856525MsoNormal" style="margin-left:.5in;">2019-10-03 18:07:39.631 318956
DEBUG neutron.agent.linux.dhcp [req-eac79995-3846-46e6-b946-c5b5ccdb7aa5
8941137e383548bda725e74a93b2f865 19f6fb7446dc47dd88c63cf03c1cce94 - - -]
Building host file: /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
_output_hosts_file
/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py:695</p>
<p class="ydp96ad3225yiv7683476856ydp9856525MsoNormal" style="margin-left:.5in;">2019-10-03 18:07:39.632 318956
DEBUG neutron.agent.linux.dhcp [req-eac79995-3846-46e6-b946-c5b5ccdb7aa5
8941137e383548bda725e74a93b2f865 19f6fb7446dc47dd88c63cf03c1cce94 - - -] Done
building host file
/var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
_output_hosts_file
/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py:734</p>
<p class="ydp96ad3225yiv7683476856ydp9856525MsoNormal" style="margin-left:.5in;">2019-10-03 18:07:39.633 318956
DEBUG neutron.agent.linux.utils [req-eac79995-3846-46e6-b946-c5b5ccdb7aa5
8941137e383548bda725e74a93b2f865 19f6fb7446dc47dd88c63cf03c1cce94 - - -]
Running command (rootwrap daemon): ['ip', 'netns', 'exec',
'qdhcp-077aa2d1-605c-48ec-842d-7dd6767bfd01', 'ip', 'addr', 'show',
'ns-8387b854-d1'] execute_rootwrap_daemon
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:108</p>
<p class="ydp96ad3225yiv7683476856ydp9856525MsoNormal" style="margin-left:.5in;">2019-10-03 18:07:40.263 318956
DEBUG neutron.agent.linux.utils [req-eac79995-3846-46e6-b946-c5b5ccdb7aa5
8941137e383548bda725e74a93b2f865 19f6fb7446dc47dd88c63cf03c1cce94 - - -]
Running command (rootwrap daemon): ['kill', '-HUP', '319109']
execute_rootwrap_daemon
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:108</p>
<p class="ydp96ad3225yiv7683476856ydp9856525MsoNormal" style="margin-left:.5in;">2019-10-03 18:07:40.843 318956
DEBUG neutron.agent.linux.dhcp [req-eac79995-3846-46e6-b946-c5b5ccdb7aa5
8941137e383548bda725e74a93b2f865 19f6fb7446dc47dd88c63cf03c1cce94 - - -]
Reloading allocations for network: 077aa2d1-605c-48ec-842d-7dd6767bfd01
reload_allocations
/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py:524</p>
<p class="ydp96ad3225yiv7683476856ydp9856525MsoNormal" style="margin-left:.5in;">2019-10-03 18:07:40.843 318956
DEBUG neutron.agent.linux.utils [req-eac79995-3846-46e6-b946-c5b5ccdb7aa5
8941137e383548bda725e74a93b2f865 19f6fb7446dc47dd88c63cf03c1cce94 - - -]
Running command (rootwrap daemon): ['ip', 'netns', 'exec',
'qdhcp-077aa2d1-605c-48ec-842d-7dd6767bfd01', 'ip', '-4', 'route', 'list',
'dev', 'ns-8387b854-d1'] execute_rootwrap_daemon
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:108</p>
<p class="ydp96ad3225yiv7683476856ydp9856525MsoNormal" style="margin-left:.5in;">2019-10-03 18:07:41.462 318956
DEBUG neutron.agent.linux.utils [req-eac79995-3846-46e6-b946-c5b5ccdb7aa5 8941137e383548bda725e74a93b2f865
19f6fb7446dc47dd88c63cf03c1cce94 - - -] Running command (rootwrap daemon):
['ip', 'netns', 'exec', 'qdhcp-077aa2d1-605c-48ec-842d-7dd6767bfd01', 'ip',
'-6', 'route', 'list', 'dev', 'ns-8387b854-d1'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:108</p>
<p class="ydp96ad3225yiv7683476856ydp9856525MsoNormal" style="margin-left:.5in;">2019-10-03 18:07:42.101 318956
DEBUG oslo_concurrency.lockutils [req-eac79995-3846-46e6-b946-c5b5ccdb7aa5
8941137e383548bda725e74a93b2f865 19f6fb7446dc47dd88c63cf03c1cce94 - - <span style="background:yellow;">-] Releasing semaphore
"dhcp-agent-network-lock-077aa2d1-605c-48ec-842d-7dd6767bfd01" lock
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:228</span></p><div class="ydp96ad3225yiv7683476856yqt8434578522" id="ydp96ad3225yiv7683476856yqtfd68629">
<div> </div></div></div><div class="ydp96ad3225yiv7683476856yqt8434578522" id="ydp96ad3225yiv7683476856yqtfd58360"><div><span style="font-size:11.0pt;font-family:Calibri, sans-serif;"><br clear="none">
<br clear="none">
</span></div><br clear="none"></div></div><div class="ydp96ad3225yiv7683476856yqt8434578522" id="ydp96ad3225yiv7683476856yqtfd64541"><div dir="ltr"><div><div style="counter-increment:snippet 1;margin-left:32px;padding:4px 8px 0px;position:relative;color:rgb(29, 28, 29);font-family:Monaco, Menlo, Consolas, Courier New, monospace;"><pre style="margin-top:0px;margin-bottom:0px;padding:0px 4px;font-size:inherit;line-height:inherit;white-space:pre-wrap;font-family:inherit;border-width:0px;border-radius:0px;background:transparent;color:inherit;z-index:2;position:relative;"><br clear="none"></pre></div></div></div></div></div></div></div></div>
</div>
</div></body></html>