<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Maybe double check that your rootwrap config is up to date?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
/etc/neutron/<span style="font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important">rootwrap<span> .conf and /etc/neutron/<span style="font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important">rootwrap.d</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
(Make sure to pick the appropriate branch in github)</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<a href="https://github.com/openstack/neutron/blob/master/etc/rootwrap.conf" id="LPNoLP923608">https://github.com/openstack/neutron/blob/master/etc/rootwrap.conf</a><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<a href="https://github.com/openstack/neutron/tree/master/etc/neutron/rootwrap.d" id="LPNoLP513599">https://github.com/openstack/neutron/tree/master/etc/neutron/rootwrap.d</a><br>
</div>
<br>
<br>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Albert Braden <Albert.Braden@synopsys.com><br>
<b>Sent:</b> Thursday, October 10, 2019 1:45 PM<br>
<b>To:</b> Erik Olof Gunnar Andersson <eandersson@blizzard.com>; Chris Apsey <bitskrieg@bitskrieg.net><br>
<b>Cc:</b> openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org><br>
<b>Subject:</b> RE: Port creation times out for some VMs in large group</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"MS Gothic"}
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:Consolas}
@font-face
{}
@font-face
{font-family:"MS PGothic"}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"MS PGothic",sans-serif}
a:link, span.x_MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.x_MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
pre
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"MS Gothic"}
span.x_HTMLPreformattedChar
{font-family:Consolas}
p.x_msonormal0, li.x_msonormal0, div.x_msonormal0
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"MS PGothic",sans-serif}
p.x_xmsonormal, li.x_xmsonormal, div.x_xmsonormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
p.x_xmsonormal0, li.x_xmsonormal0, div.x_xmsonormal0
{margin-right:0in;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
p.x_xmsochpdefault, li.x_xmsochpdefault, div.x_xmsochpdefault
{margin-right:0in;
margin-left:0in;
font-size:12.0pt;
font-family:"Calibri",sans-serif}
span.x_xmsohyperlink
{color:blue;
text-decoration:underline}
span.x_xmsohyperlinkfollowed
{color:purple;
text-decoration:underline}
span.x_xemailstyle20
{font-family:"Calibri",sans-serif;
color:windowtext}
span.x_EmailStyle26
{font-family:"Calibri",sans-serif;
color:windowtext}
span.x_EmailStyle27
{font-family:"Calibri",sans-serif;
color:windowtext}
span.x_EmailStyle28
{font-family:"Calibri",sans-serif;
color:windowtext}
span.x_EmailStyle30
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="blue" vlink="purple">
<div class="x_WordSection1">
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">The errors appear to start with this line:</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">2019-10-10 13:42:48.261 1211336 ERROR neutron.agent.linux.utils [req-42c530f6-6e08-47c1-8ed4-dcb31c9cd972 - - - - -] Rootwrap error running command: ['iptables-save', '-t',
'raw']: Exception: Failed to spawn rootwrap process.</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">We’re not running iptables. Do we need it, to use the rootwrap daemon?</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> Albert Braden <Albert.Braden@synopsys.com>
<br>
<b>Sent:</b> Thursday, October 10, 2019 12:13 PM<br>
<b>To:</b> Erik Olof Gunnar Andersson <eandersson@blizzard.com>; Chris Apsey <bitskrieg@bitskrieg.net><br>
<b>Cc:</b> openstack-discuss@lists.openstack.org<br>
<b>Subject:</b> RE: Port creation times out for some VMs in large group</span></p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">It looks like something is still missing. I added the line to /etc/sudoers.d/neutron_sudoers:</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">root@us01odc-qa-ctrl3:/var/log/neutron# cat /etc/sudoers.d/neutron_sudoers</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">Defaults:neutron !requiretty</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf *</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">Then I restarted neutron services and the error was gone… for a few minutes, and then it came back on ctrl3. Ctrl1/2 aren’t erroring at this time. I changed neutron’s shell
and tested the daemon command and it seems to work:</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">root@us01odc-qa-ctrl3:~# su - neutron</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">neutron@us01odc-qa-ctrl3:~$ /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">/tmp/rootwrap-5b1QoP/rootwrap.sock</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">Z%</span><span style="font-size:11.0pt; font-family:"Arial",sans-serif">▒</span><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">"</span><span style="font-size:11.0pt; font-family:"Arial",sans-serif">▒▒▒</span><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">Vs</span><span style="font-size:11.0pt; font-family:"Arial",sans-serif">▒▒</span><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">5-</span><span style="font-size:11.0pt; font-family:"Arial",sans-serif">▒</span><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">,a</span><span style="font-size:11.0pt; font-family:"Arial",sans-serif">▒▒▒▒</span><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">G</span><span style="font-size:11.0pt; font-family:"Arial",sans-serif">▒▒▒▒</span><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">v</span><span style="font-size:11.0pt; font-family:"Arial",sans-serif">▒▒</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">But neutron-linuxbridge-agent.log still scrolls errors:</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"><a href="https://urldefense.com/v3/__http://paste.openstack.org/show/782740/__;!2E0gRdhhnqPNNL0!z5cwPxQ1y_zz0MvtFzMZSCIh7-3d80kxciHbPtkj4LbHCzSkzNpf36RwLi8kWGm1Ew$">http://paste.openstack.org/show/782740/</a></span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">It appears that there is another factor besides the config, because even when the sudoers line was missing, it would work for hours or days before the error started. It
has been working in our prod cluster for about a week now, without the sudoers line. It seems like it should not work that way. What am I missing?</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com">eandersson@blizzard.com</a>>
<br>
<b>Sent:</b> Thursday, October 10, 2019 11:08 AM<br>
<b>To:</b> Albert Braden <<a href="mailto:albertb@synopsys.com">albertb@synopsys.com</a>>; Chris Apsey <<a href="mailto:bitskrieg@bitskrieg.net">bitskrieg@bitskrieg.net</a>><br>
<b>Cc:</b> <a href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a><br>
<b>Subject:</b> RE: Port creation times out for some VMs in large group</span></p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">Yea – if you look at your sudoers its only allowing the old traditional rootwrap, and not the new daemon. You need both.</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">Defaults:neutron !requiretty</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<div>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">Best Regards, Erik Olof Gunnar Andersson</span></p>
</div>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> Albert Braden <<a href="mailto:Albert.Braden@synopsys.com">Albert.Braden@synopsys.com</a>>
<br>
<b>Sent:</b> Thursday, October 10, 2019 11:05 AM<br>
<b>To:</b> Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com">eandersson@blizzard.com</a>>; Chris Apsey <<a href="mailto:bitskrieg@bitskrieg.net">bitskrieg@bitskrieg.net</a>><br>
<b>Cc:</b> <a href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a><br>
<b>Subject:</b> RE: Port creation times out for some VMs in large group</span></p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">I have the neutron sudoers line under sudoers.d:</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"><a href="mailto:root@us01odc-qa-ctrl1:/etc/sudoers.d">root@us01odc-qa-ctrl1:/etc/sudoers.d#</a> cat neutron_sudoers</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">Defaults:neutron !requiretty</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">Whatever is causing this didn’t start until I had been running the rootwrap daemon for 2 weeks, and it has not started in our prod cluster.</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com">eandersson@blizzard.com</a>>
<br>
<b>Sent:</b> Wednesday, October 9, 2019 6:40 PM<br>
<b>To:</b> Albert Braden <<a href="mailto:albertb@synopsys.com">albertb@synopsys.com</a>>; Chris Apsey <<a href="mailto:bitskrieg@bitskrieg.net">bitskrieg@bitskrieg.net</a>><br>
<b>Cc:</b> <a href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a><br>
<b>Subject:</b> Re: Port creation times out for some VMs in large group</span></p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black">You are probably missing an entry in your sudoers file.<br>
<br>
You need something like</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black"> </span></p>
</div>
<div>
<pre style="margin-bottom:7.5pt; line-height:15.0pt; background:whitesmoke; word-break:break-all; border-radius:4px; border:rgba(0,0,0,0.15)"><span style="font-size:10.0pt; font-family:Consolas; color:#333333">neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf</span></pre>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black"> </span></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="x_MsoNormal"><b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:black">From:</span></b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:black"> Albert Braden <<a href="mailto:Albert.Braden@synopsys.com">Albert.Braden@synopsys.com</a>><br>
<b>Sent:</b> Wednesday, October 9, 2019 5:20 PM<br>
<b>To:</b> Chris Apsey <<a href="mailto:bitskrieg@bitskrieg.net">bitskrieg@bitskrieg.net</a>><br>
<b>Cc:</b> <a href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a> <<a href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> RE: Port creation times out for some VMs in large group</span> </p>
<div>
<p class="x_MsoNormal"> </p>
</div>
</div>
<div>
<div>
<p class="x_xmsonormal">We tested this in dev and qa and then implemented in production and it did make a difference, but 2 weeks later we started seeing an issue, first in dev, and then in qa. In syslog we see neutron-linuxbridge-agent.service stopping and
starting[1]. In neutron-linuxbridge-agent.log we see a rootwrap error[2]: “Exception: Failed to spawn rootwrap process.”</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">If I comment out ‘root_helper_daemon = "sudo /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf"’ and restart neutron services then the error goes away.</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">How can I use the root_helper_daemon setting without creating this new error?</p>
<p class="x_xmsonormal"> </p>
<div>
<p class="x_xmsonormal"><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_782622_&d=DwMFAg&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=hT1YhRjyM0zYEXl5feVL1lmrbHaM7sytttrPvi1aZzg&s=mxxRA-SpuIF9xc1Pgx9RrbC3UjGdAFrXm4X6lH6UbR8&e=">http://paste.openstack.org/show/782622/</a></p>
<p class="x_xmsonormal"> </p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>