<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=koi8-r">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Gulim;
panose-1:2 11 6 0 0 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"MS PGothic";
panose-1:2 11 6 0 7 2 5 8 2 4;}
@font-face
{font-family:"\@MS PGothic";}
@font-face
{font-family:"\@Gulim";
panose-1:2 11 6 0 0 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Gulim",sans-serif;
mso-fareast-language:KO;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Gulim",sans-serif;
mso-fareast-language:KO;}
p.xxmsonormal, li.xxmsonormal, div.xxmsonormal
{mso-style-name:x_x_msonormal;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Gulim",sans-serif;
mso-fareast-language:KO;}
p.xxxmsonormal, li.xxxmsonormal, div.xxxmsonormal
{mso-style-name:x_x_xmsonormal;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Gulim",sans-serif;
mso-fareast-language:KO;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">It appears that the extra * was the issue. After removing it I can run the rootwrap daemon without errors. I’m not 100% sure because the issue took
2 weeks to show up after the initial config change, but this seems to have fixed the problem.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Erik Olof Gunnar Andersson <eandersson@blizzard.com>
<br>
<b>Sent:</b> Thursday, October 10, 2019 6:21 PM<br>
<b>To:</b> Albert Braden <albertb@synopsys.com>; Chris Apsey <bitskrieg@bitskrieg.net><br>
<b>Cc:</b> openstack-discuss@lists.openstack.org<br>
<b>Subject:</b> Re: Port creation times out for some VMs in large group<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">Btw I still think your suders is slightly incorrect. I feel like that is significant, but not a hundred.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">Drop the star at the end of the last line.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#201F1E">root@us01odc-qa-ctrl3:/var/log/neutron# cat /etc/sudoers.d/neutron_sudoers</span><span style="font-family:"MS PGothic",sans-serif;color:#201F1E"><o:p></o:p></span></p>
<p style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#201F1E">Defaults:neutron !requiretty</span><span style="font-family:"MS PGothic",sans-serif;color:#201F1E"><o:p></o:p></span></p>
<p style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#201F1E"> </span><span style="font-family:"MS PGothic",sans-serif;color:#201F1E"><o:p></o:p></span></p>
<p style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#201F1E">neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *</span><span style="font-family:"MS PGothic",sans-serif;color:#201F1E"><o:p></o:p></span></p>
<p style="background:white"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#201F1E">neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf</span><span style="font-family:"MS PGothic",sans-serif;color:#201F1E"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com">eandersson@blizzard.com</a>><br>
<b>Sent:</b> Thursday, October 10, 2019 6:18 PM<br>
<b>To:</b> Albert Braden <<a href="mailto:Albert.Braden@synopsys.com">Albert.Braden@synopsys.com</a>>; Chris Apsey <<a href="mailto:bitskrieg@bitskrieg.net">bitskrieg@bitskrieg.net</a>><br>
<b>Cc:</b> <a href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a> <<a href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> Re: Port creation times out for some VMs in large group</span> <o:p>
</o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Maybe double check that your rootwrap config is up to date?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">/etc/neutron/<span style="background:white">rootwrap .conf and /etc/neutron/rootwrap.d</span><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">(Make sure to pick the appropriate branch in github)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_neutron_blob_master_etc_rootwrap.conf&d=DwMFIw&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=29xksG5hik0V-K7r47f3d2k4DKuxpLDhvkdGmQ2rX5o&s=gAk-Efh-K5lP8Zm2rRhHbkXiZvUUFvZJtNKuqbmMukM&e=">https://github.com/openstack/neutron/blob/master/etc/rootwrap.conf</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_neutron_tree_master_etc_neutron_rootwrap.d&d=DwMFIw&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=29xksG5hik0V-K7r47f3d2k4DKuxpLDhvkdGmQ2rX5o&s=asd9vqO5bTt0d0a1Y2ckk_5HyxY4criuTMgVu-MQlDU&e=">https://github.com/openstack/neutron/tree/master/etc/neutron/rootwrap.d</a><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Albert Braden <<a href="mailto:Albert.Braden@synopsys.com">Albert.Braden@synopsys.com</a>><br>
<b>Sent:</b> Thursday, October 10, 2019 1:45 PM<br>
<b>To:</b> Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com">eandersson@blizzard.com</a>>; Chris Apsey <<a href="mailto:bitskrieg@bitskrieg.net">bitskrieg@bitskrieg.net</a>><br>
<b>Cc:</b> <a href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a> <<a href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> RE: Port creation times out for some VMs in large group</span> <o:p>
</o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">The errors appear to start with this line:</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">2019-10-10 13:42:48.261 1211336 ERROR neutron.agent.linux.utils [req-42c530f6-6e08-47c1-8ed4-dcb31c9cd972 - - - - -] Rootwrap error running command: ['iptables-save', '-t',
'raw']: Exception: Failed to spawn rootwrap process.</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">We’re not running iptables. Do we need it, to use the rootwrap daemon?</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="xxmsonormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Albert Braden <<a href="mailto:Albert.Braden@synopsys.com">Albert.Braden@synopsys.com</a>>
<br>
<b>Sent:</b> Thursday, October 10, 2019 12:13 PM<br>
<b>To:</b> Erik Olof Gunnar Andersson <<a href="mailto:eandersson@blizzard.com">eandersson@blizzard.com</a>>; Chris Apsey <<a href="mailto:bitskrieg@bitskrieg.net">bitskrieg@bitskrieg.net</a>><br>
<b>Cc:</b> <a href="mailto:openstack-discuss@lists.openstack.org">openstack-discuss@lists.openstack.org</a><br>
<b>Subject:</b> RE: Port creation times out for some VMs in large group</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
</div>
</div>
<p class="xxmsonormal"><span style="font-family:"MS PGothic",sans-serif"> <o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">It looks like something is still missing. I added the line to /etc/sudoers.d/neutron_sudoers:</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">root@us01odc-qa-ctrl3:/var/log/neutron# cat /etc/sudoers.d/neutron_sudoers</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Defaults:neutron !requiretty</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf *</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Then I restarted neutron services and the error was gone… for a few minutes, and then it came back on ctrl3. Ctrl1/2 aren’t erroring at this time. I changed neutron’s shell
and tested the daemon command and it seems to work:</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">root@us01odc-qa-ctrl3:~# su - neutron</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">neutron@us01odc-qa-ctrl3:~$ /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">/tmp/rootwrap-5b1QoP/rootwrap.sock</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Z%</span><span style="font-size:11.0pt;font-family:"Arial",sans-serif">‘</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">"</span><span style="font-size:11.0pt;font-family:"Arial",sans-serif">‘‘‘</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Vs</span><span style="font-size:11.0pt;font-family:"Arial",sans-serif">‘‘</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">5-</span><span style="font-size:11.0pt;font-family:"Arial",sans-serif">‘</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">,a</span><span style="font-size:11.0pt;font-family:"Arial",sans-serif">‘‘‘‘</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">G</span><span style="font-size:11.0pt;font-family:"Arial",sans-serif">‘‘‘‘</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">v</span><span style="font-size:11.0pt;font-family:"Arial",sans-serif">‘‘</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">But neutron-linuxbridge-agent.log still scrolls errors:</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><a href="https://urldefense.com/v3/__http:/paste.openstack.org/show/782740/__;!2E0gRdhhnqPNNL0!z5cwPxQ1y_zz0MvtFzMZSCIh7-3d80kxciHbPtkj4LbHCzSkzNpf36RwLi8kWGm1Ew$">http://paste.openstack.org/show/782740/</a></span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">It appears that there is another factor besides the config, because even when the sudoers line was missing, it would work for hours or days before the error started. It has
been working in our prod cluster for about a week now, without the sudoers line. It seems like it should not work that way. What am I missing?</span><span style="font-family:"MS PGothic",sans-serif"><o:p></o:p></span></p>
<p class="xxmsonormal"><span style="font-family:"MS PGothic",sans-serif"><o:p> </o:p></span></p>
<div>
<div>
<div>
<p class="xxxmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>