<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-2022-jp">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
You are probably missing an entry in your sudoers file.<br>
<br>
You need something like</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<pre style="padding: 9.5px; font-family: Monaco, Menlo, Consolas, "Courier New", monospace; font-size: 13px; color: rgb(51, 51, 51); border-radius: 4px; display: block; margin: 0px 0px 10px; line-height: 20px; word-break: break-all; overflow-wrap: break-word; background-color: rgb(245, 245, 245); border: 1px solid rgba(0, 0, 0, 0.15)">neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf</pre>
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Albert Braden <Albert.Braden@synopsys.com><br>
<b>Sent:</b> Wednesday, October 9, 2019 5:20 PM<br>
<b>To:</b> Chris Apsey <bitskrieg@bitskrieg.net><br>
<b>Cc:</b> openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org><br>
<b>Subject:</b> RE: Port creation times out for some VMs in large group</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.x_MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
p.x_msonormal0, li.x_msonormal0, div.x_msonormal0
{margin-right:0in;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle20
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-family:"Calibri",sans-serif}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="blue" vlink="purple">
<div class="x_WordSection1">
<p class="x_MsoNormal">We tested this in dev and qa and then implemented in production and it did make a difference, but 2 weeks later we started seeing an issue, first in dev, and then in qa. In syslog we see neutron-linuxbridge-agent.service stopping and
starting[1]. In neutron-linuxbridge-agent.log we see a rootwrap error[2]: $B!H(BException: Failed to spawn rootwrap process.$B!I(B</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">If I comment out $B!F(Broot_helper_daemon = "sudo /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf"$B!G(B and restart neutron services then the error goes away.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">How can I use the root_helper_daemon setting without creating this new error?</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">[1]</p>
<p class="x_MsoNormal">Oct 9 13:48:38 us01odc-qa-ctrl1 systemd[1]: neutron-linuxbridge-agent.service: Main process exited, code=exited, status=1/FAILURE</p>
<p class="x_MsoNormal">Oct 9 13:48:38 us01odc-qa-ctrl1 systemd[1]: neutron-linuxbridge-agent.service: Failed with result 'exit-code'.</p>
<p class="x_MsoNormal">Oct 9 13:48:38 us01odc-qa-ctrl1 systemd[1]: neutron-linuxbridge-agent.service: Service hold-off time over, scheduling restart.</p>
<p class="x_MsoNormal">Oct 9 13:48:38 us01odc-qa-ctrl1 systemd[1]: neutron-linuxbridge-agent.service: Scheduled restart job, restart counter is at 2.</p>
<p class="x_MsoNormal">Oct 9 13:48:38 us01odc-qa-ctrl1 systemd[1]: Stopped Openstack Neutron Linux Bridge Agent.</p>
<p class="x_MsoNormal">Oct 9 13:48:38 us01odc-qa-ctrl1 systemd[1]: Starting Openstack Neutron Linux Bridge Agent...</p>
<p class="x_MsoNormal">Oct 9 13:48:38 us01odc-qa-ctrl1 systemd[1]: Started Openstack Neutron Linux Bridge Agent.</p>
<p class="x_MsoNormal">Oct 9 13:48:41 us01odc-qa-ctrl1 systemd[1]: neutron-linuxbridge-agent.service: Main process exited, code=exited, status=1/FAILURE</p>
<p class="x_MsoNormal">Oct 9 13:48:41 us01odc-qa-ctrl1 systemd[1]: neutron-linuxbridge-agent.service: Failed with result 'exit-code'.</p>
<p class="x_MsoNormal">Oct 9 13:48:41 us01odc-qa-ctrl1 systemd[1]: neutron-linuxbridge-agent.service: Service hold-off time over, scheduling restart.</p>
<p class="x_MsoNormal">Oct 9 13:48:41 us01odc-qa-ctrl1 systemd[1]: neutron-linuxbridge-agent.service: Scheduled restart job, restart counter is at 3.</p>
<p class="x_MsoNormal">Oct 9 13:48:41 us01odc-qa-ctrl1 systemd[1]: Stopped Openstack Neutron Linux Bridge Agent.</p>
<p class="x_MsoNormal">Oct 9 13:48:41 us01odc-qa-ctrl1 systemd[1]: Starting Openstack Neutron Linux Bridge Agent...</p>
<p class="x_MsoNormal">Oct 9 13:48:41 us01odc-qa-ctrl1 systemd[1]: Started Openstack Neutron Linux Bridge Agent.</p>
<p class="x_MsoNormal">Oct 9 13:48:43 us01odc-qa-ctrl1 systemd[1]: neutron-linuxbridge-agent.service: Main process exited, code=exited, status=1/FAILURE</p>
<p class="x_MsoNormal">Oct 9 13:48:43 us01odc-qa-ctrl1 systemd[1]: neutron-linuxbridge-agent.service: Failed with result 'exit-code'.</p>
<p class="x_MsoNormal">Oct 9 13:48:43 us01odc-qa-ctrl1 systemd[1]: neutron-linuxbridge-agent.service: Service hold-off time over, scheduling restart.</p>
<p class="x_MsoNormal">Oct 9 13:48:43 us01odc-qa-ctrl1 systemd[1]: neutron-linuxbridge-agent.service: Scheduled restart job, restart counter is at 4.</p>
<p class="x_MsoNormal">Oct 9 13:48:43 us01odc-qa-ctrl1 systemd[1]: Stopped Openstack Neutron Linux Bridge Agent.</p>
<p class="x_MsoNormal">Oct 9 13:48:43 us01odc-qa-ctrl1 systemd[1]: neutron-linuxbridge-agent.service: Start request repeated too quickly.</p>
<p class="x_MsoNormal">Oct 9 13:48:43 us01odc-qa-ctrl1 systemd[1]: neutron-linuxbridge-agent.service: Failed with result 'exit-code'.</p>
<p class="x_MsoNormal">Oct 9 13:48:43 us01odc-qa-ctrl1 systemd[1]: Failed to start Openstack Neutron Linux Bridge Agent.<br>
<br>
</p>
<p class="x_MsoNormal">[2]</p>
<p class="x_MsoNormal">2019-10-09 17:05:24.519 5803 INFO neutron.common.config [-] Logging enabled!</p>
<p class="x_MsoNormal">2019-10-09 17:05:24.519 5803 INFO neutron.common.config [-] /usr/bin/neutron-linuxbridge-agent version 13.0.4</p>
<p class="x_MsoNormal">2019-10-09 17:05:24.520 5803 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface mappings: {'physnet1': 'eno1'}</p>
<p class="x_MsoNormal">2019-10-09 17:05:24.520 5803 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Bridge mappings: {}</p>
<p class="x_MsoNormal">2019-10-09 17:05:24.522 5803 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/linuxbridge_agent.ini',
'--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpmdyxcD/privsep.sock']</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.071 5803 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.022 5828 INFO oslo.privsep.daemon [-] privsep daemon starting</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.025 5828 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.027 5828 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/none</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.027 5828 INFO oslo.privsep.daemon [-] privsep daemon running as pid 5828</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.125 5803 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Agent initialized successfully, now running...</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.193 5803 ERROR neutron.agent.linux.utils [req-8aaf64a2-8f0d-44ce-888f-09ae3d1acd78 - - - - -] Rootwrap error running command: ['iptables-save', '-t', 'raw']: Exception: Failed to spawn rootwrap process.</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service [req-8aaf64a2-8f0d-44ce-888f-09ae3d1acd78 - - - - -] Error starting thread.: Exception: Failed to spawn rootwrap process.</p>
<p class="x_MsoNormal">stderr:</p>
<p class="x_MsoNormal">sudo: no tty present and no askpass program specified</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service Traceback (most recent call last):</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 794, in run_service</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service service.start()</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 158, in wrapper</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service result = f(*args, **kwargs)</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 86, in start</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service self.setup_rpc()</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 158, in wrapper</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service result = f(*args, **kwargs)</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 153, in setup_rpc</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service self.context, self.sg_plugin_rpc, defer_refresh_firewall=True)</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 58, in __init__</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service self.init_firewall(defer_refresh_firewall, integration_bridge)</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py", line 83, in init_firewall</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service self.firewall = firewall_class()</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/iptables_firewall.py", line 88, in __init__</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service zone_per_port=self.CONNTRACK_ZONE_PER_PORT)</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service return f(*args, **kwargs)</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_conntrack.py", line 58, in get_conntrack</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service execute, namespace, zone_per_port)</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_conntrack.py", line 75, in __init__</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service self._populate_initial_zone_map()</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_conntrack.py", line 182, in _populate_initial_zone_map</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service rules = self.get_rules_for_table_func('raw')</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/iptables_manager.py", line 477, in get_rules_for_table</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service return self.execute(args, run_as_root=True).split('\n')</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 122, in execute</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service execute_rootwrap_daemon(cmd, process_input, addl_env))</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 109, in execute_rootwrap_daemon</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service LOG.error("Rootwrap error running command: %s", cmd)</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service self.force_reraise()</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service six.reraise(self.type_, self.value, self.tb)</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 106, in execute_rootwrap_daemon</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service return client.execute(cmd, process_input)</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_rootwrap/client.py", line 148, in execute</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service self._ensure_initialized()</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_rootwrap/client.py", line 115, in _ensure_initialized</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service self._initialize()</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_rootwrap/client.py", line 85, in _initialize</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service (stderr,))</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service Exception: Failed to spawn rootwrap process.</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service stderr:</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service sudo: no tty present and no askpass program specified</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.194 5803 ERROR oslo_service.service</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.197 5803 INFO neutron.plugins.ml2.drivers.agent._common_agent [-] Stopping Linux bridge agent agent.</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 CRITICAL neutron [-] Unhandled error: AttributeError: 'CommonAgentLoop' object has no attribute 'state_rpc'</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron Traceback (most recent call last):</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron File "/usr/bin/neutron-linuxbridge-agent", line 10, in <module></p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron sys.exit(main())</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/cmd/eventlet/plugins/linuxbridge_neutron_agent.py", line 21, in main</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron agent_main.main()</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 1051, in main</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron launcher.wait()</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 392, in wait</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron status, signo = self._wait_for_exit_or_signal()</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 377, in _wait_for_exit_or_signal</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron self.stop()</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 292, in stop</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron self.services.stop()</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 760, in stop</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron service.stop()</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 158, in wrapper</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron result = f(*args, **kwargs)</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 117, in stop</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron self.set_rpc_timeout(self.quitting_rpc_timeout)</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 158, in wrapper</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron result = f(*args, **kwargs)</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 476, in set_rpc_timeout</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron self.state_rpc):</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron AttributeError: 'CommonAgentLoop' object has no attribute 'state_rpc'</p>
<p class="x_MsoNormal">2019-10-09 17:05:25.198 5803 ERROR neutron</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><b>From:</b> Chris Apsey <bitskrieg@bitskrieg.net> <br>
<b>Sent:</b> Friday, September 27, 2019 9:34 AM<br>
<b>To:</b> Albert Braden <albertb@synopsys.com><br>
<b>Cc:</b> openstack-discuss@lists.openstack.org<br>
<b>Subject:</b> Re: Port creation times out for some VMs in large group</p>
<p class="x_MsoNormal"> </p>
<div>
<p class="x_MsoNormal">Albert,</p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal">Do this: <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__cloudblog.switch.ch_2017_08_28_starting-2D1000-2Dinstances-2Don-2Dswitchengines_&d=DwMGaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=izFVaT90rpGh_939STWbrLr4vnSwK2KBtqFKv_J8Gfs&s=UO9bh6wArCWKbxRWfWjt8egNaw9cxrbDwCZ8-2t0GmE&e=">
https://cloudblog.switch.ch/2017/08/28/starting-1000-instances-on-switchengines/</a></p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal">The problem will go away. I'm of the opinion that daemon mode for rootwrap should be the default since the performance improvement is an order of magnitude, but privsep may obviate that concern once its fully implemented.</p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal">Either way, that should solve your problem.</p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<div>
<div>
<p class="x_MsoNormal">r</p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal">Chris Apsey</p>
</div>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal">$B!>!>!>!>!>!>!>(B Original Message $B!>!>!>!>!>!>!>(B</p>
</div>
<div>
<p class="x_MsoNormal">On Friday, September 27, 2019 12:17 PM, Albert Braden <<a href="mailto:Albert.Braden@synopsys.com">Albert.Braden@synopsys.com</a>> wrote:</p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<blockquote style="margin-top:5.0pt; margin-bottom:5.0pt">
<div>
<p>When I create 100 VMs in our prod cluster:</p>
<p> </p>
<p>openstack server create --flavor s1.tiny --network it-network --image cirros-0.4.0-x86_64 --min 100 --max 100 alberttest</p>
<p> </p>
<p>Most of them build successfully in about a minute. 5 or 10 will stay in BUILD status for 5 minutes and then fail with $B!H(B BuildAbortException: Build of instance <UUID> aborted: Failed to allocate the network(s), not rescheduling.$B!I(B</p>
<p> </p>
<p>If I build smaller numbers, I see less failures, and no failures if I build one at a time. This does not happen in dev or QA; it appears that we are exhausting a resource in prod. I tried reducing various config values in dev but am not able to duplicate
the issue. The neutron servers don$B!G(Bt appear to be overloaded during the failure.</p>
<p> </p>
<p>What config variables should I be looking at?</p>
<p> </p>
<p>Here are the relevant log entries from the HV:</p>
<p> </p>
<p>2019-09-26 10:10:43.001 57008 INFO os_vif [req-dea54d9a-3f3e-4d47-b901-a4f41b1947a8 d28c3871f61e4c8c8f8c7600417f7b14 e9621e3b105245ba8660f434ab21016c - default 4fb72165eee4468e8033cdc7d506ddf0] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:8b:45:07,bridge_name='brq49cbe55d-51',has_traffic_filtering=True,id=18f4e419-b19c-4b62-b6e4-152ec78e72bc,network=Network(49cbe55d-5188-4183-b5ad-e65f9b46f8f2),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap18f4e419-b1')</p>
<p>2019-09-26 10:15:44.029 57008 WARNING nova.virt.libvirt.driver [req-dea54d9a-3f3e-4d47-b901-a4f41b1947a8 d28c3871f61e4c8c8f8c7600417f7b14 e9621e3b105245ba8660f434ab21016c - default 4fb72165eee4468e8033cdc7d506ddf0] [instance: dc58f154-00f9-4c45-8986-94b10821cbc9]
Timeout waiting for [('network-vif-plugged', u'18f4e419-b19c-4b62-b6e4-152ec78e72bc')] for instance with vm_state building and task_state spawning.: Timeout: 300 seconds</p>
<p> </p>
<p>More logs and data:</p>
<p> </p>
<p><a href="https://urldefense.com/v3/__http://paste.openstack.org/show/779524/__;!2E0gRdhhnqPNNL0!wGtOA-OJ_wlWtnIouvVoT4cU2U8Lxu_O3U6DeY1vBMUEV0iavwJCBL9kUcLkfiUm4w$">http://paste.openstack.org/show/779524/</a></p>
<p> </p>
</div>
</blockquote>
<div>
<p class="x_MsoNormal"> </p>
</div>
</div>
</div>
</body>
</html>