<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>Hi Kevin</div>
<div>Sure will log a bug</div>
<div>Also does the config change involve having both these lines in the neutron.conf file?</div>
<div>
<div>[agent]</div>
<div>root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf</div>
<div>root_helper_daemon = sudo neutron-rootwrap-daemon /etc/neutron/rootwrap.conf</div>
</div>
<div><br>
</div>
<div>If I have only the second line I see the exception below on neutron openvswitch agent bring up:</div>
<div><br>
</div>
<div>
<div>2017-09-12 09:23:03.633 35 DEBUG neutron.agent.linux.utils [req-0f8fe685-66bd-44d7-beac-bb4c24f0ccfa - - - - -] Running command: ['ps', '--ppid', '103', '-o', 'pid='] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:89</div>
<div>2017-09-12 09:23:03.762 35 ERROR ryu.lib.hub [req-0f8fe685-66bd-44d7-beac-bb4c24f0ccfa - - - - -] hub: uncaught exception: Traceback (most recent call last):</div>
<div> File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch</div>
<div> return func(*args, **kwargs)</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 42, in agent_main_wrapper</div>
<div> ovs_agent.main(bridge_classes)</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2184, in main</div>
<div> agent.daemon_loop()</div>
<div> File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper</div>
<div> return f(*args, **kwargs)</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2100, in daemon_loop</div>
<div> self.ovsdb_monitor_respawn_interval) as pm:</div>
<div> File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__</div>
<div> return self.gen.next()</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 35, in get_polling_manager</div>
<div> pm.start()</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 57, in start</div>
</div>
<div>
<div> while not self.is_active():</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 100, in is_active</div>
<div> self.pid, self.cmd_without_namespace)</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 159, in pid</div>
<div> run_as_root=self.run_as_root)</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 297, in get_root_helper_child_pid</div>
<div> pid = find_child_pids(pid)[0]</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 179, in find_child_pids</div>
<div> log_fail_as_error=False)</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 128, in execute</div>
<div> _stdout, _stderr = obj.communicate(_process_input)</div>
<div> File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate</div>
<div> return self._communicate(input)</div>
<div> File "/usr/lib64/python2.7/subprocess.py", line 1403, in _communicate</div>
<div> stdout, stderr = self._communicate_with_select(input)</div>
<div> File "/usr/lib64/python2.7/subprocess.py", line 1504, in _communicate_with_select</div>
<div> rlist, wlist, xlist = select.select(read_set, write_set, [])</div>
<div> File "/usr/lib/python2.7/site-packages/eventlet/green/select.py", line 86, in select</div>
<div> return hub.switch()</div>
<div> File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch</div>
<div> return self.greenlet.switch()</div>
<div>Timeout: 5 seconds</div>
<div><br>
</div>
<div>2017-09-12 09:23:03.860 35 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=95</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Ajay</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div id="MAC_OUTLOOK_SIGNATURE"></div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Kevin Benton <<a href="mailto:kevin@benton.pub">kevin@benton.pub</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Monday, September 11, 2017 at 1:12 PM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Cc: </span>"Ian Wells (iawells)" <<a href="mailto:iawells@cisco.com">iawells@cisco.com</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [neutron]OVS connection tracking cleanup<br>
</div>
<div><br>
</div>
<span style="mso-bookmark:_MailOriginalBody">
<div>
<div>
<div dir="ltr">Can you start a bug on launchpad and upload the conntrack attachment to the bug?
<div><br>
</div>
<div>Switching to the rootwrap daemon should also help significantly.</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Sep 11, 2017 at 12:32 PM, Ajay Kalambur (akalambu)
<span dir="ltr"><<a href="mailto:akalambu@cisco.com" target="_blank">akalambu@cisco.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>
<div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">Hi Kevin</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">The information you asked for</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">For 1 compute node with 45 Vms here is the number of connection tracking entries getting deleted</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<div>cat conntrack.file | wc -l</div>
<div> 38528</div>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">The file with output is 14MB so ill email it to Ian and he can share it if needed</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">Security group rules</div>
<div>
<div><font face="Calibri,sans-serif"><span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Direction<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Ether Type<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>IP
Protocol<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Port Range<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Remote IP Prefix<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Remote
Security Group<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Actions</font></div>
<div><font face="Calibri,sans-serif">Egress<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>IPv4<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span><a href="http://0.0.0.0/0" target="_blank">0.0.0.0/0</a><span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span></font></div>
<div><font face="Calibri,sans-serif">Ingress<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>IPv6<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>-<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>default<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span></font></div>
<div><font face="Calibri,sans-serif">Egress<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>IPv6<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>::/0<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>-<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span></font></div>
<div><font face="Calibri,sans-serif">Ingress<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>IPv4<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>-<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span></font></div>
</div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">Please let me know if u need the dump of conntrack entries if so I can email it to email address of your choice</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">Ajay</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<div id="m_4511050849682920103MAC_OUTLOOK_SIGNATURE"></div>
</div>
</div>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br>
</div>
<span id="m_4511050849682920103OLK_SRC_BODY_SECTION" style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<div style="font-family:Calibri;font-size:12pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<span style="font-weight:bold">From: </span>Ajay Kalambur <<a href="mailto:akalambu@cisco.com" target="_blank">akalambu@cisco.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.<wbr>openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Monday, September 11, 2017 at 10:02 AM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.<wbr>openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [neutron]OVS connection tracking cleanup<br>
</div>
<div>
<div class="h5">
<div><br>
</div>
<span>
<div>
<div dir="auto">
<div>Hi Kevin</div>
<div id="m_4511050849682920103AppleMailSignature">Thanks for your response it was about 50 vms </div>
<div id="m_4511050849682920103AppleMailSignature">Ajay<br>
<br>
<br>
</div>
<div><br>
On Sep 11, 2017, at 9:49 AM, Kevin Benton <<a href="mailto:kevin@benton.pub" target="_blank">kevin@benton.pub</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">The biggest improvement will be switching to native netlink calls: <a href="https://review.openstack.org/#/c/470912/" target="_blank">https://review.<wbr>openstack.org/#/c/470912/</a>
<div><br>
</div>
<div>How many VMs were on a single compute node?</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Sep 11, 2017 at 9:15 AM, Ajay Kalambur (akalambu)
<span dir="ltr"><<a href="mailto:akalambu@cisco.com" target="_blank">akalambu@cisco.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">Hi</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">I am performing a scale test and I see that after creating 500 Vms with ping traffic between them it took almost 1 hr for the connection tracking </div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">To clean up and ovs agent was busy doing this and unable to service any new port bind requests on some computes for almost an hr </div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">It took that long for conntrack clean up to complete</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">I see the following bug</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><a href="https://bugs.launchpad.net/neutron/+bug/1513765" target="_blank">https://bugs.launchpad.net/neu<wbr>tron/+bug/1513765</a></div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br>
</div>
<div>And I also have the fix below</div>
<div><a href="https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7aeb8dd4b1d122e17eef8687192cd122b79fd6e" target="_blank">https://git.openstack.org/cgit<wbr>/openstack/neutron/commit/?id=<wbr>d7aeb8dd4b1d122e17eef8687192cd<wbr>122b79fd6e</a></div>
<div><br>
</div>
<div><br>
</div>
<div>Still see really long times for conntrack cleanup</div>
<div><br>
</div>
<div>What is the solution to this problem in scale scenarios?</div>
<span class="m_4511050849682920103HOEnZb"><font color="#888888">
<div>Ajay</div>
<div><br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<div id="m_4511050849682920103m_-6662988751102660720MAC_OUTLOOK_SIGNATURE"></div>
</div>
</font></span></div>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>______________________________<wbr>______________________________<wbr>______________</span><br>
<span>OpenStack Development Mailing List (not for usage questions)</span><br>
<span>Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" target="_blank">
OpenStack-dev-request@lists.<wbr>openstack.org</a>?subject:<wbr>unsubscribe</span><br>
<span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a></span><br>
</div>
</blockquote>
</div>
</div>
</span></div>
</div>
</span></div>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</span></span>
</body>
</html>